Information is a subject of multiple efforts of conceptualization leading to controversies. Not frequently sufficient effort is made to formulate the concept of information in a way leading to its formal mathematical theory. Discussions of conceptualizations of information usually are focusing on the articulation of definitions, but not on their consequences for theoretical studies. This paper compares two conceptualizations of information exploring their mathematical theories. One of these concepts and its mathematical theory were introduced in earlier publications of the author. Information was defined in terms of the opposition of one and many and its theory was formulated in terms of closure spaces. The other concept of information was formulated in a rather open-ended way by Bateson as “any difference that makes a difference”. There are some similarities between Bateson’s concept of information and that of MacKay. In this paper a mathematical theory is formulated for this alternative approach to information founded on the concept of a difference in terms of generalized orthogonality relation. Finally, the mathematical formalisms for both approaches are compared and related. In conclusion of that comparison the approach to information founded on the concept of difference is a special case for the approach based on one-and-many opposition.