p 75

Axiomatic characterization of information

The Shannon formula (2.22) may be characterized by the following three axioms (Fadeev’s axiom). H is:

From these three conditions (2.22) is fixed except for the base of the logarithm. The key point of the proof is to show f(p) ∝ p log p + (1 － p) log(1 － p). If this is shown we may use (3) repeatedly to obtain (2.22). The key step to determine f(p) is not very trivial. See M. Tuerberg, Math. Scand. 6, 297 (1958).

 The remaining question after Fadeev was to relax (1). That is to weaken the requirement of continuity. Tuerberg replaced the continuity with the integrability on [0,1]. The ultimate version is probably the one due to P. M. Lee, ``On the axiom of information theory,’’ Ann. Mat. Statistics  35 , 415 (1964). This requires only the Lebesgue measurability of f (the paper is not self-contained).

Physical meaning of information

Everyone believes that information is quite a fundamental quantity as energy. However, the basic theory even slightly related to information may be only thermodynamics. 

 Thermodynamics is not regarded as the fundamental theory by many, so we could assert that information has not been taken very seriously in physics. It is believed that information is conserved microscopically just as energy. If we adopt the point of view that the information of a state is determined by the number of microscopic states realizing the state, as long as we regard microscopic states to be elementary events, information must be conserved. If each elementary event is regarded as consisting of more fundamental events, we must re-count the number of elementary events, and the total information is boosted by the information in each original elementary event (which is no more elementary), but the conservation law remains intact. It is usually the case that if the information conservation is violated, we believe that we miscounted (or misrecognized/misidentified) the number of elementary events. If we consider in this way, the conservation law of information is not an empirical fact, but an expression of the belief that the world can be described in terms of fundamental elementary events. However, this may be false (recall the difference between the Jordan measure and the Lebesgue measure on p70)．