The Entropy Sign
Information is not uncertainty
“random is a matter of perspective in this case. It means that the perfect transmission code looks random to an outsider”.
Jeff Abramsohn
jabramsohn@jhancock.com
“If someone says that information = uncertainty = entropy, then they are confused, or something was not stated that should have been. Those equalities lead to a contradiction, since entropy of a system increases as the system becomes more disordered. So information corresponds to disorder according to this confusion.”
“If you always take information to be a decrease in uncertainty at the receiver and you will get straightened out”.
e.g. R = Hbefore - Hafter (where R is the information transferred)
in an alphabet: c, g, a or t:
and a single character is transmitted in a noiseless channel:
uncertainty before is 4 bits, after is 0 bits.
Basically information is the amount to uncertainty eliminated. In the Shannon entropy scenario, information is something which has been transferred, rather like energy vs potential energy. A large random string can carry a lot of information, it has high ‘potential entropy’ but without transfer and elimination of uncertainty it has no information from receiver’s frame of reference. Suppose that only a certain portion of the bits in a transmitter could be interpreted by a receiver, then the remainder will be perceived as noise.
The thing that is odd about entropy is not so much the apparent but false paradox of information and randomness, but the units of entropy. Energy / temperature merely gives the signal above the noise, whereas a more interesting possible unit would be some property which included the variance in energy per bit (expand with Dyson Sphere entropy measure example).
For example, what if all information exchange (and system-wide net entropy increase) resulted in a lower energy per bit, e.g. lower frequency photons.
*********
NB entropy is unitless.
In thermodynsmics, entropy is measured in J/K where temperature and energy are both energy measures.
T=kT.
In other words, in thermodynamics you could say entropy is relative (lack of) energy.
rss
April 7th, 2010 at 3:31 am
Another way to look at the entropy sign and to avoid the information = randomness fallacy.
Entropy = unfree energy. (i.e. energy you cannot do anything useful with)
This involves things like a bunch of gas particles whizzing around, the more gas particles the more energy, but in equilibrium, the less net energy in any particular direction.
Logical entropy = unusable bits = noise = ‘potential’ information
This involves things like a bunch of random bits which have no meaning, the more bits, the more potential information, but in a random state, less information.
In other words bits are like energy and information is like free energy.
[ Since the information content of a bitstream is dependent on the decoder and the history of previously decoded messages (things learn, and the more they know the more meaning a particular message may have (e.g. a physics paper shown to a 2 year old, yields different information content to when a physicist reads it) the information it contains is relative to the person measuring. Why isn't the energy measure also relative to the person measuring ]
April 7th, 2010 at 3:34 am
http://www-lmmb.ncifcrf.gov/~toms/information.is.not.uncertainty.html