Entropy

From EvoWiki

Since creationists are very happy about the word “entropy”, its relation to order, disorder, and information, and whether it decreases or increases, it might be worth having a look at, what is actually meant by that word.

Entropy in thermodynamics

During the first half of the 19^th century researchers, such as Sadi Carnot, had noted that not all energy could be converted to work. For instance, while a process transferring energy from a hotter to a cooler body is possible and will occur spontaneously if the two bodies are connected, this process will halt, when the two bodies have the same temperature. And a process transferring energy from a cooler to a hotter body is not possible.

The term “entropy” was originally coined by Rudolf Clausius in the 1850s to refer to this equalization of a thermodynamic potential, and the 2^nd law of thermodynamics states that in an isolated system, the entropy will either remain the same or increase; that is, potentials are minimized, whereas the entropy is maximized. If you put an open can of hot water into a colder room, you create a thermodynamic potential, the temperature difference, which will generate a movement of heat from the can of water to the surrounding air, until equilibrium is attained. The potential gradient functions as a thermodynamic “force”, and natural processes cannot work against the gradient; natural processes can only increase, not decrease entropy. That is, the process is irreversible, and the 2^nd law therefore induces an orientation of time.

In thermodynamics only the boundaries between systems are considered. A polar bear has a smaller boundary (surface area) compared to its volume than, asay, an african elephant. It also has a thick fur to provide insulation. The polar bear lives in areas with a temperature much lower than its own internal temperature and; that is, between the polar bear and its surroundings there is a steep thermodynamic gradient. The body shape of the polar bear keeps energy transfer from the bear to its surroundings at a low speed, an adaptation to the environment. Bear species living further south, where the thermodynamic gradient is less steep, have smaller bodies and larger extremities (e.g. ear lopes and tails). The african elephant has the opposite problem, to keep cooled. It has no fur to prevent air movement, and its large ear lopes can be used as fans to circulate air, an adaptation to a different environment.

Entropy in statistical mechanics

The thermodynamic entropy of Clausius is macroscopic in that it is not concerned with the interior composition of the system in question; but as scientists learned about molecules, classical thermodynamics grew into statistical mechanics - an area where unequal probabilities were indeed well known. Ludwig Boltzmann had first introduced the quantity</p>

H = - Σ f_i log f_i

as a measure of the diversity in states available to a single particle in a gas of like particles. In this definition f represents the relative frequency distribution of each possible state. Boltzmann argued mathematically that the effect of collisions between the particles would cause the value of H to increase from any initial configuration until equilibrium was reached. This microscopic state function was identified as the expalanation for Clausius’ macoscopic entropy.

Whereas Boltzmann was concerned with an individual particle in a group of like particles, the American mathematical physicist J. Willard Gibbs changed the formulation to consider the complete microstate i of the entire system, and thereby arriving at the following formula:

S = - k_B å p_i ln p_i

where k_B is Boltzmann’s constant, which more correctly correspond with Clausius' thermodynamical definition.

Entropy in information theory

Claude Shannon’s model was that of a sender sending one out of N possible messages through a noise-induced channel. At the receiving end, how much certainty could there be about, which message was sent?

In his 1948 paper A Mathematical Theory of Communication, p.2., Shannon writes:

Suppose we have a set of possible events whose probabilities of occurrence are p₁, p₂,…, p_n. These probabilities are known but that is all we know concerning which event will occur. Can we find a measure of how much “choice” is involved in the selection of the event or of how uncertain we are of the outcome?

If there is such a measure, say H(p₁, p₂,…, p_n), it is reasonable to require of it the following properties: …

At p. 10-11 of the same paper, Shannon discusses the definition of a measure of “choice” or uncertainty in somewhat more general terms. Assume a (finite) set of possible events with probability p_i of occurrence of the i^th event. If all that is known concerning which event will occur is these probabilities, how can we define a measure H(p₁, p₂,…, p_n) of how much “choice” is involved in the selection of the event or of how uncertain we are of the outcome?