Editing Entropy (information theory) (section)

===Relative entropy===
{{main|Generalized relative entropy}}
Another useful measure of entropy that works equally well in the discrete and the continuous case is the '''relative entropy''' of a distribution. It is defined as the [[Kullback–Leibler divergence]] from the distribution to a reference measure {{math|''m''}} as follows. Assume that a probability distribution {{math|''p''}} is [[absolutely continuous]] with respect to a measure {{math|''m''}}, i.e. is of the form {{math|''p''(''dx'') {{=}} ''f''(''x'')''m''(''dx'')}} for some non-negative {{math|''m''}}-integrable function {{math|''f''}} with {{math|''m''}}-integral 1, then the relative entropy can be defined as
<math display="block">D_{\mathrm{KL}}(p \| m ) = \int \log (f(x)) p(dx) = \int f(x)\log (f(x)) m(dx) .</math>

In this form the relative entropy generalizes (up to change in sign) both the discrete entropy, where the measure {{math|''m''}} is the [[counting measure]], and the differential entropy, where the measure {{math|''m''}} is the [[Lebesgue measure]]. If the measure {{math|''m''}} is itself a probability distribution, the relative entropy is non-negative, and zero if {{math|''p'' {{=}} ''m''}} as measures. It is defined for any measure space, hence coordinate independent and invariant under co-ordinate reparameterizations if one properly takes into account the transformation of the measure {{math|''m''}}. The relative entropy, and (implicitly) entropy and differential entropy, do depend on the "reference" measure {{math|''m''}}.