Editing Shannon–Hartley theorem

{{Short description|Theorem that tells the maximum rate at which information can be transmitted}}
{{distinguish|Nyquist–Shannon sampling theorem}}
{{Information theory}}
In [[information theory]], the '''Shannon–Hartley theorem''' tells the maximum rate at which information can be transmitted over a communications channel of a specified [[bandwidth (signal processing)|bandwidth]] in the presence of [[noise (electronics)|noise]]. It is an application of the [[noisy-channel coding theorem]] to the archetypal case of a [[continuous-time]] [[Analog signal|analog]] [[communications channel]] subject to [[Gaussian noise]]. The theorem establishes Shannon's [[channel capacity]] for such a communication link, a bound on the maximum amount of error-free [[information]] per time unit that can be transmitted with a specified [[Bandwidth (signal processing)|bandwidth]] in the presence of the noise interference, assuming that the signal power is bounded, and that the Gaussian noise process is characterized by a known power or power spectral density. The law is named after [[Claude Elwood Shannon|Claude Shannon]] and [[Ralph Hartley]].

==Statement of the theorem==

The Shannon–Hartley theorem states the [[channel capacity]] <math>C</math>, meaning the theoretical [[Tightness of measures|tightest]] upper bound on the [[information rate]] of data that can be communicated at an arbitrarily low [[bit error rate|error rate]] using an average received signal power <math>S</math> through an analog communication channel subject to [[additive white Gaussian noise]] (AWGN) of power {{nowrap|<math>N</math>:}}

:<math>C = B \log_2 \left( 1+\frac{S}{N} \right) </math>

where
* <math>C</math> is the [[channel capacity]] in [[bits per second]], a theoretical upper bound on the [[net bit rate]] (information rate, sometimes denoted <math>I</math>) excluding error-correction codes;
* <math>B</math> is the [[bandwidth (signal processing)|bandwidth]] of the channel in [[hertz]] ([[passband]] bandwidth in case of a bandpass signal);
* <math>S</math> is the average received signal power over the bandwidth (in case of a carrier-modulated passband transmission, often denoted ''[[Carrier-to-noise ratio|C]]''), measured in watts (or volts squared);
* <math>N</math> is the average power of the noise and interference over the bandwidth, measured in watts (or volts squared); and
* <math>S/N</math> is the [[signal-to-noise ratio]] (SNR) or the [[carrier-to-noise ratio]] (CNR) of the communication signal to the noise and interference at the receiver (expressed as a linear power ratio, not as logarithmic [[decibels]]).

==Historical development==

During the late 1920s, [[Harry Nyquist]] and [[Ralph Hartley]] developed a handful of fundamental ideas related to the transmission of information, particularly in the context of the [[telegraph]] as a communications system. At the time, these concepts were powerful breakthroughs individually, but they were not part of a comprehensive theory. In the 1940s, [[Claude Elwood Shannon|Claude Shannon]] developed the concept of channel capacity, based in part on the ideas of Nyquist and Hartley, and then formulated a complete theory of information and its transmission.

===Nyquist rate===
{{main|Nyquist ISI criterion}}
In 1927, Nyquist determined that the number of independent pulses that could be put through a telegraph channel per unit time is limited to twice the one-sided [[Bandwidth (signal processing)|bandwidth]] of the channel. In symbolic notation,

:<math>f_p \le 2B </math>

where <math>f_p</math> is the pulse frequency (in pulses per second) and <math>B</math> is the one-sided bandwidth{{clarify|reason=what is the one sided bandwidth?|date=May 2025}} (in hertz). The quantity <math>2B</math> later came to be called the ''[[Nyquist rate#Relative to signaling|Nyquist rate]]'', and transmitting at the limiting pulse rate of <math>2B</math> pulses per second as ''signalling at the Nyquist rate''. Nyquist published his results in 1928 as part of his paper "Certain topics in Telegraph Transmission Theory".<ref name="Nyquist">
{{cite journal|last=Nyquist|first=Harry|title=Certain topics in telegraph transmission theory|journal=Trans. AIEE|volume=47|pages=617–44|date=April 1928|issue=2|doi=10.1109/T-AIEE.1928.5055024|bibcode=1928TAIEE..47..617N|url=https://bayes.wustl.edu/Manual/CertainTopicsInTelegraphTransmissionTheory.pdf}} Also 2002 Reprint {{doi|10.1109/5.989873}}
</ref>

===Hartley's law===

During 1928, Hartley formulated a way to quantify information and its [[line rate]] (also known as [[data signalling rate]] ''R'' bits per second).<ref>{{cite journal | first = R. V. L.|last=Hartley | title = Transmission of Information | url = http://www.dotrose.com/etext/90_Miscellaneous/transmission_of_information_1928b.pdf | journal = Bell System Technical Journal |date=July 1928| volume = 7 | issue = 3 | pages = 535–563 | doi = 10.1002/j.1538-7305.1928.tb01236.x }}</ref> This method, later known as Hartley's law, became an important precursor for Shannon's more sophisticated notion of channel capacity.

Hartley argued that the maximum number of distinguishable pulse levels that can be transmitted and received reliably over a communications channel is limited by the dynamic range of the signal amplitude and the precision with which the receiver can distinguish amplitude levels. Specifically, if the amplitude of the transmitted signal is restricted to the range of [−''A'' ... +''A''] volts, and the precision of the receiver is ±Δ''V'' volts, then the maximum number of distinct pulses ''M'' is given by

:<math>M = 1 + { A \over \Delta V } </math>.

By taking information per pulse in bit/pulse to be the base-2-[[logarithm]] of the number of distinct messages ''M'' that could be sent, Hartley<ref>{{cite book | title = Information Theory; and its Engineering Applications | first= D. A.| last=Bell | edition = 3rd | year = 1962 | publisher = Pitman | location = New York|isbn=9780273417576|url=https://books.google.com/books?id=_O1SAAAAMAAJ}}</ref> constructed a measure of the line rate ''R'' as:

:<math> R  = f_p \log_2(M), </math>

where <math>f_p</math> is the pulse rate, also known as the symbol rate, in symbols/second or [[baud]].

Hartley then combined the above quantification with Nyquist's observation that the number of independent pulses that could be put through a channel of one-sided bandwidth <math>B</math> [[hertz]] was <math>2B</math> pulses per second, to arrive at his quantitative measure for achievable line rate.

Hartley's law is sometimes quoted as just a proportionality between the [[bandwidth (signal processing)|analog bandwidth]], <math>B</math>, in Hertz and what today is called the [[bandwidth (computing)|digital bandwidth]], <math>R</math>, in bit/s.<ref>{{cite book | title = Introduction to Telecommunications | edition = 2nd |first= Anu A. |last=Gokhale | publisher = Thomson Delmar Learning | year = 2004 | isbn = 1-4018-5648-9 | url = https://books.google.com/books?id=QowmxWAOEtYC&q=%22hartley%27s+law%22+proportional&pg=PA37 }}</ref>
Other times it is quoted in this more quantitative form, as an achievable line rate of <math>R</math> bits per second:<ref>{{cite book | title = Telecommunications Engineering | first1 = John|last1=Dunlop|first2=D. Geoffrey|last2=Smith | publisher = CRC Press | year = 1998 | url = https://books.google.com/books?id=-kyPyn3Dst8C&q=%22hartley%27s+law%22&pg=RA4-PA30 | isbn = 0-7487-4044-9 }}</ref>

:<math> R \le 2B \log_2(M). </math>

Hartley did not work out exactly how the number ''M'' should depend on the noise statistics of the channel, or how the communication could be made reliable even when individual symbol pulses could not be reliably distinguished to ''M'' levels; with Gaussian noise statistics, system designers had to choose a very conservative value of <math>M</math> to achieve a low error rate.

The concept of an error-free capacity awaited Claude Shannon, who built on Hartley's observations about a logarithmic measure of information and Nyquist's observations about the effect of bandwidth limitations.

Hartley's rate result can be viewed as the capacity of an errorless ''M''-ary channel of <math>2B</math> symbols per second. Some authors refer to it as a capacity. But such an errorless channel is an idealization, and if M is chosen small enough to make the noisy channel nearly errorless, the result is necessarily less than the Shannon capacity of the noisy channel of bandwidth <math>B</math>, which is the Hartley–Shannon result that followed later.

===Noisy channel coding theorem and capacity===
{{main|Noisy-channel coding theorem}}
[[Claude Elwood Shannon|Claude Shannon]]'s development of [[information theory]] during World War II provided the next big step in understanding how much information could be reliably communicated through noisy channels. Building on Hartley's foundation, Shannon's [[noisy channel coding theorem]] (1948) describes the maximum possible efficiency of [[error-correcting code|error-correcting methods]] versus levels of noise interference and data corruption.<ref>{{cite book | first = C. E.|last=Shannon | author-link = Claude E. Shannon | title = The Mathematical Theory of Communication | location = Urbana, IL|publisher=University of Illinois Press | orig-year = 1949| year = 1998|url=https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf}}</ref><ref>{{cite journal |first = C. E. |last=Shannon |author-link = Claude E. Shannon |title = Communication in the presence of noise |journal = [[Proceedings of the Institute of Radio Engineers]] |volume = 37 |issue = 1 |pages = 10–21 |date = January 1949 |doi=10.1109/JRPROC.1949.232969 |s2cid=52873253 |url=http://www.stanford.edu/class/ee104/shannonpaper.pdf |archive-url=https://web.archive.org/web/20100208112344/http://www.stanford.edu/class/ee104/shannonpaper.pdf |archive-date=8 February 2010 |url-status=dead}}</ref> The proof of the theorem shows that a randomly constructed error-correcting code is essentially as good as the best possible code; the theorem is proved through the statistics of such random codes.

Shannon's theorem shows how to compute a [[channel capacity]] from a statistical description of a channel, and establishes that given a noisy channel with capacity <math>C</math> and information transmitted at a line rate <math>R</math>, then if

:<math> R < C </math>

there exists a coding technique which allows the probability of error at the receiver to be made arbitrarily small. This means that theoretically, it is possible to transmit information nearly without error up to nearly a limit of <math>C</math> bits per second.

The converse is also important. If

:<math> R > C </math>

the probability of error at the receiver increases without bound as the rate is increased, so no useful information can be transmitted beyond the channel capacity. The theorem does not address the rare situation in which rate and capacity are equal.

The Shannon–Hartley theorem establishes what that channel capacity is for a finite-bandwidth [[continuous-time]] channel subject to Gaussian noise. It connects Hartley's result with Shannon's channel capacity theorem in a form that is equivalent to specifying the ''M'' in Hartley's line rate formula in terms of a signal-to-noise ratio, but achieving reliability through error-correction coding rather than through reliably distinguishable pulse levels.

If there were such a thing as a noise-free analog channel, one could transmit unlimited amounts of error-free data over it per unit of time (Note that an infinite-bandwidth analog channel could not transmit unlimited amounts of error-free data absent infinite signal power). Real channels, however, are subject to limitations imposed by both finite bandwidth and nonzero noise.

Bandwidth and noise affect the rate at which information can be transmitted over an analog channel. Bandwidth limitations alone do not impose a cap on the maximum information rate because it is still possible for the signal to take on an indefinitely large number of different voltage levels on each symbol pulse, with each slightly different level being assigned a different meaning or bit sequence. Taking into account both noise and bandwidth limitations, however, there is a limit to the amount of information that can be transferred by a signal of a bounded power, even when sophisticated multi-level encoding techniques are used.

In the channel considered by the Shannon–Hartley theorem, noise and signal are combined by addition. That is, the receiver measures a signal that is equal to the sum of the signal encoding the desired information and a continuous random variable that represents the noise. This addition creates uncertainty as to the original signal's value. If the receiver has some information about the random process that generates the noise, one can in principle recover the information in the original signal by considering all possible states of the noise process. In the case of the Shannon–Hartley theorem, the noise is assumed to be generated by a Gaussian process with a known variance. Since the variance of a Gaussian process is equivalent to its power, it is conventional to call this variance the noise power.

Such a channel is called the Additive White Gaussian Noise channel, because Gaussian noise is added to the signal; "white" means equal amounts of noise at all frequencies within the channel bandwidth. Such noise can arise both from random sources of energy and also from coding and measurement error at the sender and receiver respectively. Since sums of independent Gaussian random variables are themselves Gaussian random variables, this conveniently simplifies analysis, if one assumes that such error sources are also Gaussian and independent.

==Implications of the theorem==

===Comparison of Shannon's capacity to Hartley's law===
Comparing the channel capacity to the information rate from Hartley's law, we can find the effective number of distinguishable levels ''M'':<ref>{{cite book | title = An Introduction to Information Theory: symbols, signals & noise
|first= John Robinson |last=Pierce | url = https://archive.org/details/introductiontoin00john | url-access = registration
| publisher = Courier | year = 1980 | isbn = 0-486-24061-4 }}</ref>

:<math>2B \log_2(M) = B \log_2 \left( 1+\frac{S}{N} \right) </math>
:<math>M = \sqrt{1+\frac{S}{N}}.</math>

The square root effectively converts the power ratio back to a voltage ratio, so the number of levels is approximately proportional to the ratio of signal [[RMS amplitude]] to noise standard deviation.

This similarity in form between Shannon's capacity and Hartley's law should not be interpreted to mean that <math>M</math> pulse levels can be literally sent without any confusion. More levels are needed to allow for redundant coding and error correction, but the net data rate that can be approached with coding is equivalent to using that <math>M</math> in Hartley's law.

==Frequency-dependent (colored noise) case==

In the simple version above, the signal and noise are fully uncorrelated, in which case <math>S+N</math> is the total power of the received signal and noise together. A generalization of the above equation for the case where the additive noise is not white (or that the {{tmath|S/N}} is not constant with frequency over the bandwidth) is obtained by treating the channel as many narrow, independent Gaussian channels in parallel:

:<math> C = \int_{0}^B  \log_2 \left( 1+\frac{S(f)}{N(f)} \right) df  </math>

where
* <math>C</math> is the [[channel capacity]] in bits per second;
* <math>B</math> is the bandwidth of the channel in Hz;
* <math>S(f)</math> is the signal [[power spectrum]]
* <math>N(f)</math> is the noise power spectrum
* <math>f</math> is frequency in Hz.

Note: the theorem only applies to Gaussian [[stationary process]] noise. This formula's way of introducing frequency-dependent noise cannot describe all continuous-time noise processes. For example, consider a noise process consisting of adding a random wave whose amplitude is 1 or &minus;1 at any point in time, and a channel that adds such a wave to the source signal. Such a wave's frequency components are highly dependent. Though such a noise may have a high power, it is fairly easy to transmit a continuous signal with much less power than one would need if the underlying noise was a sum of independent noises in each frequency band.

==Approximations==
[[File:Channel Capacity with Power- and Bandwidth-Limited Regimes.png|thumb|[[Additive white Gaussian noise|AWGN]] channel capacity with the power-limited regime and bandwidth-limited regime indicated. Here, <math>\frac{S}{N_0} = 1</math>; ''B'' and ''C'' can be scaled proportionally for other values.]]

For large or small and constant signal-to-noise ratios, the capacity formula can be approximated:

===Bandwidth-limited case===
When the SNR is large ({{math|''S''/''N'' ≫ 1}}), the logarithm is approximated by

:<math>\log_2 \left( 1+\frac{S}{N} \right) \approx \log_2 \frac{S}{N}  = \frac{\ln 10}{\ln 2} \cdot \log_{10} \frac{S}{N}  \approx 3.32 \cdot \log_{10} \frac{S}{N} ,</math>

in which case the capacity is logarithmic in power and approximately linear in bandwidth (not quite linear, since {{mvar|N}} increases with bandwidth, imparting a logarithmic effect). This is called the '''bandwidth-limited regime'''.

:<math> C \approx 0.332 \cdot B \cdot \mathrm{SNR\ (in\ dB)} </math>
where
:<math>\mathrm{SNR\ (in \ dB)} = 10\log_{10}{S \over N}. </math>

===Power-limited case===
Similarly, when the SNR is small (if {{tmath|S/N \ll 1}}), applying the approximation to the logarithm:

:<math>\log_2 \left( 1+\frac{S}{N} \right) = \frac{1}{\ln 2} \cdot \ln \left( 1+\frac{S}{N} \right) \approx \frac{1}{\ln 2} \cdot \frac{S}{N} \approx 1.44 \cdot {S \over N};</math>

then the capacity is linear in power. This is called the '''power-limited regime'''.

:<math> C \approx 1.44 \cdot B \cdot {S \over N}.</math>

In this low-SNR approximation, capacity is independent of bandwidth if the noise is white, of [[spectral density]] <Math>N_0</math> watts per hertz, in which case the total noise power is <math>N = B \cdot N_0</math>.

:<math> C \approx 1.44  \cdot {S \over N_0}</math>

==Examples==
# At a SNR of 0&nbsp;dB (Signal power = Noise power) the Capacity in bits/s is equal to the bandwidth in hertz. 
# If the SNR is 20&nbsp;dB, and the bandwidth available is 4&nbsp;kHz, which is appropriate for telephone communications, then C = 4000 log<sub>2</sub>(1 + 100) = 4000 log<sub>2</sub> (101) = 26.63&nbsp;kbit/s. Note that the value of S/N = 100 is equivalent to the SNR of 20&nbsp;dB.
# If the requirement is to transmit at 50 kbit/s, and a bandwidth of 10&nbsp;kHz is used, then the minimum S/N required is given by 50000 = 10000 log<sub>2</sub>(1+S/N) so C/B = 5 then S/N = 2<sup>5</sup> − 1 = 31, corresponding to an SNR of 14.91&nbsp;dB (10 x log<sub>10</sub>(31)).
# What is the channel capacity for a signal having a 1&nbsp;MHz bandwidth, received with a SNR of −30&nbsp;dB ? That means a signal deeply buried in noise. −30&nbsp;dB means a S/N = 10<sup>−3</sup>. It leads to a maximal rate of information of 10<sup>6</sup> log<sub>2</sub> (1 + 10<sup>−3</sup>) = 1443 bit/s. These values are typical of the received ranging signals of the GPS, where the navigation message is sent at 50 bit/s (below the channel capacity for the given S/N), and whose bandwidth is spread to around 1&nbsp;MHz by a pseudo-noise multiplication before transmission.
# As stated above, channel capacity is proportional to the bandwidth of the channel and to the logarithm of SNR. This means channel capacity can be increased linearly either by increasing the channel's bandwidth given a fixed SNR requirement or, with fixed bandwidth, by using [[higher-order modulation]]s that need a very high SNR to operate. As the modulation rate increases, the [[spectral efficiency]] improves, but at the cost of the SNR requirement. Thus, there is an exponential rise in the SNR requirement if one adopts a [[Quadrature amplitude modulation|16QAM or 64QAM]]; however, the spectral efficiency improves.

==See also==
* [[Nyquist–Shannon sampling theorem]]
* [[Eb/N0]]

==Notes==
{{Reflist}}

==References==
{{refbegin}}
* {{cite book |first1=Herbert |last1=Taub |first2=Donald L. |last2=Schilling | title = Principles of Communication Systems | publisher = McGraw-Hill | year = 1986 |isbn=978-0-07-062956-1 |oclc=12107009}}
* {{cite book |first1=John M. |last1=Wozencraft |first2=Irwin Mark |last2=Jacobs | title = Principles of Communications Engineering | publisher = Wiley | year = 1965 |isbn=978-0-471-96240-3 |oclc=1325622  }}
{{refend}}

==External links==
* [http://www.inference.phy.cam.ac.uk/mackay/itila/ On-line textbook: Information Theory, Inference, and Learning Algorithms], by [[David MacKay (scientist)|David MacKay]] - gives an entertaining and thorough introduction to Shannon theory, including two proofs of the noisy-channel coding theorem. This text also discusses state-of-the-art methods from coding theory, such as [[low-density parity-check code]]s, and [[Turbo code]]s.
* [http://web.mit.edu/newsoffice/2010/explained-shannon-0115.html MIT News article on Shannon Limit]

{{DEFAULTSORT:Shannon-Hartley theorem}}
[[Category:Information theory]]
[[Category:Telecommunication theory]]
[[Category:Mathematical theorems in theoretical computer science]]
[[Category:Theorems in statistics]]
[[Category:Claude Shannon]]