Skip to content
Formal Sciences
Information Theory
1948
Intermediate

Shannon Entropy

H=−∑ipilog⁡2piH = -\sum_{i} p_i \log_2 p_i

Measures the average information (surprise) in a random variable—how many bits needed to encode it.

By Claude Shannon

Formal Sciences
Shannon Entropy
1948 · Claude Shannon
Expert Reviewed
89%

Rabbit Hole Mode

Five doors into the universe behind this equation. Choose your path.

Story PortalWho discovered this?Discover how Claude Shannon and others shaped this equation.Visual PortalWhat does it look like?See the equation come alive in the Visual Studio.Machine PortalWhere is it used?Explore machines powered by this equation — routers and neural networks.Math PortalWhat does it derive from?Trace the mathematical lineage from probability distributions.Future PortalWhere is it going?Quantum information theory
Launch collection — full story, visual, sound & share card
Why it matters: Defined limits of compression, communication, and laid groundwork for the internet and AI.

Discoverers: Claude Shannon (1948)

What does it mean?

Measures the average information (surprise) in a random variable—how many bits needed to encode it.

Why should I care?

Defined limits of compression, communication, and laid groundwork for the internet and AI.

Equation Compass

North — Prerequisites

West — History

East — Applications

South — Derivations

Variables & Units

SymbolNameUnitMeaning
HHEntropybitsAverage information
pip_iProbability—Probability of outcome i

Worked Example

Fair coin: H = -0.5·log₂(0.5) - 0.5·log₂(0.5) = 1 bit per flip.

AI Guide (Pro)

Ask questions about equations and get answers grounded in the Equation Universe catalog.

Upgrade to ProTry demo (7 days)

Continue your trail

Neural NetworkSmartphoneArtificial IntelligenceTelecommunications

Pictures & video

Photograph of Claude Shannon
Claude Shannon, founder of information theory (1948).Konrad Jacobs / Archives of the Mathematisches Forschungsinstitut Oberwolfach · CC BY-SA 2.0 DE

Sources & further reading

Entropy (information theory) — WikipediaSearch WikipediaSearch Wolfram MathWorldOpen in Wolfram|AlphaWatch explainers on YouTube

Share this equation

Equation Universe

Shannon Entropy

H=−∑ipilog⁡2piH = -\sum_{i} p_i \log_2 p_i

Real-world impact

Wireless technology

Electromagnetic waves carry information worldwide.

Photo: Unsplash — network infrastructure

Measures the average information (surprise) in a random variable—how many bits needed to encode it.

equation-universe.vercel.app

Post