Skip to content
All paths
Advanced5 equations~10 hours

AI and Information Theory

Entropy, optimization, and the math behind intelligence.

0 of 5 equations completed0%
  1. 1

    Shannon Entropy

    Intermediate
    H=−∑ipilog⁡2piH = -\sum_{i} p_i \log_2 p_i

    Measures the average information (surprise) in a random variable—how many bits needed to encode it.

    Formal Sciences
    Information Theory
    View
    Open equationCreate art
  2. 2

    Gradient Descent Update Rule

    Intermediate
    θt+1=θt−η∇θL(θt)\theta_{t+1} = \theta_t - \eta \nabla_\theta L(\theta_t)

    Update parameters by stepping opposite to the gradient of the loss—learning by hill descent.

    Information Sciences
    Machine Learning
    View
    Open equationCreate art
  3. 3

    Cross-Entropy Loss

    Intermediate
    L=−∑iyilog⁡(y^i)L = -\sum_{i} y_i \log(\hat{y}_i)

    Measures difference between true labels and predicted probabilities in classification.

    Information Sciences
    Machine Learning
    View
    Open equation
  4. 4

    Transformer Attention

    Advanced
    Attention(Q,K,V)=softmax(QKTdk)V\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

    Each token attends to all others—weighted by query-key similarity, scaled by dimension.

    Information Sciences
    Deep Learning
    View
    Open equationCreate art
  5. 5

    PageRank Equation

    Advanced
    PR(pi)=1−dN+d∑pj∈M(pi)PR(pj)L(pj)PR(p_i) = \frac{1-d}{N} + d \sum_{p_j \in M(p_i)} \frac{PR(p_j)}{L(p_j)}

    Ranks web pages by importance based on the quality and quantity of links pointing to them.

    Formal Sciences
    Computer Science
    View
    Open equation