(Series Part 1) Gamma Distribution Deep Dive: The Foundation of Related Distributions

Table of Contents

• Introduction
  • Probability Density Function (PDF) of Gamma distribution
• Related Distributions
  • Erlang Distribution
  • Chi-Squared Distribution
  • Exponential Distribution
  • Weibull Distribution
  • Poisson Distribution
    • Relationship with Gamma Distribution
• Examples
  • Gamma Distribution
  • Erlang Distribution
  • Poisson Distribution
    • Derivation
    • Relationship Between Poisson and Gamma Distributions
    • Derivation: Gamma as a Conjugate Prior for Poisson
• Chi-Squared Distribution
  • Goodness-of-Fit Test
• Weibull Distribution
  • Time to Failure of a Mechanical Component
• Conclusion

Introduction

The Gamma distribution is a continuous probability distribution with two parameters: the shape parameter ($\alpha$) and the rate parameter ($\beta$). Several other distributions you already use, Erlang, Chi-squared, Exponential, and (via a different derivation) Weibull and Poisson, fall out of it through specific parameter substitutions. Knowing the substitution is often faster than looking up each distribution’s PDF from scratch.

Probability Density Function (PDF) of Gamma distribution

The probability density function (PDF) of the Gamma distribution is given by:

$$f(x; \alpha, \beta) = \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha - 1} e^{-\beta x}, \quad x > 0$$

where $\Gamma(\alpha)$ is:

$$\Gamma(\alpha) = \int_0^\infty t^{\alpha - 1} e^{-t} \, dt, \quad \text{for } \alpha > 0$$

has the following properties that will be used to derive the related distributions:

1. Recursive Relation:

   $$\Gamma(\alpha + 1) = \alpha \, \Gamma(\alpha)$$

2. Special Case for Positive Integers (Used in Erlang Distribution):

   $$\Gamma(n) = (n - 1)!, \quad \text{for } n \in \mathbb{N}$$

3. Value at $\alpha = \frac{1}{2}$ (Used in Chi-Squared Distribution):

   $$\Gamma\left(\frac{1}{2}\right) = \sqrt{\pi}$$

---

Related Distributions

Erlang Distribution

The Erlang distribution is a special case of the Gamma distribution where the shape parameter ($k$) is a positive integer.

• PDF:

  $$f(x; k, \beta) = \frac{\beta^k x^{k - 1} e^{-\beta x}}{(k - 1)!}, \quad x > 0$$

• Parameters:

  • Shape parameter $k \in \mathbb{N}$
  • Rate parameter $\beta$

• Substitution:

  • Set $\alpha = k$

---

Chi-Squared Distribution

The Chi-squared distribution is another special case of the Gamma distribution, widely used in hypothesis testing and confidence interval estimation.

• PDF:

  $$f(x; k) = \frac{1}{2^{k/2} \Gamma(k/2)} x^{(k/2) - 1} e^{-x/2}, \quad x > 0$$

• Parameters:

  • Degrees of freedom $k$

• Substitution:

  • Set $\alpha = \frac{k}{2}$
  • Set $\beta = \frac{1}{2}$

---

Exponential Distribution

The Exponential distribution is a special case of the Gamma distribution with the shape parameter set to 1.

• PDF:

  $$f(x; \beta) = \beta e^{-\beta x}, \quad x > 0$$

• Parameters:

  • Rate parameter $\beta$

• Substitution:

  • Set $\alpha = 1$

---

Weibull Distribution

While not a direct parameter substitution from the Gamma distribution, the Weibull distribution is a separate continuous probability distribution used primarily in reliability engineering and failure analysis.

• PDF:

  $$f(x; \lambda, k) = \frac{k}{\lambda} \left( \frac{x}{\lambda} \right)^{k - 1} e^{-\left(\frac{x}{\lambda}\right)^k}, \quad x \geq 0$$

• Parameters:

  • Scale parameter $\lambda$
  • Shape parameter $k$

---

Poisson Distribution

The Poisson distribution is a discrete probability distribution expressing the probability of a given number of events occurring in a fixed interval of time or space.

• Probability Mass Function (PMF): $$P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!}, \quad k = 0, 1, 2, \dots$$

Relationship with Gamma Distribution

In Bayesian statistics, the Poisson and Gamma distributions are connected through the concept of conjugate priors. If the number of events follows a Poisson distribution with an unknown rate parameter $\lambda$, and $\lambda$ is assumed to follow a Gamma distribution as a prior, then the posterior distribution of $\lambda$ given the observed data remains a Gamma distribution. This conjugate prior relationship simplifies the updating of beliefs about $\lambda$ after observing data.

---

Examples

Gamma Distribution

Waiting Time for Multiple Events

Suppose the time between events follows an Exponential distribution with rate $\beta$. The time until the $\alpha$-th event occurs follows a Gamma distribution with parameters $\alpha$ (shape) and $\beta$ (rate).

---

Erlang Distribution

Telephone Call Center

The time until the $k$-th call arrives at a call center can be modeled using the Erlang distribution with shape parameter $k$ and rate parameter $\beta$.

---

Poisson Distribution

Number of Emails Received

The number of emails a person receives in an hour can be modeled by a Poisson distribution with parameter $\lambda$ representing the average number of emails per hour.

Derivation

The Poisson distribution is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time or space. It can be derived as a limit of the Binomial distribution under specific conditions.

The Binomial distribution models the number of successes in $n$ independent trials, each with probability $p$ of success:

$$P(X = k) = \binom{n}{k} p^k (1 - p)^{n - k}, \quad k = 0, 1, 2, \dots, n$$

Limit Conditions:

• Let $n \to \infty$
• Let $p \to 0$
• Such that $\lambda = np$ remains constant

Using the approximations for large $n$ and small $p$:

$$\binom{n}{k} \approx \frac{n^k}{k!}$$

and

$$(1 - p)^n \approx e^{-np} = e^{-\lambda}$$

Substituting these into the Binomial PMF:

$$P(X = k) \approx \frac{\lambda^k e^{-\lambda}}{k!}$$

This is the Poisson PMF, demonstrating that under the conditions of a large number of trials and a small probability of success per trial, the Binomial distribution converges to the Poisson distribution with parameter $\lambda = np$.

Relationship Between Poisson and Gamma Distributions

As mentioned earlier, in Bayesian statistics, the Poisson and Gamma distributions form a conjugate pair. This means that when the rate parameter $\lambda$ of a Poisson distribution is assigned a Gamma prior, the posterior distribution of $\lambda$ after observing data remains a Gamma distribution. This property facilitates the updating of beliefs about $\lambda$ in a computationally efficient manner.

Derivation: Gamma as a Conjugate Prior for Poisson

Assume the number of events $X$ follows a Poisson distribution with rate parameter $\lambda$:

$$P(X = k | \lambda) = \frac{\lambda^k e^{-\lambda}}{k!}, \quad k = 0, 1, 2, \dots$$

Suppose the prior distribution of $\lambda$ is a Gamma distribution with parameters $\alpha$ (shape) and $\beta$ (rate):

$$p(\lambda) = \frac{\beta^\alpha}{\Gamma(\alpha)} \lambda^{\alpha - 1} e^{-\beta \lambda}, \quad \lambda > 0$$

Posterior Distribution:

Using Bayes’ theorem, the posterior distribution of $\lambda$ given $X = k$ is:

$$p(\lambda | X = k) \propto P(X = k | \lambda) \cdot p(\lambda)$$

Substituting the expressions:

$$p(\lambda | X = k) \propto \frac{\lambda^k e^{-\lambda}}{k!} \cdot \frac{\beta^\alpha}{\Gamma(\alpha)} \lambda^{\alpha - 1} e^{-\beta \lambda}$$

Simplifying (constants can be absorbed into the proportionality):

$$p(\lambda | X = k) \propto \lambda^{k + \alpha - 1} e^{-(\beta + 1)\lambda}$$

This is the kernel of a Gamma distribution with updated parameters:

• Shape parameter: $\alpha' = \alpha + k$
• Rate parameter: $\beta' = \beta + 1$

Therefore, the posterior distribution is:

$$p(\lambda | X = k) \sim \Gamma(\alpha + k, \beta + 1)$$

Interpretation:

After observing $k$ events, the updated belief about the rate parameter $\lambda$ is captured by a Gamma distribution with increased shape and rate parameters. This conjugate relationship simplifies Bayesian updating, as the posterior remains in the same family as the prior.

---

Chi-Squared Distribution

The Chi-squared distribution is a special case of the Gamma distribution and is widely used in statistical hypothesis testing and confidence interval estimation.

Goodness-of-Fit Test

In a Chi-squared goodness-of-fit test, the test statistic follows a Chi-squared distribution with degrees of freedom equal to the number of categories minus one. This test assesses whether observed frequencies differ from expected frequencies under a specific hypothesis.

---

Weibull Distribution

The Weibull distribution is a continuous probability distribution used extensively in reliability engineering and failure analysis. It is characterized by its scale parameter ($\lambda$) and shape parameter ($k$).

Time to Failure of a Mechanical Component

The lifespan of a mechanical component can be modeled using the Weibull distribution. The shape parameter $k$ indicates the failure rate behavior:

• $k < 1$: Failure rate decreases over time (infant mortality).
• $k = 1$: Corresponds to the Exponential distribution (constant failure rate).
• $k > 1$: Failure rate increases over time (wear-out failures).

Example:

A component with $k = 2$ and $\lambda = 1000$ hours suggests that the failure rate increases with time, modeling wear-out behavior.

---

Conclusion

The Gamma distribution is the parent form behind Erlang, Chi-squared, Exponential, and the Poisson-Gamma conjugate pair, and Weibull shares its use case even though it needs a separate derivation. Once you can map a problem onto one of these substitutions, you get closed-form parameter updates instead of a numerical fit.

If you are putting a model like this into production, the next steps below cover the data and observability layer, and the newsletter is where the production playbook for generative and statistical models ships first.

---