Conditional entropy API

entropy_conditional(measure::ConditionalEntropy, c::ContingencyMatrix{T, 2}) where T

Estimate the discrete version of the given ConditionalEntropy measure from its direct (sum) definition, using the probabilities from a pre-computed ContingencyMatrix, constructed from two input variables x and y. This estimation method works for both numerical and categorical data. If measure is not given, then the default is CEShannon().

The convention is to compute the entropy of the variable in the first column of c conditioned on the variable in the second column of c. To do the opposite, call this function with a new contingency matrix where the order of the variables is reversed.

Compatible measures

entropy_conditional([measure::ConditionalEntropy], est::ProbabilitiesEstimator, x, y)

Estimate the entropy of x conditioned on y, using the discrete version of the given conditional entropy (CE) measure. The CE is computed the difference of the joint entropy and the marginal entropy of y, using the ProbabilitiesEstimator est, which must compatible with multivariate data (that is, have an implementation for marginal_encodings). No bias correction is applied. If measure is not given, then the default is CEShannon().

Estimators

Joint and marginal probabilities are computed by jointly discretizing x and y using the approach given by est, and obtaining the marginal distribution for y from the joint distribution.

entropy_conditional([measure::ConditionalEntropy], est::DifferentialEntropyEstimator, x, y)

Estimate the entropy of x conditioned on y, using the differential/continuous version of the given conditional entropy (CE) measure. The CE is computed the difference of the joint entropy and the marginal entropy of y, using the DifferentialEntropyEstimator est, which must be compatible with multivariate data. No bias correction is applied. If measure is not given, then the default is CEShannon().

Estimators

The supertype for all conditional entropies.

CEShannon <: ConditionalEntropy
CEShannon(; base = 2,)

TheShannon conditional entropy measure.

Discrete definition

Sum formulation

The conditional entropy between discrete random variables $X$ and $Y$ with finite ranges $\mathcal{X}$ and $\mathcal{Y}$ is defined as

\[H^{S}(X | Y) = -\sum_{x \in \mathcal{X}, y \in \mathcal{Y}} = p(x, y) \log(p(x | y)).\]

This is the definition used when calling entropy_conditional with a ContingencyMatrix.

Two-entropies formulation

Equivalently, the following difference of entropies hold

\[H^S(X | Y) = H^S(X, Y) - H^S(Y),\]

where $H^S(\cdot$ and $H^S(\cdot | \cdot)$ are the Shannon entropy and Shannon joint entropy, respectively. This is the definition used when calling entropy_conditional with a ProbabilitiesEstimator.

Differential definition

The differential conditional Shannon entropy is analogously defined as

\[H^S(X | Y) = h^S(X, Y) - h^S(Y),\]

where $h^S(\cdot$ and $h^S(\cdot | \cdot)$ are the Shannon differential entropy and Shannon joint differential entropy, respectively. This is the definition used when calling entropy_conditional with a DifferentialEntropyEstimator.

CETsallisFuruichi <: ConditionalEntropy
CETsallisFuruichi(; base = 2, q = 1.5)

Furuichi (2006)'s discrete Tsallis conditional entropy measure.

Definition

Furuichi's Tsallis conditional entropy between discrete random variables $X$ and $Y$ with finite ranges $\mathcal{X}$ and $\mathcal{Y}$ is defined as

\[H_q^T(X | Y) = -\sum_{x \in \mathcal{X}, y \in \mathcal{Y}} p(x, y)^q \log_q(p(x | y)),\]

when $q \neq 1$. For $q = 1$, $H_q^T(X | Y)$ reduces to the Shannon conditional entropy:

\[H_{q=1}^T(X | Y) = -\sum_{x \in \mathcal{X}, y \in \mathcal{Y}} = p(x, y) \log(p(x | y))\]

CETsallisAbe <: ConditionalEntropy
CETsallisAbe(; base = 2, q = 1.5)

Abe and Rajagopal (2001)'s discrete Tsallis conditional entropy measure.

Definition

Abe & Rajagopal's Tsallis conditional entropy between discrete random variables $X$ and $Y$ with finite ranges $\mathcal{X}$ and $\mathcal{Y}$ is defined as

\[H_q^{T_A}(X | Y) = \dfrac{H_q^T(X, Y) - H_q^T(Y)}{1 + (1-q)H_q^T(Y)},\]

where $H_q^T(\cdot)$ and $H_q^T(\cdot, \cdot)$ is the Tsallis entropy and the joint Tsallis entropy.