Entropies.jl

This package provides probability and entropy estimators used for entropy computations in the CausalityTools.jl and DynamicalSystems.jl packages.

Most of the code in this package assumes that your data is represented by the Dataset-type from DelayEmbeddings.jl, where each observation is a D-dimensional data point represented by a static vector. See the DynamicalSystems.jl documentation for more info. Univariate timeseries given as AbstractVector{<:Real} also work with some estimators, but are treated differently based on which method for probability/entropy estimation is applied.

API

The main API of this package is contained in two functions:

probabilities which computes probability distributions of given datasets
genentropy which uses the output of probabilities, or a set of pre-computed Probabilities, to calculate entropies.

These functions dispatch on subtypes of ProbabilitiesEstimator, which are:

using Entropies, InteractiveUtils
subtypes(ProbabilitiesEstimator)

5-element Array{Any,1}:
 CountOccurrences
 Entropies.BinningProbabilitiesEstimator
 Entropies.CountingBasedProbabilityEstimator
 Entropies.WaveletProbabilitiesEstimator
 SymbolicProbabilityEstimator

Probabilities

Entropies.Probabilities — Type

Probabilities(x) → p

A simple wrapper type around an x::AbstractVector which ensures that p sums to 1. Behaves identically to Vector.

source

Entropies.probabilities — Function

probabilities(x::Vector_or_Dataset, est::ProbabilitiesEstimator) → p::Probabilities

Calculate probabilities representing x based on the provided estimator and return them as a Probabilities container (Vector-like). The probabilities are typically unordered and may or may not contain 0s, see the documentation of the individual estimators for more.

The configuration options are always given as arguments to the chosen estimator.

probabilities(x::Vector_or_Dataset, ε::AbstractFloat) → p::Probabilities

Convenience syntax which provides probabilities for x based on rectangular binning (i.e. performing a histogram). In short, the state space is divided into boxes of length ε, and formally we use est = VisitationFrequency(RectangularBinning(ε)) as an estimator, see VisitationFrequency.

This method has a linearithmic time complexity (n log(n) for n = length(x)) and a linear space complexity (l for l = dimension(x)). This allows computation of probabilities (histograms) of high-dimensional datasets and with small box sizes ε without memory overflow and with maximum performance. To obtain the bin information along with p, use binhist.

probabilities(x::Vector_or_Dataset) → p::Probabilities

Directly count probabilities from the elements of x without any discretization, binning, or other processing (mostly useful when x contains categorical or integer data).

source

Entropies.probabilities! — Function

probabilities!(args...)

Identical to probabilities(args...), but allows pre-allocation of temporarily used containers.

Only works for certain estimators. See for example SymbolicPermutation.

source

Entropies.ProbabilitiesEstimator — Type

An abstract type for probabilities estimators.

source

Generalized entropy

Entropies.genentropy — Function

genentropy(p::Probabilities; α = 1.0, base = Base.MathConstants.e)

Compute the generalized order-α entropy of some probabilities returned by the probabilities function. Alternatively, compute entropy from pre-computed Probabilities.

Description

Let $p$ be an array of probabilities (summing to 1). Then the generalized (Rényi) entropy is

\[H_\alpha(p) = \frac{1}{1-\alpha} \log \left(\sum_i p[i]^\alpha\right)\]

and generalizes other known entropies, like e.g. the information entropy ($\alpha = 1$, see ^{[Shannon1948]}), the maximum entropy ($\alpha=0$, also known as Hartley entropy), or the correlation entropy ($\alpha = 2$, also known as collision entropy).

genentropy(x::Vector_or_Dataset, est; α = 1.0, base)

A convenience syntax, which calls first probabilities(x, est) and then calculates the entropy of the result (and thus est can be a ProbabilitiesEstimator or simple ε::Real).

source

Fast histograms

Entropies.binhist — Function

binhist(x::Dataset, ε::Real) → p, bins
binhist(x::Dataset, ε::RectangularBinning) → p, bins

Hyper-optimized histogram calculation for x with rectangular binning ε. Returns the probabilities p of each bin of the histogram as well as the bins. Notice that bins are the starting corners of each bin. If ε isa Real, then the actual bin size is ε across each dimension. If ε isa RectangularBinning, then the bin size for each dimension will depend on the binning scheme.