Entropies and Dimensions

Entropies

In the study of dynamical systems there are many quantities that identify as "entropy". Notice that these quantities are not the more commonly known thermodynamic ones, used in Statistical Physics. Rather, they are more like the to the entropies of information theory, which represents information contained within a dataset, or information about the dimensional scaling of a dataset.

Generalized Entropy

#ChaosTools.genentropy — Function.

genentropy(α, ε, dataset::AbstractDataset)

Compute the α order generalized (Rényi) entropy [1] of a dataset, by first partitioning it into boxes of length ε using non0hist.

genentropy(α, p::AbstractArray)

Compute the entropy of an array p directly, assuming that p is sum-normalized.

log base-e is used in both cases, i.e. units of "nat".

Description

The Rényi entropy

generalizes other known entropies, like e.g. the information entropy ($\alpha = 1$, see [2]), the maximum entropy ($\alpha=0$, also known as Hartley entropy), or the correlation entropy ($\alpha = 2$, also known as collision entropy).

The following aliases are provided:

• renyi = genentropy
• shannon(args...) = genentropy(1, args...)
• hartley(args...) = genentropy(0, args...)

References

[1] : A. Rényi, Proceedings of the fourth Berkeley Symposium on Mathematics, Statistics and Probability, pp 547 (1960)

[2] : C. E. Shannon, Bell Systems Technical Journal 27, pp 379 (1948)

source

Basically, given a dataset you can partition it into boxes to calculate an entropy.

The function used internally by genentropy is non0hist:

#ChaosTools.non0hist — Function.

non0hist(ε, dataset::AbstractDataset)

Partition a dataset into tabulated intervals (boxes) of size ε and return the sum-normalized histogram in an unordered 1D form, discarding all zero elements.

Performances Notes

This method is effecient in both memory and speed, because it uses a dictionary to collect the information of bins with elements, while it completely disregards empty bins. This allows computation of entropies of high-dimensional datasets and with small box sizes ε without memory overflow.

Use e.g. fit(Histogram, ...) from StatsBase if you wish to keep information about the edges of the binning as well as the zero elements.

source

For example, the Shannon entropy of a coin-flip process should be one bit, by definition. Let's see...

using DynamicalSystems
y = Float64.(rand(Bool, 1000000)) # just some coin tosses
sh = shannon(0.1, y)  # ≡ genentropy(1, 0.0, y)
isapprox(sh, log(2),  rtol = 1e-6)

true

Because all entropies are calculated on base-$e$, the unit of measurement is "nat" and one bit is $\log(2)\times$nat.

Attractor Dimension Estimation

There are numerous methods that one can use to calculate a so-called "dimension" of a dataset, like for example the Fractal dimension. This real number can offer a lot of information about the object that the dataset represents.

Generalized Dimensions & Co.

Based on the definition of the generalized entropy, one can calculate an appropriate dimension, called generalized dimension:

#ChaosTools.generalized_dim — Function.

generalized_dim(α, dataset::AbstractDataset [, sizes]) -> D_α

Return the α order generalized dimension of the dataset, by calculating the genentropy for each ε ∈ sizes.

Description

The returned dimension is approximated by the (inverse) power law exponent of the scaling of the genentropy versus the box size ε, where ε ∈ sizes.

Calling this function performs a lot of automated steps:

1. A vector of box sizes is decided by calling sizes = estimate_boxsizes(dataset), if sizes is not given.
2. For each element of sizes the appropriate entropy is calculated, through d[i] = genentropy(α, sizes[i], dataset). Let x = -log.(sizes).
3. The curve d(x) is decomposed into linear regions, using linear_regions(x, d).
4. The biggest linear region is chosen, and a fit for the slope of that region is performed using the function linear_region.
5. This fitted slope is returned.

By doing these steps one by one yourself, you can adjust the keyword arguments given to each of these function calls, refining the accuracy of the result.

The following aliases are provided:

• α = 0 : boxcounting_dim, capacity_dim
• α = 1 : information_dim
• α = 2 : correlation_dim

source

As stated clearly, this call performs a lot of automated steps by calling the following functions with default arguments:

#ChaosTools.estimate_boxsizes — Function.

estimate_boxsizes(dataset::AbstractDataset; k::Int = 12, z = 0, w = 1)

Return a k-element logspace from the magnitude + z of the biggest absolute value of the dataset, to the magnitude + w of the minimum pair-wise distance between datapoints.

source

#ChaosTools.linear_regions — Function.

linear_regions(x, y; dxi::Int = 1, tol = 0.2) -> (lrs, tangents)

Identify regions where the curve y(x) is linear, by scanning the x-axis every dxi indices (e.g. at x[1] to x[5], x[5] to x[10], x[10] to x[15] and so on if dxi=5).

If the slope (calculated using LsqFit) of a region of width dxi is approximatelly equal to that of the previous region, within tolerance tol, then these two regions belong to the same linear region.

Return the indices of x that correspond to linear regions, lrs, and the approximated tangents at each region. lrs is a vector of Int.

A function plot_linear_regions visualizes the result of using this linear_regions (requires PyPlot).

source

#ChaosTools.linear_region — Function.

linear_region(x, y; dxi::Int = 1, tol = 0.2) -> ([ind1, ind2], slope)

Call linear_regions, identify the largest linear region and approximate the slope of the entire region using least squares fit. Return the indices where the region starts and stops (x[ind1:ind2]) as well as the approximated slope.

source

Example

For example, we will calculate the dimensions of the strange attractors of the Hénon map and the Lorenz system:

using DynamicalSystems
hen = Systems.henon(-rand(2))
ts = trajectory(hen, 1000000)
D_hen = information_dim(ts)

lor = Systems.lorenz(rand(3))
ts = trajectory(lor, 5000, dt = 0.05)
D_lor = capacity_dim(ts)

You will find that D_hen is around 1.2 and D_lor is around 1.9. As a side note, be sure that you have enough data points, otherwise the values you will get will never be correct, as is demonstrated by J.-P. Eckmann and D. Ruelle (see Physica D 56, pp 185-187 (1992)).

Kaplan-Yorke Dimension

Also known as Lyapunov Dimension:

#ChaosTools.kaplanyorke_dim — Function.

kaplanyorke_dim(lyapunovs::AbstractVector)

Calculate the Kaplan-Yorke dimension, a.k.a. Lyapunov dimension [1].

Description

The Kaplan-Yorke dimension is simply the point where cumsum(lyapunovs) becomes zero (interpolated). If the sum of the exponents never becomes negative the function will return the length of the input vector.

Useful in combination with lyapunovs.

References

[1] : J. Kaplan & J. Yorke, Chaotic behavior of multidimensional difference equations, Lecture Notes in Mathematics vol. 730, Springer (1979)

source

Notice that calling this function requires you to pass the Lyapunov exponents in an ordered vector form (largest to smallest). Example:

using DynamicalSystems
hen = Systems.henon()
D_kp = kaplanyorke_dim(lyapunovs(hen, 100000))

1.258709281073041