# Information Theory

## Entropy

The calculation of the entropy of a distribution is available as a function called `entropy` as well as the traditional `H`:

Imports and implicits

``````import edu.uci.ics.jung.graph.DirectedSparseGraph

import cats.implicits._

import spire.math._
import spire.algebra._

import axle._
import axle.probability._
import axle.stats._
import axle.quanta.Information
import axle.jung.directedGraphJung
import axle.data.Coin
import axle.game.Dice.die

implicit val fieldDouble: Field[Double] = spire.implicits.DoubleAlgebra

implicit val informationConverter = Information.converterGraphK2[Double, DirectedSparseGraph]``````

Usage

Entropy of fair 6-sided die

``````val d6 = die(6)
// d6: ConditionalProbabilityTable[Int, Rational] = ConditionalProbabilityTable(
//   p = HashMap(5 -> 1/6, 1 -> 1/6, 6 -> 1/6, 2 -> 1/6, 3 -> 1/6, 4 -> 1/6)
// )

H[Int, Rational](d6).show
// res1: String = "2.5849625007211565 b"``````

Entropy of fair and biased coins

``````val fairCoin = Coin.flipModel()
// fairCoin: ConditionalProbabilityTable[Symbol, Rational] = ConditionalProbabilityTable(
//   p = Map('HEAD -> 1/2, 'TAIL -> 1/2)
// )

H[Symbol, Rational](fairCoin).show
// res2: String = "1.0 b"

val biasedCoin = Coin.flipModel(Rational(7, 10))
// biasedCoin: ConditionalProbabilityTable[Symbol, Rational] = ConditionalProbabilityTable(
//   p = Map('HEAD -> 7/10, 'TAIL -> 3/10)
// )

entropy[Symbol, Rational](biasedCoin).show
// res3: String = "0.8812908992306927 b"``````

See also the following example of the entropy of a biased coin.

## Example: Entropy of a Biased Coin

Visualize the relationship of a coin's bias to its entropy with this code snippet.

Imports and implicits:

``````import scala.collection.immutable.TreeMap
import cats.implicits._
import spire.math.Rational
import spire.algebra._
import axle.stats.H
import axle.data.Coin
import axle.quanta.UnittedQuantity
import axle.quanta.Information

type D = TreeMap[Rational, UnittedQuantity[Information, Double]]

import edu.uci.ics.jung.graph.DirectedSparseGraph
import axle.jung.directedGraphJung
import cats.kernel.Order
import axle.quanta.unittedTics

implicit val id = {
implicit val fieldDouble: Field[Double] = spire.implicits.DoubleAlgebra
Information.converterGraphK2[Double, DirectedSparseGraph]
}

implicit val or: Order[Rational] = new cats.kernel.Order[Rational] {
implicit val doubleOrder = Order.fromOrdering[Double]
def compare(x: Rational, y: Rational): Int = doubleOrder.compare(x.toDouble, y.toDouble)
}
implicit val bitDouble = id.bit``````

Create dataset

``````val hm: D =
new TreeMap[Rational, UnittedQuantity[Information, Double]]() ++
(0 to 100).map({ i =>
val r = Rational(i.toLong, 100L)
r -> H[Symbol, Rational](Coin.flipModel(r))
}).toMap``````

Define visualization

``````import axle.visualize._

implicit val fieldDouble: Field[Double] = spire.implicits.DoubleAlgebra

val plot = Plot[String, Rational, UnittedQuantity[Information, Double], D](
() => List(("h", hm)),
connect = true,
colorOf = _ => Color.black,
drawKey = false,
``````import axle.web._