KL Divergence

Assess how much information is lost when one distribution is used to approximate another distribution. It is used as a loss function in the t-SNE algorithm.

$$D_{KL}(P\parallel Q)=\sum _{x\in X}P(x)\log \frac{P(x)}{Q(x)}$$