Derivation of Fisher matrix from KL distance

• Definition of Kullback-Leibler divergence(KL Distance): The

• $D(P,Q)={\int }_{\Omega }P(\omega )\log \frac{P(\omega )}{Q(\omega )}d\omega$

• When symmetrized:.

• $D^{\prime }(P,Q)\equiv D(p,Q)+D(Q,P)$

• $={\int }_{\Omega }P(\omega )\log \frac{P(\omega )}{Q(\omega )}+Q(\omega )\log \frac{Q(\omega )}{P(\omega )}d\omega$

• $={\int }_{\Omega }(P(\omega )-Q(\omega ))\log \frac{P(\omega )}{Q(\omega )}d\omega$

• KL distance of the probability distribution of a parameter θ and a parameter slightly (δ)-displaced:.

• $D^{\prime }(P(\omega |\theta +\delta ),P(\omega |\theta )$

• $={\int }_{\Omega }(P(\omega |\theta +\delta )-P(\omega |\theta ))\log \frac{P(\omega |\theta +\delta )}{P(\omega |\theta )}d\omega$

• Now put $\Delta \equiv P(\omega |\theta +\delta )-P(\omega |\theta )$.

• $={\int }_{\Omega }\Delta \log \frac{P(\omega |\theta )+\Delta }{P(\omega |\theta )}d\omega$

• $={\int }_{\Omega }\Delta \log \left(1+\frac{\Delta }{P(\omega |\theta )}\right)d\omega$

• By Taylor expansion of the logarithm and a first-order approximation

• $\log (1+x)\approx x$

• therefore

  • ${\int }_{\Omega }\Delta \log \left(1+\frac{\Delta }{P(\omega |\theta )}\right)d\omega \approx {\int }_{\Omega }\frac{{\Delta }^2}{P(\omega |\theta )}d\omega$

• First-order approximation of the difference in function values:.

• $f(x+dx)-f(x)\approx f^{\prime }(x)dx$

• due to

• $\Delta \equiv P(\omega |\theta +\delta )-P(\omega |\theta )\approx {\sum }_i\frac{\partial P(\omega |\theta )}{\partial {\theta }_i}{\delta }_i$

• By the derivative of the logarithm [$ (\log x)’ = x’ / x

• $=P(\omega |\theta ){\sum }_i\frac{\partial \log P(\omega |\theta )}{\partial {\theta }_i}{\delta }_i$

• Substituting Δ:.

  • ${\int }_{\Omega }\frac{{\Delta }^2}{P(\omega |\theta )}d\omega \approx {\int }_{\Omega }{\left({\sum }_i\frac{\partial \log P(\omega |\theta )}{\partial {\theta }_i}{\delta }_i\right)}^{2}P(\omega |\theta )d\omega$

To organize:.

• ${\int }_{\Omega }{\left({\sum }_i\frac{\partial \log P(\omega |\theta )}{\partial {\theta }_i}{\delta }_i\right)}^{2}P(\omega |\theta )d\omega$

• $=\mathbb{E}\left[{\left({\sum }_i\frac{\partial \log P(\omega |\theta )}{\partial {\theta }_i}{\delta }_i\right)}^2\right]$

• $={\sum }_i{\sum }_j\mathbb{E}{\left[\frac{\partial \log P(\omega |\theta )}{\partial {\theta }_i}\frac{\partial \log P(\omega |\theta )}{\partial {\theta }_j}\right]}_{ij}{\delta }_i{\delta }_j$

• $={\sum }_i{\sum }_j\mathcal{F}(\theta {)}_{ij}{\delta }_i{\delta }_j$

• To summarize so far

  • $D^{\prime }(P(\omega |\theta +\delta ),P(\omega |\theta )\approx {\sum }_i{\sum }_j\mathcal{F}(\theta {)}_{ij}{\delta }_i{\delta }_j$

• That is, the metric tensor (physics) of the space containing the KL distance is the Fisher matrix (

---

This page is auto-translated from /nishio/KL距離からのFisher行列の導出 using DeepL. If you looks something interesting but the auto-translated English is not good enough to understand it, feel free to let me know at @nishio_en. I’m very happy to spread my thought to non-Japanese readers.