Home
Blog
Music
Photo

Conditional Entropy 유도

Tags:

2023.09.24

  정보이론 기초부분의 Conditional Entropy 부분에 대한 증명을 보던 중 1학기 확통 시간에 배웠던 Law of total expectation이 떠올라서 유도를 했다.

H(Y∣X)=EX[H(Y∣x)]=EX[EY(I(y∣x))]=∑x∈X[p(x)(EY[I(y∣x)])]=∑x∈X[p(x)∑y∈Y−p(y∣x)log(p(y∣x))]=∑x∈X[p(x)∑y∈Y−p(y∣x)(log(p(x,y))−log(p(x)))]=∑x∈X[p(x){∑y∈Y−p(y∣x)log(p(x,y))+∑y∈Yp(y∣x)log(p(x))}]=∑x∈X[p(x){∑y∈Y−p(x,y)p(x)log(p(x,y))+log(p(x))∑y∈Yp(y∣x)}]=∑x∈X[−∑y∈Yp(x,y)log(p(x,y))]−∑x∈X[−p(x)log(p(x))]=H(X,Y)−H(Y)\begin{aligned} H(Y|X) & = E_{X}[H(Y|x)] =E_{X}[E_{Y}(I(y|x))] \\ & = \sum_{x\in X}^{}[p(x)(E_{Y}[I(y|x)])] \\ & = \sum_{x\in X}^{}[p(x)\sum_{y\in Y}^{}-p(y|x)log(p(y|x))] \\ & = \sum_{x\in X}^{}[p(x)\sum_{y\in Y}^{}-p(y|x)(log(p(x,y))-log(p(x)))] \\ & = \sum_{x\in X}^{}[p(x)\left\{ \sum_{y\in Y}^{}-p(y|x)log(p(x,y))+\sum_{y\in Y}^{}p(y|x)log(p(x)) \right\}] \\ & =\sum_{x\in X}^{}[p(x)\left\{ \sum_{y\in Y}^{}-\frac{p(x,y)}{p(x)}log(p(x,y))+log(p(x))\sum_{y\in Y}^{}p(y|x) \right\} ] \\ & = \sum_{x\in X}^{}[-\sum_{y\in Y}^{}p(x,y)log(p(x,y)) ] - \sum_{x\in X}^{}[-p(x)log(p(x)) ] \\ & = H(X,Y) - H(Y) \end{aligned}