Skip to main content

Derivative


Loss function​

Mean Square Error​

Loss=12∑i(yi−aiL)2Loss = \dfrac{ 1 }{ 2 } \sum_i \left( y_i - a^L_i \right)^2
∂Loss∂aiL=−(yi−aiL)\dfrac{ \partial Loss }{ \partial a^L_i } = - \left( y_i - a^L_i \right)

Binary Cross-Entropy​

Loss=∑i[−yilog⁡aiL−(1−yi)log⁡(1−aiL)]Loss = \sum_i \left[ -y_i \log a^L_i - (1 - y_i) \log \left( 1 - a^L_i \right) \right]
∂Loss∂aiL=−(yi−aiL)aiL(1−aiL)\dfrac{\partial Loss}{\partial a^L_i} = \dfrac{ - \left( y_i - a^L_i \right)}{a^L_i \left( 1 - a^L_i \right) }

Activation​

logistic​

ail=σ(zil)=logistic(zil)=11+e−zila^l_i = \sigma ( z^l_i ) = logistic ( z^l_i ) = \dfrac{ 1 }{ 1 + e^{ -z^l_i } }
∂σ(zil)∂zil=e−zil(1+e−zil)2=ail(1−ail)\dfrac{\partial \sigma ( z^l_i )}{\partial z^l_i } = \dfrac{ e^{ -z^l_i } }{ \left( 1 + e^{ -z^l_i } \right)^2 } = a^l_i \left( 1 - a^l_i \right)