Machine learning notes : Sigmoid function, softmax function, and exponential family


The sigmoid function and softmax function are commonly used in the field of machine learning. And they are like “least square error” in linear regression. They can be derived from certain basic assumptions using the general form of Exponential family. Some of the basic linear regression and classification algorithms can also be derived from the general form. Let’s dig deep and see how we obtain the myterious functions. Exponential family Exponential family includes the Gaussian, binomial, multinomial, Poisson, Gamma and many others distributions.

Read More

Machine learning notes : Least square error in linear regression

Least sqaure error is used as a cost function in linear regression. However, why should one choose sqaure error, instead of absolute error, or other choices? There’s a simple proof that can show that least sqaure error is a reasonable and natural choice. Assume the target variable and inputs are related as below: \(y^i=\sigma^Tx^i+\epsilon^i\) ,where \(\epsilon\sim\mathcal{N}(\mu,\,\sigma^{2})\) i.e. \(p(\epsilon^i)=\frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(\epsilon^i)^2}{2\sigma^2})\) implies that \(p(y^i|x^i_j\theta)=\frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(y^i-\theta^Tx)^2}{2\sigma^2})\) We would like to minimize the error by maximising the log likelihood.

Read More

first post

Hello world! It’s my first time writing articles online. Thanks for the help of my friend Carson Ip, who ported the library from hexo to hugo. So that I can use the framework implemented in golang, with my favourite theme minos, which is implemented in another framework called hexo. If you like this theme but want to use hugo, you may want to check out the GitHub repo: https://github.com/carsonip/hugo-theme-minos. Also, thanks for my friend Sunny, who encouraged me to start this blog.

Read More