User:Akela1101/Unsupervised Learning Algorithms

K-Means
Is a common algorithm of unsupervised ML allowing to separate unmarked samples into a set of clusters.

Let's have some set of samples: $$\{\bar{x}^j\} = \{ \bar{x}^1, \bar{x}^2, .. \bar{x}^m\}$$

$$\Box$$ $$\blacksquare$$
 * 1) Randomly initialize a set of clusters with p different samples: $$\{\bar{\mu}^k\} = \{ \bar{\mu}^1, \bar{\mu}^2, .. \bar{\mu}^p\}$$, p << m.
 * 2) For each $$j \in 1..m$$ set $$c^j = \underset{k}{argmin} ||x^j - \mu^k||$$ — index of cluster closest to $$x^j$$.
 * 3) For each $$k \in 1..p$$ set $$\mu^k = \underset{c^j=k}{avg}(x^j)$$ — mean of all $$x^j$$ close to k cluster.
 * 4) Stop when set of $$c^j$$ does not change anymore.

And so the error function: $$J = \frac{1}{m} \sum_{j = 1}^m ||x^j - \mu^{c^j}||^2$$

To avoid local minimum this algorithm should be better run several times, optimizing function J.

Self-Organizing Map (Kohonen Map)
The idea is similar to K-Means, but instead of getting mean vectors several times, clusters are built adding new samples one by one.

Also this algorithm is remarkable with mapping to plane $$r(\bar{x}) \in \Reals^2$$, which allows to visualize the result.

$$\Box$$ $$\blacksquare$$
 * 1) As in K-Means, let's have $$\{\bar{\mu}^k\} = \{ \bar{\mu}^1, \bar{\mu}^2, .. \bar{\mu}^p\}$$, p << m.
 * 2) For each $$j \in 1..m\ |\ \bar{x}^j \notin \{\bar{\mu}^k\}$$
 * 3) * $$\mu^k := \mu^k + \alpha \cdot e^{-\frac{||r(x^j) - r(\mu^k)||^2}{2\sigma^2}} \cdot (x^j - \mu^k)$$, thus gaussian function is taken as a coefficient of shifting clusters to a new sample. Closest clusters are getting more impact.
 * 4) * Reduce α and σ for converging.

Kohonen Map is considered to be more robust than K-Means, so there's no need in running it multiple times.

Neural Gas
Almost the same algorithm as Self-Organizing Map, but without $$r(\bar{x})$$ mapping.

Shifting step: $$\mu^k := \mu^k + \alpha \cdot e^{-\frac{k}{\lambda}} \cdot (x^j - \mu^k)$$, where $$\mu^k$$ are ordered by closeness to $$x^j$$, $$k \in 0..p\!-\!1$$.