Neural gas

Suppose we want to model a probability distribution ${\displaystyle P(x)}$  of data vectors ${\displaystyle x}$  using a finite number of feature vectors ${\displaystyle w_{i}}$ , where ${\displaystyle i=1,\cdots ,N}$ .

1. For each time step ${\displaystyle t}$ 
   1. Sample data vector ${\displaystyle x}$  from ${\displaystyle P(x)}$ 
   2. Compute the distance between ${\displaystyle x}$  and each feature vector. Rank the distances.
   3. Let ${\displaystyle i_{0}}$  be the index of the closest feature vector, ${\displaystyle i_{1}}$  the index of the second closest feature vector, and so on.
   4. Update each feature vector by:$${\displaystyle w_{i_{k}}^{t+1}=w_{i_{k}}^{t}+\varepsilon \cdot e^{-k/\lambda }\cdot (x-w_{i_{k}}^{t}),k=0,\cdots ,N-1}$$ 

In the algorithm, ${\displaystyle \varepsilon }$  can be understood as the learning rate, and ${\displaystyle \lambda }$  as the neighborhood range. ${\displaystyle \varepsilon }$  and ${\displaystyle \lambda }$  are reduced with increasing ${\displaystyle t}$  so that the algorithm converges after many adaptation steps.

The adaptation step of the neural gas can be interpreted as gradient descent on a cost function. By adapting not only the closest feature vector but all of them with a step size decreasing with increasing distance order, compared to (online) k-means clustering a much more robust convergence of the algorithm can be achieved. The neural gas model does not delete a node and also does not create new nodes.

Compared to self-organized map, the neural gas model does not assume that some vectors are neighbors. If two vectors happen to be close together, they would tend to move together, and if two vectors happen to be apart, they would tend to not move together. In contrast, in an SOM, if two vectors are neighbors in the underlying graph, then they will always tend to move together, no matter whether the two vectors happen to be neighbors in the Euclidean space.

The name "neural gas" is because one can imagine it to be what an SOM would be like if there is no underlying graph, and all points are free to move without the bonds that bind them together.

A number of variants of the neural gas algorithm exists in the literature so as to mitigate some of its shortcomings. More notable is perhaps Bernd Fritzke's growing neural gas,^[5] but also one should mention further elaborations such as the Growing When Required network^[6] and also the incremental growing neural gas.^[7] A performance-oriented approach that avoids the risk of overfitting is the Plastic Neural gas model.^[8]

Fritzke describes the growing neural gas (GNG) as an incremental network model that learns topological relations by using a "Hebb-like learning rule",^[5] only, unlike the neural gas, it has no parameters that change over time and it is capable of continuous learning, i.e. learning on data streams. GNG has been widely used in several domains,^[9] demonstrating its capabilities for clustering data incrementally. The GNG is initialized with two randomly positioned nodes which are initially connected with a zero age edge and whose errors are set to 0. Since the in the GNG input data is presented sequentially one by one, the following steps are followed at each iteration:

• It is calculated the errors (distances) between the two closest nodes to the current input data.
• The error of the winner node (only the closest one) is respectively accumulated.
• The winner node and its topological neighbors (connected by an edge) are moving towards the current input by different fractions of their respective errors.
• The age of all edges connected to the winner node are incremented.
• If the winner node and the second-winner are connected by an edge, such an edge is set to 0. Else, an edge is created between them.
• If there are edges with an age larger than a threshold, they are removed. Nodes without connections are eliminated.
• If the current iteration is an integer multiple of a predefined frequency-creation threshold, a new node is inserted between the node with the largest error (among all) and its topological neighbor presenting the highest error. The link between the former and the latter nodes is eliminated (their errors are decreased by a given factor) and the new node is connected to both of them. The error of the new node is initialized as the updated error of the node which had the largest error (among all).
• The accumulated error of all nodes is decreased by a given factor.
• If the stopping criterion is not met, the algorithm takes a following input. The criterion might be a given number of epochs, i.e., a pre-set number of times where all data is presented, or the reach of a maximum number of nodes.

Another neural gas variant inspired in the GNG algorithm is the incremental growing neural gas (IGNG). The authors propose the main advantage of this algorithm to be "learning new data (plasticity) without degrading the previously trained network and forgetting the old input data (stability)."^[7]

Having a network with a growing set of nodes, like the one implemented by the GNG algorithm was seen as a great advantage, however some limitation on the learning was seen by the introduction of the parameter λ, in which the network would only be able to grow when iterations were a multiple of this parameter.^[6] The proposal to mitigate this problem was a new algorithm, the Growing When Required network (GWR), which would have the network grow more quickly, by adding nodes as quickly as possible whenever the network identified that the existing nodes would not describe the input well enough.

The ability to only grow a network may quickly introduce overfitting; on the other hand, removing nodes on the basis of age only, as in the GNG model, does not ensure that the removed nodes are actually useless, because removal depends on a model parameter that should be carefully tuned to the "memory length" of the stream of input data.

The "Plastic Neural Gas" model^[8] solves this problem by making decisions to add or remove nodes using an unsupervised version of cross-validation, which controls an equivalent notion of "generalization ability" for the unsupervised setting.

While growing-only methods only cater for the incremental learning scenario, the ability to grow and shrink is suited to the more general streaming data problem.

To find the ranking ${\displaystyle i_{0},i_{1},\ldots ,i_{N-1}}$  of the feature vectors, the neural gas algorithm involves sorting, which is a procedure that does not lend itself easily to parallelization or implementation in analog hardware. However, implementations in both parallel software ^[10] and analog hardware^[11] were actually designed.

• T. Martinetz, S. Berkovich, and K. Schulten. "Neural-gas" Network for Vector Quantization and its Application to Time-Series Prediction. IEEE-Transactions on Neural Networks, 4(4):558–569, 1993.
• Martinetz, T.; Schulten, K. (1994). "Topology representing networks". Neural Networks. 7 (3): 507–522. doi:10.1016/0893-6080(94)90109-0.

• DemoGNG.js Javascript simulator for Neural Gas (and other network models)
• Java Competitive Learning Applications Unsupervised Neural Networks (including Self-organizing map) in Java with source codes.
• formal description of Neural gas algorithm
• A GNG and GWR Classifier implementation in Matlab