Pages

Wednesday, December 14, 2011

Hopfield Neural Network

The Hopfield 4 Neurons Single Layer Neural Network
One of the simplest Neural Networks out there. Unlike other more complex Neural Nets, HNN employes a single network layer which is the input and output layer at the same time. This implementation of HNN contains four neurons acting together to recognize any four-digit binary pattern.

When the network is first initialized, it needs to be trained to recognize a specific binary pattern. Depending on the pattern we choose, the system will build what is called a weight matrix - during the training -. This weight matrix will help recognize the specific pattern we chose. When we feed the system with a new pattern and run the network, it will run the new pattern against the weight matrix and identify the pattern if it was the same original pattern or its inverse. Moreover, if the pattern was closely similar it will try to auto-correct the entered pattern. This is the auto-associative property of the Hopfield NN.

To construct the Hopfield NN we must first understand the mathematics behind it. Most of the math behind HNN is simple matrix operations and linear algebra. To further understand HNN I will take you through the process step by step.

This simple Hopfield network consists of four neurons and twelve connections constituting a one layer neural network. You can think of the network as a bidirectional graph of nodes. Each node has three outbound connections and three inbound connections as shown in the connections matrix below. Note that an idividual node (neuron) can't be connected to itself.

Furthermore, both the neurons and the connections between them are said to be a layer of the network. The Hopfield NN is categorized as a one layer NN. Meaning, the entire neural network is a single layer which is the input and output layer at the same time. We simply feed the patterns to this layer of neurons, and the neurons will decide wither to fire back output or not depending on the weight matrix.

Connections Matrix
Let us assume that the input pattern we wish the network to learn is (0,1,0,1). By default, the Hopfield network will learn this pattern and its inverse (1,0,1,0). O.K so the first step now is to teach the network how to recognize this pattern. We do this by constructing the weight matrix. Think of the weight matrix as the means for the network to understand, and the mechanism for it to recognize the pattern.

Constructing the weight matrix is a simple three-step process:
  1. Convert the original input pattern to its bipolar equivalent pattern.
  2. Multiply the bipolar pattern with its transpose vector to construct the contribution matrix.
  3. Set the Northwest diagonal to zeros.
Converting the original input to its bipolar equivalent is nothing more than to represent the same patterns in (-1) and (1) because when dealing with binary digits (0) is not the inverse of (1). Rather (-1) is the inverse of (1). Thus we change all (0) in the input to (-1) as in the right figure. The bipolar equivalent now is (-1,1,-1,1).
Multiplying the bipolar vector with its transpose vector is a simple matrix operation. The resulting 4x4 matrix is nothing but the first row of the bipolar multiplied by the first column then the second column then the third and finally the fourth. Then the second row multiplied by the first column then second and so on. Finally we set the northwest diagonal to zeros to come up with the final weight matrix. This is the matrix that will do all the pattern recognition.


Each row in the weight matrix will be the weight vector of each neuron respectively. So neuron one will have a weight vector of (0,-1,1,-1) and neuron two's weight vector will be (-1,0,-1,1) and so on. The weight vector of each neuron will help decide wither the neuron should fire or not with the assistance of a threshold function. 

O.K so now we assume that the network has been trained to recognize (0,1,0,1) and during the training it constructed the weight matrix shown above. Now I will demonstrate how the system will recognize the new input (0,1,0,1). First digit (0) will be feed to neuron one and the second digit (1) will be feed to neuron two and so on. Each neuron will sum up digits from its weight vector where the input pattern is equal (1). The following should demonstrate this step for neuron one.


The first row is the input we want to recognize, and the second row is the weight vector of neuron one. We will only sum up the digits from the weight vector where the input patter is equal (1). So we should get (-1 + -1 = -2). For all the neurons we should get the following.


These values are said to be the act of each neuron. Acts by themselves don't determine if the neuron should fire or not. To decide that we should pass these values individually to a threshold function that will determine if the neuron should fire or not. Many neural network books discuss various threshold functions. For our purposes we will use the simple hyperbolic tangent function as our threshold function.


So now simply we substitute the act of each neuron in the above equation as (x) and come up with the hyperbolic tangent. If the tangent is greater than or equal to (0), the neuron fires. If the neuron fires then the equivalent output should be (1); otherwise, the equivalent output is (0). 

That's it. Now you fully understand the Hopfield Neural Network. My upcoming exercise is to make this HNN dynamic by trying to determine the size of the network during run-time. To test the network yourself you can download my Java source code and compile the files yourself. All thanks goes to Jeff Heaton for his tremendous efforts in simplifying Neural Nets.


2 comments :

  1. This comment has been removed by the author.

    ReplyDelete
  2. this is great to understand the Hopfield NN, thank you!

    ReplyDelete