Neural Network (Part 6) : Back Propagation, a worked example

A worked example of a Back-propagation training cycle.

In this example we will create a 2 layer network (as seen above), to accept 2 readings, and produce 2 outputs. The readings are (0,1) and the expectedOutputs in this example are (1,0).

Step 1: Create the network

NeuralNetwork NN = new NeuralNetwork();   

NN.addLayer(2,2);

NN.addLayer(2,2);

float[] readings = {0,1};

float[] expectedOutputs = {1,0};

NN.trainNetwork(readings,expectedOutputs);

This neural network will have randomised weights and biases when created.

Let us assume that the network generates the following random variables:

LAYER1.Neuron1

Layer1.Neuron1.Connection1.weight = cW111 = 0.3

Layer1.Neuron1.Connection2.weight = cW112 = 0.8

Layer1.Neuron1.Bias = bW11 = 0.5

LAYER1.Neuron2

Layer1.Neuron2.Connection1.weight = cW121 =  0.1

Layer1.Neuron2.Connection2.weight = cW122 =  0.1

Layer1.Neuron2.Bias = bW12 = 0.2

LAYER2.Neuron1

Layer2.Neuron1.Connection1.weight = cW211 = 0.6

Layer2.Neuron1.Connection2.weight = cW212 = 0.4

Layer2.Neuron1.Bias = bW21 = 0.4

LAYER2.Neuron2

Layer2.Neuron2.Connection1.weight = cW221 = 0.9

Layer2.Neuron2.Connection2.weight = cW222 = 0.9

Layer2.Neuron2.Bias = bW22 = 0.5

Step 2: Process the Readings through the Neural Network

a) Provide the Readings to the first layer, and calculate the neuron outputs

The readings provided to the neural network is (0,1), which go straight through to the first layer (layer1).

Starting with Layer 1:

Layer1.INPUT1 = 0

Layer1.INPUT2 =1

   Calculate Layer1.Neuron1.NeuronOutput

   ConnExit (cEx111) = ConnEntry (cEn111)  x Weight (cW111) = 0 x 0.3 = 0;

   ConnExit (cEx112) = ConnEntry (cEn112)  x Weight (cW112) = 1 x 0.8 = 0.8;

   Bias (bEx11) = ConnEntry (1) x Weight (bW11) = 1 x 0.4 = 0.4

   NeuronInputValue11 = 0 + 0.8 + 0.4 = 1.2

   NeuronOutputValue11 = 1/(1+EXP(-1 x 1.2)) = 0.768525

  

  Calculate Layer1.Neuron2.NeuronOutput

   ConnExit (cEx121) = ConnEntry (cEn121)  x Weight (cW121) = 0 x 0.1 = 0;

   ConnExit (cEx122) = ConnEntry (cEn122)  x Weight (cW122) = 1 x 0.1 = 0.1;

   Bias (bEx12) = ConnEntry (1) x Weight (bW12) = 1 x 0.2 = 0.2

   NeuronInputValue12 = 0 + 0.1 + 0.2 = 0.3

   NeuronOutputValue12 = 1/(1+EXP(-1 x 0.3)) = 0.574443

b) Provide LAYER2 with Layer 1 Outputs.

Now lets move to  Layer 2:

Layer2.INPUT1 = NeuronOutputValue11 = 0.768525

Layer2.INPUT2 = NeuronOutputValue12 = 0.574443

   Calculate Layer2.Neuron1.NeuronOutput

   ConnExit (cEx211) = (cEn211)  x Weight (cW211) = 0.768525 x 0.6 = 0.461115;

   ConnExit (cEx212) = (cEn212)  x Weight (cW212) = 0.574443 x 0.4 = 0.229777;

   Bias (bEx21) = ConnEntry (1) x Weight (bW21) = 1 x 0.4 = 0.4

   NeuronInputValue21 = 0.461115 + 0.229777 + 0.4 = 1.090892

   NeuronOutputValue21 = 1/(1+EXP(-1 x 1.090892)) = 0.74855

  

  Calculate Layer2.Neuron2.NeuronOutput

   ConnExit (cEx221) = (cEn221)  x Weight (cW221) = 0.768525  x 0.1 = 0.076853;

   ConnExit (cEx222) = (cEn222)  x Weight (cW222) = 0.574443  x 0.1 = 0.057444;

   Bias(bEx22) = ConnEntry (1) x Weight (bW22) = 1 x 0.5 = 0.5

   NeuronInputValue22 = 0.076853 + 0.057444 + 0.5 = 0.634297  

   NeuronOutputValue22 = 1/(1+EXP(-1 x 0.634297)) = 0.653463

Step 3) Calculate the delta error for neurons in layer 2

     -Because layer 2 is the last layer in this neural network -

      we will use the expected output data (1,0) to calculate the delta error.

   

LAYER2.Neuron1:

Let Layer2.ExpectedOutput1 = eO21 = 1    

      Layer2.ActualOutput1= aO21 = NeuronOutputValue21= 0.74855         

      Layer2.Neuron1.deltaError1 = dE21

dE21 =     aO21       x      (1 - aO21)     x  (eO21 - aO21)

       =  (0.74855)  x  (1 - 0.74855)  x  (1 - 0.74855)

        = (0.74855)  x     (0.25145)     x    (0.25145)

        = 0.047329

LAYER2.Neuron2:

Let Layer2.ExpectedOutput2 = eO22 = 0         

      Layer2.ActualOutput2     = aO22 = NeuronOutputValue22 = 0.653463      

      Layer2.Neuron2.deltaError = dE22

dE22  =      aO22       x      (1 - aO22)       x  (eO22 - aO22)

        = (0.653463)  x  (1 - 0.653463)  x  (0 - 0.653463)

        = (0.653463)  x     (0.346537)     x    (-0.653463)

        = -0.14797

Step 4) Calculate the delta error for neurons in layer 1

LAYER1.Neuron1 delta Error calculation

Let              Layer1.Neuron1.deltaError  = dE11 

                            Layer1.actualOutput1  = aO11 = NeuronOutputValue11 =  0.768525

      Layer2.Neuron1.Connection1.weight = cW211   =  0.6

                     Layer2.Neuron1.deltaError = dE21 =  0.047329

      Layer2.Neuron2.Connection1.weight = cW221   =  0.9

                     Layer2.Neuron2.deltaError = dE22 = -0.14797

dE11 = (aO11)          x  (1 -   aO11)         x ( [cW211   x   dE21]      +   [cW221  x    dE22] )

           = (0.768525) x   (1 - 0.768525)     x   ([0.6        x  0.047329]  +   [  0.9      x  -0.14797]  )

           = -0.01864

LAYER1.Neuron2 delta Error calculation

Let              Layer1.Neuron2.deltaError  = dE12 

                            Layer1.actualOutput2  = aO12    = NeuronOutputValue12 =  0.574443

      Layer2.Neuron1.Connection2.weight = cW212   =  0.4

                     Layer2.Neuron1.deltaError = dE21 =  0.047329

      Layer2.Neuron2.Connection2.weight = cW222   =  0.9

                     Layer2.Neuron2.deltaError = dE22 = -0.14797

dE12  = (aO12)          x  (1 -   aO12)         x ( [cW212  x     dE21]      +   [cW222  x    dE22] )

           = (0.574443) x   (1 - 0.574443)  x     ([0.4      x  0.047329]  +      [  0.9      x  -0.14797]  )

           = -0.02793

Step 5) Update Layer_2 neuron connection weights and bias (with a learning rate (LR) = 0.1)

Layer 2, Neuron 1 calculations:

Let

Layer2.Neuron1.Connection1.New_weight = New_cW211

Layer2.Neuron1.Connection1.Old_weight   =   Old_cW211   = 0.6

Layer2.Neuron1.Connection1.connEntry =                 cEn211 = 0.768525

Layer2.Neuron1.deltaError =                                       dE21 = 0.047329

New_cW211 = Old_cW211 + (LR x cEn211 x dE21)

                     =    0.6            + (0.1 x 0.768525 x 0.047329)

                     =    0.6            + ( 0.003627)

                     =    0.603627

Layer2.Neuron1.Connection2.New_weight = New_cW212

Layer2.Neuron1.Connection2.Old_weight   =   Old_cW212 = 0.4

Layer2.Neuron1.Connection2.connEntry =                cEn212 = 0.574443

Layer2.Neuron1.deltaError =                                      dE21 = 0.047329

New_cW212 = Old_cW212 + (LR x cEn212 x dE21)

                     =    0.4            + (0.1 x 0.574443 x 0.047329)

                     =    0.4            + (0.002719)

                     =    0.402719

Layer2.Neuron1.New_Bias = New_Bias21

Layer2.Neuron1.Old_Bias =    Old_Bias21 = 0.4

Layer2.Neuron1.deltaError =             dE21 = 0.047329

New_Bias21 = Old_Bias21 + (LR x  1  x  de21)

                     =  0.4              + (0.1 x 1  x 0.047329)

                     =  0.4              + (0.0047329)

                     =  0.4047329

--------------------------------------------------------------------

Layer 2, Neuron 2 calculations:

Layer2.Neuron2.Connection1.New_weight = New_cW221

Layer2.Neuron2.Connection1.Old_weight =    Old_cW221 = 0.9

Layer2.Neuron2.Connection1.connEntry =               cEn221 = 0.768525

Layer2.Neuron2.deltaError =                                     dE22 = -0.14797

New_cW221 = Old_cW221 + (LR x cEn221 x dE22)

                     =    0.9            + (0.1 x 0.768525 x -0.14797)

                     =    0.9            + ( -0.01137)

                     =    0.88863

Layer2.Neuron2.Connection2.New_weight = New_cW222

Layer2.Neuron2.Connection2.Old_weight =    Old_cW222 = 0.9

Layer2.Neuron2.Connection2.connEntry =              cEn222 = 0.574443

Layer2.Neuron2.deltaError =                                    dE22 = -0.14797

New_cW222 = Old_cW222 + (LR x cEn222 x dE22)

                     =    0.9            + (0.1 x 0.574443 x -0.14797)

                     =    0.9            + (-0.0085)

                     =    0.8915

Layer2.Neuron2.New_Bias = New_Bias22

Layer2.Neuron2.Old_Bias =    Old_Bias22 =  0.5

Layer2.Neuron2.deltaError =             dE22 = -0.14797

New_Bias22 = Old_Bias22 + (LR x  1  x  de22)

                     =  0.5              + (0.1 x  1  x  -0.14797)

                     =  0.5            +   (-0.014797)

                     =  0.485203

--------------------------------------------------------------------------

Step 6) Update Layer_1 neuron connection weights and bias.

Layer 1, Neuron 1 calculations:

Let

Layer1.Neuron1.Connection1.New_weight = New_cW111

Layer1.Neuron1.Connection1.Old_weight   =   Old_cW111   =  0.3

Layer1.Neuron1.Connection1.connEntry =                 cEn111 = 0

Layer1.Neuron1.deltaError =                                       dE11 = -0.01864

New_cW111 = Old_cW111 + (LR   x  cEn111   x   dE11)

                     =  0.3              +   (0.1   x     0      x    -0.01864)

                     =  0.3              +   ( 0 )

                     =  0.3    

Layer1.Neuron1.Connection2.New_weight = New_cW112

Layer1.Neuron1.Connection2.Old_weight   =   Old_cW112 = 0.8

Layer1.Neuron1.Connection2.connEntry =               cEn112 = 1

Layer1.Neuron1.deltaError =                                      dE11 = -0.01864

New_cW112 = Old_cW112 + (LR   x  cEn112   x   dE11)

                     =  0.8    +            (0.1     x    1     x     -0.01864)

                     =  0.8    +            (-0.001864)

                     =  0.798136   

Layer1.Neuron1.New_Bias = New_Bias11

Layer1.Neuron1.Old_Bias =    Old_Bias11 = 0.5

Layer1.Neuron1.deltaError =             dE11 = -0.01864

New_Bias11 = Old_Bias11 + (LR   x  1   x  dE11)

                     =  0.5              + (0.1   x 1   x -0.01864 )

                     =  0.5              + (-0.001864)

                     =  0.498136

--------------------------------------------------------------------

Layer 1, Neuron 2 calculations:

Layer1.Neuron2.Connection1.New_weight = New_cW121

Layer1.Neuron2.Connection1.Old_weight =    Old_cW121 = 0.1

Layer1.Neuron2.Connection1.connEntry =               cEn121 = 0

Layer1.Neuron2.deltaError =                                     dE12 =   -0.02793

New_cW121 = Old_cW121 + (LR  x  cEn121 x dE12)

                     =  0.1               + (0.1  x     0     x  -0.02793 )

                     =  0.1   +   (0)

                     =  0.1

Layer1.Neuron2.Connection2.New_weight = New_cW122

Layer1.Neuron2.Connection2.Old_weight =    Old_cW122 = 0.1

Layer1.Neuron2.Connection2.connEntry =              cEn122 = 1

Layer1.Neuron2.deltaError =                                    dE12 =  -0.02793

New_cW122 = Old_cW122 + (LR  x  cEn122  x   dE12)

                     =  0.1                + (0.1   x    1      x  -0.02793)

                     =  0.1    +  (-0.002793)

                     =  0.097207

Layer1.Neuron2.New_Bias = New_Bias12

Layer1.Neuron2.Old_Bias =    Old_Bias12 =  0.2

Layer1.Neuron2.deltaError =             dE12 =  -0.02793

New_Bias12 = Old_Bias12 + (LR    x  1  x  de12)

                     =  0.2             +   (0.1  x  1  x  -0.02793)

                     =  0.2             +  (-0.002793)

                     =  0.197207

----------------------------------------------------------------------

All done. That was just one training cycle. Thank goodness we have computers !

A computer can process these calculations really quickly, and depending on how complicated your neural network is (ie. number of layers, and number of neurons per layer), you may find that the training procedure may take some time. But believe me, if you have designed it right, it is well worth the wait.

Because once you have the desired weights and bias values set up, you are good to go, and as you receive data, the computer can do a single forward pass in a fraction of a second, and you will get your desired output, hopefully :)

Here is a complete Processing.org script that demonstrates the use of my neural network.

Neural Network (Part 7): Cut and Paste Code (click here).

If you liked my tutorial - please let me know in the comments. It is sometimes hard to know if anyone is actually reading this stuff. If you use my code in your own project, I am also happy for you to leave a link to a YouTube video etc in the comments also.

To go back to the table of contents click here

Description: Neural Network (Part 6) : Back Propagation, a worked example Rating: 3.5 Reviewer: Unknown ItemReviewed: Neural Network (Part 6) : Back Propagation, a worked example

Neural Network (Part 1) - The Connection

Introduction

In this tutorial, I am going to walk you through my interpretation of a neural network. I will use terminology that makes sense to me, hoping that Neural Network enthusiasts don't get offended by my novice approach.

This is what a feed-forward Neural network normally looks like:

 The input layer receives input from the outside world, and passes this value to the hidden layer.

The value that reaches the hidden layer depends on the connection between the layers.

Each connection has a weight. This weight multiplier can either increase or decrease the value coming from the input layer. Like water coming out of a tap, you can make it come out faster or slower (or not at all). This weight can even be negative, which would mean that the water is being sucked up, rather than pouring out.

The hidden layer then processes the values, and pass them onto the output layer. The connections between the hidden layer and the output layer also have weights. The values in the output layer are processed and produce a final set of results. The final results can be used to make yes/no decisions, or can be used to make certain classifications etc etc. In my case, I would like to receive sensor data from the arduino, pass this info to the neural network, and get it to classify the data into 4 different classes (Red, Yellow, Green or Ambient), but we will get to see this in another tutorial. For now, we are just going to design a neural network that can be applied to any of your arduino projects... so lets start from the ground up.

I have programmed this neural network in the Processing language, and have decided to break the neural network into smaller bits. Each structural component of the neural network is a class (as you will soon discover).

The connection

ConnEntry (x) : is the value being fed into the connection.
Weight (m) : Is the value that either amplifies or weakens the ConnEntry value.
ConnExit (y): Is the output value of the connection.

The relationship between y and x is linear, and can be described by the following formula.

• y = mx

In other words, you multiply the ConnEntry value by the Weight to get the ConnExit value.

Here is the code for the Connection class:

processing code Connection Class

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

/* -------------------------------------------------------------------------

A connection determines how much of a signal is passed through to the neuron. 

------------------------------------------------------------------------*/



class Connection{

  float connEntry;

  float weight;

  float connExit;

  

  /*This is the default constructor for a Connection  */

  Connection(){

    randomiseWeight();

  }

  

  /*A custom weight for this Connection constructor  */

  Connection(float tempWeight){

    setWeight(tempWeight);

  }

  

  /*Function to set the weight of this connection  */

  void setWeight(float tempWeight){

    weight=tempWeight;

  }

  

  /*Function to randomise the weight of this connection  */

  void randomiseWeight(){

    setWeight(random(2)-1);

  }

  

 /*Function to calculate and store the output of this Connection  */

  float calcConnExit(float tempInput){

    connEntry = tempInput;

    connExit = connEntry * weight;

    return connExit;

  }

}

code formatter

When a connection class is constructed, it automatically randomises the weight for you.
Here are some of the other functions of this class:

• setWeight() : allows you to specifically set the weight of this connection.


• randomiseWeight() : can be used to wipe the "memory" of the connection if required.


• calcConnExit() function is the main function of this class. It is the one that multiplies the connEntry value with the Weight to produce a connExit value as described above.

Up next: Neural Network (Part 2): The Neuron

-------------------------------------------------------------------------

To go back to the table of contents click here

Description: Neural Network (Part 1) - The Connection Rating: 3.5 Reviewer: Unknown ItemReviewed: Neural Network (Part 1) - The Connection

Poor Man's Colour Detector (Part 1) - The concept

Well, I have finally done it. Not sure if I should post the tutorial, because it is not very basic. But then again, a novice like me managed to do it, so why not !! We will need to pull together a number of previous blogs, but rather than jumping back and forth, I think I will just walk you through from beginning to end, and hope that I don't lose you along the way. 

The concept

In my earlier blog posts, I managed to get LEDs to detect light. And through a bit of trial an error, plus a bit of internet research, I found out that an LED will detect light of the same wavelength that it emits. Therefore a red LED will detect RED light, and a yellow LED will detect yellow light etc etc.

I decided to test this theory by placing different coloured MEGA-BLOKs over the LEDs to see if there was any difference ? And from my simplistic experiments, I could see that the RED LED responded much better to a RED Mega-blok than any other colour, and a YELLOW LED responded much better to a Yellow mega-blok than any other colour.

I decided to test out another theory.
Could I detect other mega-blok colours using a red and yellow LED?

While I was aware that I would be better off using primary colour LEDs (RYB - red yellow and blue) or even RGB (red green and blue) in this experiment, I was limited by the LEDs that came in the Sparkfun's Inventor Kit. So I had to use red and yellow.

I developed a little program in "Processing" that would change the colour of the computer screen based on the colour of the mega-blok sitting over the LEDs. I would use specific cut-off values to separate the different readings and translate them into 4 different classes. This was a bit hit and miss. Depending on the ambient light level, the cut-off values would shift. Plus there was a bit of imprecision in the readings.

I then decided to introduce an RGB LED to shine light on the subject. This helped a bit, however, changes in ambient light were proving to be my enemy. I then introduced a photo-cell, however, by this stage, I had so many values and readings and conditions to meet, that I almost gave up.

That was until I read something about neural networks. Then the real fun began. One month later, and a determined novice that was keen to see this project through to the end, my Arduino UNO can now detect coloured mega-bloks!! How did I do it?  I made the computer figure it out !!

Using a feed-forward neural network with a supervised learning back-propagation model (don't switch off just yet), I got the computer to learn the colours under different light conditions and voila ! I ended up with an Arduino UNO and a couple of LEDs (and photocell) that could tell me what colour Mega-blok was sitting over it. The end result is not 100% perfect, but it works quite well.

In the next part, I will walk you through my neural network step by step. Not sure how this tutorial will pan out, but I'll do my best. (Click here for the Neural Network Tutorial)

Or go here for the table of contents

Description: Poor Man's Colour Detector (Part 1) - The concept Rating: 3.5 Reviewer: Unknown ItemReviewed: Poor Man's Colour Detector (Part 1) - The concept