Cave of Python

In this post we’ll take a look at a simpler example of a neural network in Python than the iris example we saw previously.

First we’ll generate some sine data. The task of the ANN will be to try to output the square of the input values.

First we’ll add the necessary imports, then we’ll generate some data and split it into train and test segments.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from keras.utils import to_categorical
from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import r2_score
import numpy as np

X = np.arange(20)
y = X**2

X_train, X_test, y_train, y_test = train_test_split(X, y, shuffle=True, train_size=0.7)

Next we’ll construct an ANN and then print a summary of the model.

Our model here consists of three layers of neurons.

The first and second layers have 50 neurons each, and the third (output) layer has only one neuron, because we want the ANN to output a single value.

We specify an input size of 1 for the first layer; the input is just a single number, and the idea is the ANN outputs the square of that number.

We’ve used the basic ReLu activation function for the first two layers. This just sets the output of the neuron to zero if it’s below zero. Even though that sounds like it only just meets the definition of a “non linear” function (which is what we need), it does the trick nicely for many purposes.

The output layer has no activation function; we’ll just read its output directly.

To assess “loss” (how far away the output is from what we want), we’ll just use the mean squared error. That is, we’ll square the difference between the desired output and actual output and just use that as the loss. We want this to decrease during training.

Finally, we print a summary of the model.

model = Sequential()
model.add(Dense(50, input_dim=1, activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(1))

model.compile(loss='mean_squared_error', optimizer='adam')

print(model.summary())

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (None, 50)                100       
                                                                 
 dense_1 (Dense)             (None, 50)                2550      
                                                                 
 dense_2 (Dense)             (None, 1)                 51        
                                                                 
=================================================================
Total params: 2,701
Trainable params: 2,701
Non-trainable params: 0
_________________________________________________________________

Now we can train the network on our data. We’ll train the ANN for 5000 epochs (5000 passes over the training data).

Finally we’ll get the predictions the network makes for our test data segment, and we’ll compare them to the correct values using an R squared score.

model.fit(X_train, y_train, epochs=5000)

y_predicted = model.predict(X_test)

print("Score:", r2_score(y_test, y_predicted))

Here’s the complete program. I find it completes in less than a minute and the R squared score is nearly 1.

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from keras.utils import to_categorical
from keras.models import Sequential
from keras.layers import Dense
from sklearn.metrics import r2_score
import numpy as np

X = np.arange(20)
y = X**2

X_train, X_test, y_train, y_test = train_test_split(X, y, shuffle=True, train_size=0.7)

model = Sequential()
model.add(Dense(50, input_dim=1, activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(1))

model.compile(loss='mean_squared_error', optimizer='adam')

print(model.summary())

model.fit(X_train, y_train, epochs=5000)

y_predicted = model.predict(X_test)

print("Score:", r2_score(y_test, y_predicted))

...
Epoch 4998/5000
1/1 [==============================] - 0s 3ms/step - loss: 0.5179
Epoch 4999/5000
1/1 [==============================] - 0s 5ms/step - loss: 0.5176
Epoch 5000/5000
1/1 [==============================] - 0s 3ms/step - loss: 0.5174
1/1 [==============================] - 0s 84ms/step
Score: 0.9983980688830657