Table of contents

If you are human and curious about your future, then the recurrent neural network (RNN) is definitely a tool to consider. Part 1 will demonstrate some simple RNNs using TensorFlow 2.0 and Keras functional API.

## What is RNN

An **RNN** is a class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence (time series). This allows it to exhibit temporal dynamic behaviour.

RNNs come in many variants, such as fully recurrent, Elman networks and Jordan networks, Long short-term memory, Bi-directional, etc. Nevertheless, the basic idea of RNN is to memory patterns from the past using cells to predict the future.

## The time-series data

In this demo, we first generate a time series of data using a sinus function. We then fetch the data into an RNN model for training and then get some prediction data.

First, we enable TensorFlow 2.0 and import some libraries

1 2 3 4 5 6 7 | %tensorflow_version 2.x import numpy as np import tensorflow as tf import tensorflow.keras as keras import matplotlib as mpl import matplotlib.pyplot as plt %matplotlib inline |

The data follows the simple equation:

\(x(t) = 0.5 \times sin (f(t))\)

\(f(t) = (t_i – o_i)\times(10f_i + 10)\) where \(t_i, o_i, f_i \in [0, 1)\)

The following function generates time-series data following above equation.

1 2 3 4 5 | def generate_time_series(batch_size, n_steps): freq1, offsets1 = np.random.rand(2, batch_size, 1) time = np.linspace(0, 1, n_steps) series = 0.5 * np.sin((time - offsets1) * (freq1 * 10 + 10)) # sine wave return series[..., np.newaxis].astype(np.float32) |

We want to generate the train, validation, and test data where every `n_steps`

(n data points) we have one next value (one step ahead for t+1) (see Figure 2).

1 2 3 4 5 6 7 | n_steps = 50 #generate 10k data points series = generate_time_series(10000, n_steps + 1) X_train, y_train = series[:7000, :n_steps], series[:7000, -1] X_valid, y_valid = series[7000:9000, :n_steps], series[7000:9000, -1] X_test, y_test = series[9000:, :n_steps], series[9000:, -1] |

The shapes of `x_train`

and `y_train`

:

1 2 | X_train.shape, y_train.shape #((7000, 50, 1), (7000, 1)) |

## Prediction with the simplest RNN

With Keras, things can be very pretty straightforward. We will first try with a shallow RNN with only one layer.

1 2 3 | model = keras.models.Sequential([ keras.layers.SimpleRNN(1, input_shape=[None, 1]) ]) |

That is it, then we compile the model using `Adam`

optimizer.

1 2 | optimizer = keras.optimizers.Adam(lr=0.005) model.compile(loss="mse", optimizer=optimizer) |

After that, we train the model and log the mean squared error each epoch.

1 | history = model.fit(X_train, y_train, epochs=20, validation_data=(X_valid, y_valid)) |

Remember that, if you don’t pass validation data, then you won’t have `val_loss`

in `history`

. Now, we can plot the training loss using the flowing function:

1 2 3 4 5 6 7 8 9 | def plot_learning_curves(loss, val_loss): plt.plot(np.arange(len(loss)) + 0.5, loss, "b.-", label="Training loss") plt.plot(np.arange(len(val_loss)) + 1, val_loss, "r.-", label="Validation loss") plt.gca().xaxis.set_major_locator(mpl.ticker.MaxNLocator(integer=True)) plt.axis([1, 20, 0, 0.05]) plt.legend(fontsize=14) plt.xlabel("Epochs") plt.ylabel("Loss") plt.grid(True) |

1 2 | plot_learning_curves(history.history["loss"], history.history["val_loss"]) plt.show() |

Predict on validation data and plot the result (the last point on the series):

1 2 3 | y_pred = model.predict(X_valid) plot_series(X_valid[0, :, 0], y_valid[0, 0], y_pred[0, 0]) plt.show() |

## Forecasting several steps ahead

This is the fun part to predict some more steps in the future (so you don’t have to go to a foreteller :D). First, we generate new data.

1 2 3 | series = generate_time_series(1, n_steps + 30) X_new, Y_new = series[:, :n_steps], series[:, n_steps:] X = X_new |

Next, we will predict `30`

steps ahead (\(t_i \to t_{i+30}\)) * one by one*.

1 2 3 4 5 | for step_ahead in range(30): y_pred_one = model.predict(X[:, step_ahead:])[:, np.newaxis, :] X = np.concatenate([X, y_pred_one], axis=1) Y_pred = X[:, n_steps:] |

As shown in Figure 3, the accuracy is not really good with one-by-one prediction. Now we can modify the model so that it can predict several at once. The new model supports 10 steps ahead at once:

1 2 3 4 5 | model = keras.models.Sequential([ keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]), keras.layers.SimpleRNN(20), keras.layers.Dense(10) ]) |

We then see a better result with this model.

## Note on replacing `outputprojectionwrapper`

from TF1.X

If you work with TensorFlow 1.X, there is a good chance that you use `OUTPUTPROJECTIONWRAPPER`

for your RNN model.

Unfortunately, this function was deprecated in TF2.0 and was not included in the `tf.compat.v1`

. A dense layer is recommended for that based on this post. Therefore, I include another example for ones who want to convert their model from TF1.x to TF2.x. We simply stack up another dense layer at the end (similar to the previous example).

1 2 3 4 | model = keras.models.Sequential([ keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]), keras.layers.Dense(1) ]) |

## Conclusion

Working with a recurrent neural network is fun. Nevertheless, the current setting did not fully exploit the power of the RNN. The next part, we will create more complicated data and apply with deeper models of RNN and different cell types, such as LSTM or GRU.