Recurrent Neural Networks (RNN) for Sequence Modeling

RNNs & Sequence Data

RNNs process sequential data by maintaining a hidden state that evolves over time.

Common RNN cells

• SimpleRNN – basic recurrent cell.
• LSTM – long short‑term memory, mitigates vanishing gradients.
• GRU – gated recurrent unit, simpler than LSTM.

Example: Sentiment classification with IMDB reviews

import tensorflow as tf
from tensorflow.keras import layers, models

# Load dataset (already tokenized)
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.imdb.load_data(num_words=10000)

# Pad sequences to same length
maxlen = 200
X_train = tf.keras.preprocessing.sequence.pad_sequences(X_train, maxlen=maxlen)
X_test = tf.keras.preprocessing.sequence.pad_sequences(X_test, maxlen=maxlen)

model = models.Sequential([
    layers.Embedding(input_dim=10000, output_dim=128, input_length=maxlen),
    layers.LSTM(64),
    layers.Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()

history = model.fit(X_train, y_train, epochs=5, batch_size=64, validation_split=0.2)

test_loss, test_acc = model.evaluate(X_test, y_test)
print('Test accuracy:', test_acc)

Visualizing training progress

import matplotlib.pyplot as plt
plt.plot(history.history['accuracy'], label='train')
plt.plot(history.history['val_accuracy'], label='val')
plt.legend()
plt.title('RNN Training Accuracy')
plt.show()

Tips for training RNNs

• Truncate/Pad sequences to a uniform length.
• Use masking for variable‑length inputs.
• Apply gradient clipping to avoid exploding gradients (optimizer = tf.keras.optimizers.Adam(clipnorm=1.0)).

When to use RNNs

• Text sentiment analysis, language modeling.
• Time‑series forecasting.
• Speech recognition.