> For the complete documentation index, see [llms.txt](https://ztlevi.gitbook.io/ml-101/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ztlevi.gitbook.io/ml-101/nlp/lstm_ext.md).

# LSTM Ext.

## Standard RNN

![](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-686e0e0c861ced9c13ff97f5470610c1c607b4da%2Frnn1.png?alt=media)

## Stacked RNN & LSTM

![](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-00341625442af138840b0f4f09832277936544bb%2Fstacked_rnn_1.png?alt=media)

### Sample code for stacked LSTM

```python
from keras.models import Sequential
from keras.layers import LSTM, Embedding, Dense

vocabulary = 10000
embedding_dim = 32
word_num = 500
state_dim = 32

model = Sequential()
model.add(Embedding(vocabulary, embedding_dim, input_length=word_num))
model.add(LSTM(state_dim, return_sequences=True, dropout=0.2))
model.add(LSTM(state_dim, return_sequences=True, dropout=0.2))
model.add(LSTM(state_dim, return_sequences=False, dropout=0.2))
model.add(Dense(1, activation='sigmoid'))
```

| Layer (type)             | Output Shape    | Param  |
| ------------------------ | --------------- | ------ |
| embedding\_1 (Embedding) | (None, 500, 32) | 320000 |
| lstm\_1(LSTM)            | (None, 500, 32) | 8320   |
| lstm\_2(LSTM)            | (None, 500, 32) | 8320   |
| lstm\_3(LSTM)            | (None, 32)      | 8320   |
| dense\_1 (Dense)         | (None, 1)       | 33     |

Total params: 344,993

Trainable params: 344,993

Non-trainable params: 0

## Bidirectional RNN && LSTM

![](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-d8d0fdb9b32bb76b7630fd7ccf0f406b718372c6%2Fbidirectional-rnn-1.png?alt=media)

### Sample code for Bi-LSTM

```python
from keras.model import Sequential
from keras.layers import LSTM, Embedding, Dense, Bidirectional

vocabulary = 10000
embedding_dim = 32
word_num = 500
state_dim = 32

model = Sequential()
model.add(Embedding(vocabulary, embedding_dim, input_length=word_num))
model.add(Bidirectional(LSTM(state_dim, return_sequences=False, dropout=0.2)))
model.add(Dense(1, activation='sigmoid'))
```

| Layer (type)                   | Output Shape    | Param  |
| ------------------------------ | --------------- | ------ |
| embedding\_1 (Embedding)       | (None, 500, 32) | 320000 |
| bidirectional\_1 (Bidirection) | (None, 64)      | 16640  |
| dense\_1 (Dense)               | (None, 1)       | 65     |

Total params: 336,705 Trainable params: 336,705 Non-trainable params: 0

## Summary

* SimpleRNN and LSTM are two kinds of RNNs; always use LSTM instead of SimpleRNN.
* Use Bi-RNN instead of RNN whenever possible.
* Stacked RNN may be better than a single RNN layer (if n is big).
* Pretrain the embedding layer (if n is small).