> For the complete documentation index, see [llms.txt](https://ztlevi.gitbook.io/ml-101/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ztlevi.gitbook.io/ml-101/nlp/self-attention.md).

# Self Attention

## Simple RNN + Self Attention

$$
c\_0 = 0 \ h\_0 = 0 \\
$$

Simple RNN: $$h\_i = tanh(A \cdot \[ \begin{matrix} x\_i \ h\_{i-1}\end{matrix} ] + b)$$

Simple RNN + Self Attention: $$h\_i = tanh(A \cdot \[ \begin{matrix} x\_i \ c\_{i-1}\end{matrix} ] + b)$$

![calculate h1](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-99b1ad3c7e0aa87a3e46a37f23c2864e1fa06d0c%2Fscreen-shot-2021-08-14-at-5.13.59-pm.png?alt=media)

![calculate h2](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-fdd0334b5e39bd0db3a507fc77703688b7e947f8%2Fscreen-shot-2021-08-14-at-5.14.46-pm.png?alt=media)

Calculate Weights: $$\alpha\_i=align(h\_i, h\_2)$$

![calculate c2](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-139153f12da01ae91fbc42368773a1a584790038%2Fscreen-shot-2021-08-14-at-5.16.29-pm.png?alt=media)

## Summary

* With self-attention, RNN is less likely to forget.
* Pay attention to the context relevant to the new input.

![self attention focus](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-9795e73c303d5032b1a537eaa3c082445c84976f%2Fscreen-shot-2021-08-14-at-5.20.08-pm.png?alt=media)

## Reference

* Cheng, Dong, & Lapata. Long Short-Term Memory-Networks for Machine Reading. In EMNLP, 2016.