# Attention without RNN

## Attention Layer: Attention without RNN

Compared with Simple RNN + Attention, it's not using hidden states from RNN to compute Key, Value, Query parameters.

* We study Seq2Seq model (encoder + decoder).
* Encoder’s inputs are vectors $$x_1,x_2,...,x_m$$.
* Decoder’s inputs are vectors $$x'_1,x'_2,...,x'_t$$.
* Keys and values are based on encoder’s inputs $$x_1,x_2,...,x_m$$.
  * Key: $$k_{:i}=W_K x_i$$
  * Value: $$v_{:i}=W_V x_i$$
* Queries are based on decoder’s inputs $$x'_1,x'_2,...,x'_t$$
* Query: $$q_{:j}=W_Q x'_j$$

![](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-9e2822341370fe12a5720ad8f5375cecf1d130a5%2Fscreen-shot-2021-08-14-at-7.34.53-pm.png?alt=media)

* Compute weights: $$\alpha_{:j}=Softmax(K^T q_{:j}) \in R^m$$

![](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-c54a74ea6eb8178af6259142f88fc26997c6d9d2%2Fscreen-shot-2021-08-14-at-7.37.36-pm.png?alt=media)

* Context vector: $$c_j=\alpha_{ij} v_{:1} + ... +\alpha_{mj} v_{:m}=V \alpha_{:j}$$
  * Thus, $$c_{:j}$$is a function of $$x'_j$$and $$[x_1,...x_m]$$

![](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-df3cd1b94250df9173943c0b3f9ec3e303a56647%2Fscreen-shot-2021-08-14-at-7.37.41-pm.png?alt=media)

* Output of attention layer: $$C=[c_{:1}, c_{:2},c_{:3},...,c_{:t}]$$

### Attention layer summary

* Attention layer: $$C=Attn(X,X')$$
  * Encoder's inputs: $$X=[x_1, x_2,...,x_m]$$
  * Decoder's inputs: $$X'=[x'_1,x'_2,...,x'_t]$$
  * Parameters: $$W_Q,W_K, W_V$$

![](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-028578b8e3d90cb2ccd707476d542f1469737eec%2Fscreen-shot-2021-08-14-at-7.44.35-pm.png?alt=media)

### Self-Attention layer summary

* Self-Attention layer: $$C=Attn(X,X)$$
  * Inputs: $$X=[x_1, x_2,...,x_m]$$
  * Parameters: $$W_Q,W_K, W_V$$

![](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-f9b209ca5d8e6c2fd5b0c2426205a6d7fa93a2fb%2Fscreen-shot-2021-08-14-at-7.46.26-pm.png?alt=media)

## Summary

* Attention was originally developed for Seq2Seq RNN models \[1].
* Self-attention: attention for all the RNN models (not necessarily Seq2Seq models \[2].
* Attention can be used without RNN \[3].
* We learned how to build attention layer and self-attention layer.

## Reference

* Original paper: Vaswani et al. Attention Is All You Need. In NIPS, 2017.
* [youtube](https://www.youtube.com/watch?v=aButdUV0dxI\&t=1s)
