> For the complete documentation index, see [llms.txt](https://ztlevi.gitbook.io/ml-101/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ztlevi.gitbook.io/ml-101/loss/cross_entropy_loss.md).

# Cross-Entropy Loss

[Refreence](https://gombru.github.io/2018/05/23/cross_entropy_loss/)

## Cross-Entropy loss

The **Cross-Entropy Loss** is actually the only loss we are discussing here. The other losses names written in the title are other names or variations of it. The CE Loss is defined as:

$$
CE = -\sum\_{i}^{C}t\_{i} log (s\_{i})
$$

Where $$t\_i$$ and $$s\_i$$ are the ground truth and the CNN score for each $$class\_i$$ in $$C$$. As **usually an activation function (Sigmoid / Softmax) is applied to the scores before the CE Loss computation**, we write $$f(s\_i)$$ to refer to the activations.

In a **binary classification problem**, where $$C'=2$$, the Cross Entropy Loss can be defined also as [\[discussion\]](https://datascience.stackexchange.com/questions/9302/the-cross-entropy-error-function-in-neural-networks):

$$
CE = -\sum\_{i=1}^{C'=2}t\_{i} log (s\_{i}) = -t\_{1} log(s\_{1}) - (1 - t\_{1}) log(1 - s\_{1})
$$

Where it’s assumed that there are two classes: $$C\_1$$ and $$C\_2$$. $$t\_1$$ \[0,1] and $$s\_1$$ are the ground truth and the score for $$C\_1$$, and $$t\_2=1-t\_1$$ and $$s\_2=1-s\_1$$ are the ground truth and the score for $$C\_2$$. That is the case when we split a Multi-Label classification problem in $$C$$ binary classification problems. See next Binary Cross-Entropy Loss section for more details.

**Logistic Loss** and **Multinomial Logistic Loss** are other names for **Cross-Entropy loss**. [\[Discussion\]](https://stats.stackexchange.com/questions/166958/multinomial-logistic-loss-vs-cross-entropy-vs-square-error/172790)

```python
def softmax(X):
    exps = np.exp(X)
    return exps / np.sum(exps)


def cross_entropy(predictions, targets):
    N = predictions.shape[0]
    ce = -np.sum(targets * np.log(predictions)) / N
    return ce


predictions = np.array([[0.25, 0.25, 0.25, 0.25], [0.01, 0.01, 0.01, 0.97]]) # (N, num_classes)
targets = np.array([[1, 0, 0, 0], [0, 0, 0, 1]]) # (N, num_classes)

cross_entropy(predictions, targets)
# 0.7083767843022996

log_loss(targets, predictions)
# 0.7083767843022996

log_loss(targets, predictions) == cross_entropy(predictions, targets)
# True
```

The layers of Caffe, Pytorch and Tensorflow than use a Cross-Entropy loss without an embedded activation function are:

* Caffe: [Multinomial Logistic Loss Layer](http://caffe.berkeleyvision.org/tutorial/layers/multinomiallogisticloss.html). Is limited to multi-class classification (does not support multiple labels).
* Pytorch: [BCELoss](https://pytorch.org/docs/master/nn.html#bceloss). Is limited to binary classification (between two classes).
* TensorFlow: [log\_loss](https://www.tensorflow.org/api_docs/python/tf/losses/log_loss).