> For the complete documentation index, see [llms.txt](https://ztlevi.gitbook.io/ml-101/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ztlevi.gitbook.io/ml-101/computer-vision/facenet.md).

# FaceNet

## Model structure

![image-20190801163738683](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-2389dbb5d7617674d3b0ac7d73c7920fd5c01bd8%2Fimage-20190801163738683.png?alt=media)

The network consists of a batch input layer and a deep CNN followed by **L2 normalization**, which results in the **face embedding**. This is followed by the **triplet loss** during training.

Namely, the authors strive for an embedding $$f(x)$$, from an image $$x$$ into a feature space $$\mathbf{R}^d$$, such that:

* squared distance between all faces, independent of imaging conditions, of the same identity is small.
* the squared distance between a pair of face images from different identities is large.

## Triplet Loss

The embedding is represented by $$f (x) \in \mathbf{R}^d$$ . It embeds an image x into a d-dimensional Euclidean space. Additionally, we constrain this embedding to live on the d-dimensional hypersphere, i.e. $$||f(x)||^2 = 1$$.

Here the objective is that we want to make sure that an image $$x^a\_i$$(**anchor**) of a specific person is closer to all other images $$x^p\_i$$ (**positive**) of the same person than it is to any image $$x^n\_i$$ (**negative**) of any other person.

![image-20190801164742807](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-62723765f808d76d0f8fca4591b946897976852d%2Fimage-20190801164742807.png?alt=media)

* $$|f(x^a\_i)-f(x^p\_i)|^2 +\alpha<|f(x^a\_i)-f(x^n\_i)|^2$$ for $$\forall (f(x^a\_i),f(x^p\_i),f(x^n\_i))\in \tau$$
* $$\alpha$$ is a margin that is enforced between positive and negative pairs.
* $$\tau$$ is the set of all possible triplets in the training set and has cardinality $$N$$
* The the objective is to minimize the Loss: $$L = \sum \[|f(x^a\_i)-f(x^p\_i) |^2-|f(x^a\_i)-f(x^n\_i)|^2+\alpha]\_+$$

## Triplet Selection

Given $$x^a\_i$$ , we want to select an $$x^p\_i$$ (**hard positive**) such that $$argmax\_{x^p\_i} |f (x^a\_i )-f (x^p\_i )|^2$$ and similarly $$x^n\_i$$ (**hard negative**) such that $$argmin\_{x^n\_i}|f(x^a\_i )-f (x^n\_i )|^2$$ .

Two obvious choices:

* Generate triplets offline every n steps, using the most recent network checkpoint and computing the argmin and argmax on a subset of the data.
* **Generate triplets online**. This can be done by select- ing the hard positive/negative exemplars from within a mini-batch.

### Online Triplets Generation

* To have a meaningful representation of the anchor- positive distances, it needs to be ensured that **a minimal number of exemplars of any one identity is present in each minibatch**.
  * around 40 faces are selected per identity per minibatch.
  * randomly sampled negative faces are added to each mini-batch.
* Instead of picking the hardest positive, we use all anchor-positive pairs in a mini-batch while still selecting the hard negatives. For all anchor-positive method was **more stable and converged slightly faster at the beginning of training**.
* Selecting the hardest negatives can in practice lead to bad local minima early on in training, specifically it can result in a collapsed model (i.e. $$f(x) = 0$$). In order to mitigate this, it helps to select $$x^n\_i$$ such that

  $$|f(x^a\_i)-f(x^p\_i)|^2\_2<|f(x^a\_i)-f(x^n\_i)|^2\_2$$

  * these negative exemplars **semi-hard​**, as they are further away from the anchor than the positive exemplar, but still hard because the squared distance is close to the anchor-positive distance.
  * Those negatives lie inside the margin $$\alpha$$
* In most experiments the author used a batch size of around 1,800 exemplars.

## Deep Convolutional Networks

![image-20190801185039847](https://637078585-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MYsi-h_n0zY_8MKKgyu%2Fuploads%2Fgit-blob-f6ea5190fcf9f6f048bd8a942b38c1e10183bff6%2Fimage-20190801185039847.png?alt=media)

* Use rectified linear units as the non-linear activation function
* Now use Inception+ResNet

## Model Evaluation

The author evaluate our method on the face verification task. I.e. given a pair of two face images a squared L2 distance threshold $$D(x\_i,x\_j)$$ is used to determine the classification of **same** and **different**. All faces pairs $$(i, j)$$ of the same identity are denoted with $$\mathcal{P}*{same}$$, whereas all pairs of different identities are denoted with $$\mathcal{P}*{diff}$$

* the set of all **true accepts** as:

  $$TA(d) = { (i,j) \in \mathcal{P}\_{same}$$, with $$\space D(xi,xj) <= d }$$.

  These are the face pairs $$(i, j)$$ that were correctly classified as same at threshold $$d$$.
* the set of all pairs that was incorrectly classified as same(**false accept**) is:

  $$TP(d) = { (i,j) \in \mathcal{P}\_{diff}$$, with $$\space D(xi,xj) <= d }$$.
* The validation rate $$VAL(d)$$ and the false accept rate $$FAR(d)$$ for a given face distance $$d$$ are then defined as:

  $$VAL(d)=\frac{|TA(d)|}{|P\_{same}|}$$ ,

  $$FAR(d)=\frac{|TP(d)|}{|P\_{diff}|}$$