ML_101
  • Introduction
  • ML Fundamentals
    • Basics
    • Optimization
    • How to prevent overfitting
    • Linear Algebra
    • Clustering
    • Calculate Parameters in CNN
    • Normalization
    • Confidence Interval
    • Quantization
  • Classical Machine Learning
    • Basics
    • Unsupervised Learning
  • Neural Networks
    • Basics
    • Activation function
    • Different Types of Convolution
    • Resnet
    • Mobilenet
  • Loss
    • L1 and L2 Loss
    • Hinge Loss
    • Cross-Entropy Loss
    • Binary Cross-Entropy Loss
    • Categorical Cross-Entropy Loss
    • (Optional) Focal Loss
    • (Optional) CORAL Loss
  • Computer Vision
    • Two Stage Object Detection
      • Metrics
      • ROI
      • R-CNN
      • Fast RCNN
      • Faster RCNN
      • Mask RCNN
    • One Stage Object Detection
      • FPN
      • YOLO
      • Single Shot MultiBox Detector(SSD)
    • Segmentation
      • Panoptic Segmentation
      • PSPNet
    • FaceNet
    • GAN
    • Imbalance problem in object detection
  • NLP
    • Embedding
    • RNN
    • LSTM
    • LSTM Ext.
    • RNN for text prediction
    • BLEU
    • Seq2Seq
    • Attention
    • Self Attention
    • Attention without RNN
    • Transformer
    • BERT
  • Parallel Computing
    • Communication
    • MapReduce
    • Parameter Server
    • Decentralized And Ring All Reduce
    • Federated Learning
    • Model Parallelism: GPipe
  • Anomaly Detection
    • DBSCAN
    • Autoencoder
  • Visualization
    • Saliency Maps
    • Fooling images
    • Class Visualization
Powered by GitBook
On this page
  • Class Imbalance Problem of One-Stage Detector
  • alpha-Balanced CE Loss
  • Focal Loss
  • alpha-Balanced Variant of FL

Was this helpful?

  1. Loss

(Optional) Focal Loss

Class Imbalance Problem of One-Stage Detector

  • A much larger set of candidate object locations is regularly sampled across an image (~100k locations), which densely cover spatial positions, scales and aspect ratios.

  • The training procedure is still dominated by easily classified background examples. It is typically addressed via bootstrapping or hard example mining. But they are not efficient enough.

alpha-Balanced CE Loss

CE(pt)=−αttilog(pt)CE(p_t) = - \alpha_t t_i log(p_t)CE(pt​)=−αt​ti​log(pt​)
  • To address the class imbalance, one method is to add a weighting factor α\alphaα for class 1 and 1−α1 - \alpha1−α for class -1. α\alphaα may be set by inverse class frequency or treated as a hyperparameter to set by cross validation.

Focal Loss

FL=−∑i=1C=n(1−pi)γtilog(pi)FL = -\sum_{i=1}^{C=n}(1 - p_{i})^{\gamma }t_{i} log (p_{i})FL=−i=1∑C=n​(1−pi​)γti​log(pi​)
  • The loss function is reshaped to down-weight easy examples and thus focus training on hard negatives. A modulating factor (1−pt)γ(1-p_{t})^{\gamma}(1−pt​)γ is added to the cross entropy loss where γ\gammaγ is tested from [0,5][0,5][0,5] in the experiment.

  • There are two properties of the FL:

  • When an example is misclassified and ptp_{t}pt​ is small, the modulating factor is near 1 and the loss is unaffected. As pt→1p_{t} \rightarrow 1pt​→1, the factor goes to 0 and the loss for well-classified examples is down-weighted.

  • The focusing parameter γ\gammaγ smoothly adjusts the rate at which easy examples are down-weighted. When γ=0\gamma = 0γ=0, FL is equivalent to CE. When γ\gammaγ is increased, the effect of the modulating factor is likewise increased. (γ=2\gamma = 2γ=2 works best in experiment.)

alpha-Balanced Variant of FL

FL=−∑i=1C=n−αt(1−pi)γtilog(pi)FL = -\sum_{i=1}^{C=n} - \alpha_t (1 - p_{i})^{\gamma }t_{i} log (p_{i})FL=−i=1∑C=n​−αt​(1−pi​)γti​log(pi​)
  • The above form is used in experiment in practice where α is added into the equation, which yields slightly improved accuracy over the one without α. And using sigmoid activation function for computing p resulting in greater numerical stability.

  • γ\gammaγ: Focus more on hard examples.

  • α\alphaα: Offset class imbalance of number of examples.

PreviousCategorical Cross-Entropy LossNext(Optional) CORAL Loss

Last updated 3 years ago

Was this helpful?