Neural Network Study

THis is a repository for the study at IITR, KISTI. The goal of this study is to understand the recent work of Neural Network quantization and other neural network compression techniques. The topics of this study covered here are as follows:

Quantization

Binarization

• BinaryNet: Training deep neural networks with weights and activations constrained to +1 or -1, Courbariaux et al., arXive:1602.02830, 2016
  [https://arxiv.org/abs/1602.02830]
• Binaryconnect: Training deep neural networks with binary weights during propagations." Courbariaux et al., NIPS'15
  [http://papers.nips.cc/paper/5647-binaryconnect-training-deep-neural-networks-with-b]
• Binarized Neural Network, Hubara et al., NIPS'16
  [http://papers.nips.cc/paper/6573-binarized-neural-networks]
• XNOR-NET: Imagenet classification using binary convolutional neural networks." Rastegari et al., ECCV'16 [https://link.springer.com/chapter/10.1007/978-3-319-46493-0_32]

k-bits quantization

• DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients.", Zhou et al., " arXiv preprint arXiv:1606.06160 (2016)
  [https://arxiv.org/abs/1606.06160]
• LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks, Dongqing Zhang et al., ECCV'18
  [http://openaccess.thecvf.com/content_ECCV_2018/html/Dongqing_Zhang_Optimized_Quantization_for_ECCV_2018_paper.html]

Training

• How to train a compact binary neural network with high accuracy?, Wei Tang et al., AAAI'17
  [https://www.aaai.org/ocs/index.php/AAAI/AAAI17/paper/download/14619/14454]
• Towards Efficient Training for neural network quantization, Qing Jin et al., arXiv:1912.10207
  [https://arxiv.org/abs/1912.10207]
• Quantization Network, Jiwei Yang et al., CVPR'19
  [http://openaccess.thecvf.com/content_CVPR_2019/papers/Yang_Quantization_Networks_CVPR_2019_paper.pdf]
• Understanding STE in training activation quantized neural nets, Penghang Yin et al. , ICLR'19
  [https://arxiv.org/abs/1903.05662]
• Learning low-precision neural networks without STE, Zhi-Gang Liu et al., arXiv:1903.01061
  [https://arxiv.org/abs/1903.01061]
• Deep Learning with low precision by half-wave Gaussian quantization, Zhaowei Cai et al., CVPR'17
  [http://openaccess.thecvf.com/content_cvpr_2017/html/Cai_Deep_Learning_With_CVPR_2017_paper.html]

Distillation-based approach

• And the bit goes down: Revisiting the Quantization of Neural Networks, Pierre Stock et al., ICLR'20
  [https://arxiv.org/abs/1907.05686]

---

Contributors

Main Contributor

• Kyong-Ha Lee(bart7449@gmail.com), Eunhui Kim(kim.eunhui@gmail.com)