Skip to content

refactor quant table #3911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tpoisonooo wants to merge 18 commits into
base: master
Choose a base branch
from
Open

refactor quant table #3911

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
2a5a296
feat(tools/quantize): support toml
tpoisonooo Jun 13, 2022
e8ad914
apply code-format changes
tpoisonooo Jun 13, 2022
77e6546
feat(tools/quantize): add .ini parser
tpoisonooo Jun 15, 2022
8e2f806
apply code-format changes
tpoisonooo Jun 15, 2022
146b8ba
improvement(tools/quantize): add ini config
tpoisonooo Jun 16, 2022
12f075f
Merge branch 'master' of https://github.com/tencent/ncnn into ncnn-in…
tpoisonooo Jun 16, 2022
f719ee7
Merge branch 'ncnn-int8-toml' of https://github.com/tpoisonooo/ncnn i…
tpoisonooo Jun 16, 2022
9863b26
improvement(tools/quantize): refactor code
tpoisonooo Jun 16, 2022
1612caf
apply code-format changes
tpoisonooo Jun 16, 2022
be66fac
test(tools/quantize/ncnn2int8): test quant sqznet
tpoisonooo Jun 16, 2022
ba6640d
improvement(CMakeLists): downgrade to cxx11
tpoisonooo Jun 16, 2022
d106fc0
apply code-format changes
tpoisonooo Jun 16, 2022
fab112d
Update CMakeLists.txt
tpoisonooo Jun 16, 2022
77cf07a
Update ncnn2table.cpp
tpoisonooo Jun 16, 2022
9262515
Merge branch 'ncnn-int8-toml' of https://github.com/tpoisonooo/ncnn i…
tpoisonooo Jun 16, 2022
9d473f5
fix(CI): remove cxx17 grammar
tpoisonooo Jun 16, 2022
181714e
fix(tools/quantize): typo
tpoisonooo Jun 16, 2022
b32dd56
docs(ncnn2int8): add ini description
tpoisonooo Jun 17, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 26 additions & 2 deletions docs/how-to-use-and-FAQ/quantized-int8-inference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Some imagenet sample images here https://github.com/nihui/imagenet-sample-images

```shell
find images/ -type f > imagelist.txt
./ncnn2table mobilenet-opt.param mobilenet-opt.bin imagelist.txt mobilenet.table mean=[104,117,123] norm=[0.017,0.017,0.017] shape=[224,224,3] pixel=BGR thread=8 method=kl
./ncnn2table mobilenet-opt.param mobilenet-opt.bin imagelist.txt mobilenet.table mean=[104,117,123] norm=[0.017,0.017,0.017] shape=[224,224,3] pixel=BGR thread=8 method=kl format=txt
```

* mean and norm are the values you passed to ```Mat::substract_mean_normalize()```
Expand All @@ -35,6 +35,7 @@ find images/ -type f > imagelist.txt
* pixel is the pixel format of your model, image pixels will be converted to this type before ```Extractor::input()```
* thread is the CPU thread count that could be used for parallel inference
* method is the post training quantization algorithm, kl and aciq are currently supported
* format is the output file type of quantization parameters, choose `ini` for `txt`. Using `txt` by default

If your model has multiple input nodes, you can use multiple list files and other parameters

Expand All @@ -60,7 +61,7 @@ mobilenet.load_model("mobilenet-int8.bin");

## mixed precision inference

Before quantize your model, comment the layer weight scale line in table file, then the layer will do the float32 inference
Before quantize your model, comment layer weight scale line in the table file with `txt` format, then the layer will do the float32 inference

```
conv1_param_0 156.639840536
Expand All @@ -69,3 +70,26 @@ conv1_param_0 156.639840536
```
#conv1_param_0 156.639840536
```

If you are using `ini` format, just remove whole quantization parameters of the layer, for example:

```
[conv0]
type = "Conv"
weight = [ 156.639840536 ]
input_scale = 1.23

[fire]
type = "Gemm"
weight = [ 156.639840536 ]
input_scale = 1.23
```

to

```
[fire]
type = "Gemm"
weight = [ 156.639840536 ]
input_scale = 1.23
```
2 changes: 1 addition & 1 deletion tools/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ if(NCNN_VULKAN)
target_link_libraries(ncnn2mem PRIVATE ${Vulkan_LIBRARY})
endif()

add_executable(ncnnoptimize ncnnoptimize.cpp)
add_executable(ncnnoptimize ncnnoptimize.cpp modelwriter.cpp)
target_link_libraries(ncnnoptimize PRIVATE ncnn)
if(NCNN_VULKAN)
target_link_libraries(ncnnoptimize PRIVATE ${Vulkan_LIBRARY})
Expand Down
Loading