Bert model prediction is slow , consider more practical implementation

[alex-movila]

There is a new approach from AliBaba described here.
ALIBABA AI BEATS HUMANS IN READING-COMPREHENSION TEST

A Deep Cascade Model for Multi-Document Reading Comprehension

It seems faster with better scalability. Not sure there is code available.

[n0thingLLM]

Hi @alex-movila

Thank you for recommending this implementation!

Speed and scalability are indeed very important to use cdQA in production. At this stage, our primary focus is to make end-to-end question answering easy for our users but we might need to dig deeper into this new approach in the future, probably as soon as Alibaba has released some code (I couldn't find it yet).

We'll follow their updates on the topic closely 😉

[alex-movila]

There are some other tips here to make BERT better suited for production:
https://hanxiao.github.io/2019/01/02/Serving-Google-BERT-in-Production-using-Tensorflow-and-ZeroMQ/

[andrelmfarias]

Thanks for the tips @alex-movila !

However, I am afraid they are not compatible with cdQA for two reasons:

1. These tips are for the TF version of BERT, while we use the PyTorch version, provided by Hugging Face

2. The bert-as-service project has one feature: map a variable-length sentence to a fixed-length vector. This is a good feature to use for Information Extraction (it can be used for the Retriever component of cdQA for example - for now we use a TF-IDF approach). But it is not useful for the Question Answering part of the pipeline (i.e. the Reader), where we apply the model BertForQuestionAnswering.

I will take a deeper look at it to see if I can extract some useful stuffs to our package.

[alex-movila]

I think the idea of quantization + pruning could be useful to make BERT smaller.
Also for production there is need of concurrent requests and loading the model only at initialization.

Here is another idea. I think this one should be easy. To implement the reader with allennlp.:
https://allenai.github.io/bi-att-flow/
https://allennlp.org/models

seems 2 lines of code.. I will try it myself.
Though the problem remains that was not trained with non-answerable questions.

[alex-movila]

Related:
Here is another paper for XML model which is available at huggingface
Large Memory Layers with Product Keys (https://arxiv.org/abs/1907.05242)
""outperforms a baseline transformer model with 24 layers, while being twice faster at inference time"

[jacobdanovitch]

Fwiw, using QAPipeline(reader='models/bert_qa_vGPU-sklearn.joblib', predict_batch_size=128, verbose_logging=True).to('cuda') gives me very reasonable inference time over a pretty large set of documents.

Obviously this might not be feasible for everyone, I'm just running this in Colab.

[alex-movila]

Well we must consider production where we have 1000 users doing inference concurrently.
Also not everyone has GPU.

[n0thingLLM]

I think if we can achieve #209 and make cdQA modular (having possibility to chose different retriever/reader impementations) hopefully everyone will be able to have a solution given his/her specific constraints.

[jacobdanovitch]

Well we must consider production where we have 1000 users doing inference concurrently.
Also not everyone has GPU.

For sure on the first point, it obviously doesn't scale. Just wanted to point out that it does work on some level.

That said, I think it might be reasonable to expect the availability of a GPU, even if not a large one. Don't know much about deploying these models, but would inference on CPU really be feasible?

[andrelmfarias]

but would inference on CPU really be feasible?

Currently, inference on CPU is feasible but slow (about 10 to 20s per inference depending on the question and the CPU).

We are thinking about some solutions to improve inference time on CPU, although it will probably take some time to these solutions to be implemented.

If you are interested in contributing to cdQA working with this particular issue, it can be very helpful and we could implement the solutions sooner 😃