Does it really take 2 days to evaluate llama-3.1-8b at 128k length?

[eldarkurtic]

Hi,

I am attempting to reproduce the results for the Llama-3.1-8B-Instruct model by following the steps provided in the README. Everything is set up within your Docker environment, and I am using vLLM for inference. My setup includes a single H100 GPU with a batch size of 8, as specified in the example scripts.

With this configuration, the runtime for processing a 128k context length (synthetic task) is approximately 2 days. Is this runtime expected? If not, could you please share the configuration or optimizations you used to efficiently handle this context length?

[hsiehjackson]

Hi @eldarkurtic, I don't apply any additional optimizations when running inference. I usually use 8 GPUs with TP=8 using vLLM. It takes around 2 hours to run 128K length with 500 samples for Llama-3.1-8B-Instruct.

[eldarkurtic]

Which batch size are you using?

[hsiehjackson]

I am using batch size = 1.