ddr memory requrement in long-context (8k or more)

[sigpro]

In long-context cases (e.g., Qwen2.5-3B or Qwen3-1.7B with 8K prefill), how much DDR memory is typically needed to run on a Qualcomm NPU?

[chenghuaWang]

Hi @sigpro , We have not actually tested the specific memory requirements for an 8K context. You can try building a model with an 8K context length to evaluate it. I estimate that the maximum required activation memory can be approximated by calculating the memory needed for the attention component.

[oreomaker]

The total memory usage basically includes model weights(1GB+ for 1.7B model), prefill inputs & outputs(chunked input ids and past KV for every layers) and decode inputs & outputs(the past KV is shared between prefill stage). Given the 4 GB memory limit of the Qualcomm NPU, it may be necessary to shard the model when supporting larger models or longer context lengths.