In auto regressive models like transformers, kv cache outputs are reused in inputs. In this situation, we could speed up inference speed by directly memcpy kv cache outputs' virtual address to kv cache's inputs, rather than copying to numpy objects then copying to model inputs.
Similar ideas from genai
In auto regressive models like transformers, kv cache outputs are reused in inputs. In this situation, we could speed up inference speed by directly memcpy kv cache outputs' virtual address to kv cache's inputs, rather than copying to numpy objects then copying to model inputs.
Similar ideas from genai