vocab : call reserve() for building plamo-2-translate suffix (#17343)

haiyuewa · web-flow · commit a045492088dd · 2025-11-18T18:58:22.000+01:00
Test 'Q4_K_M' quantization on https://huggingface.co/pfnet/plamo-2-translate The 'suffix_to_score' size is 193510, it needs 19 memory allocation with final capacity 262144 to hold the value, if not preserve the memory. Signed-off-by: Haiyue Wang <haiyuewa@163.com>
diff --git a/src/llama-vocab.cpp b/src/llama-vocab.cpp
@@ -1281,6 +1281,7 @@ struct llm_tokenizer_plamo2 : llm_tokenizer {
 
         // Build suffix list in lexicographical order of reversed strings
         std::vector<std::string> suffixes;
+        suffixes.reserve(suffix_to_score.size() + 1);
         for (const auto & pair : suffix_to_score) {
             suffixes.push_back(pair.first);
         }

Original file line number	Diff line number	Diff line change
`@@ -1281,6 +1281,7 @@ struct llm_tokenizer_plamo2 : llm_tokenizer {`
`1281`	`1281`
`1282`	`1282`	`// Build suffix list in lexicographical order of reversed strings`
`1283`	`1283`	`std::vector<std::string> suffixes;`
	`1284`	`+ suffixes.reserve(suffix_to_score.size() + 1);`
`1284`	`1285`	`for (const auto & pair : suffix_to_score) {`
`1285`	`1286`	`suffixes.push_back(pair.first);`
`1286`	`1287`	`}`