It amazes me that this implementation of BPE doesn't require LOOP. How to make it correct? #20
Description
If a pair is the best in frequency, their combination should reduce the count of both single tokens respectively, which may result in the single tokens' difficulty in keeping high ranks in frequency sorting. But your implementation seems to count all 1-gram, 2-gram .. max-gram in one shot. Is it correct?