Accuracy drops in dflash

[shahizat]

Hello Dflash team,

I was testing DFlash on a Blackwell 6000 Pro to evaluate how speculative decoding affects output generation. I observed that it performs worse on MMLU math questions than the vanilla meta-llama/Llama-3.1-8B-Instruct when using SGLang.

Question: Find the product of the given polynomials in the given polynomial ring. f(x) = 4x - 5, g(x) = 2x^2 - 4x + 2 in Z_8.

Answer using dflash draft decoding: To meta-llama/Llama-3.1-8B-Instruct - Not correct answer

To find the product of the given polynomials in Z_8[x], we need to multiply them as we would in a regular polynomial ring, but we need to perform the operations modulo 8.

f(x) = 4x - 5

g(x) = 2x^2 - 4x + 2

First, we multiply each term of f(x) by each term of g(x):

(4x)(2x^2) = 8x^3 (mod 8) = 0x^3

(4x)(-4x) = -16x^2 (mod 8) = 0x^2

(4x)(2) = 8x (mod 8) = 0x

(-5)(2x^2) = -10x^2 (mod 8) = 6x^2

(-5)(-4x) = 20x (mod 8) = 4x

(-5)(2) = -10 (mod 8) = 6

Now, we combine like terms:

0x^3 + 0x^2 + 0x + 6x^2 + 4x + 6

Combine like terms:

-6x^2 + 4x + 6

So, the product of f(x) and g(x) in Z_8[x] is -6x^2 + 4x + 6.

Answer without dflash (meta-llama/Llama-3.1-8B-Instruct) - Correct

To find the product of the given polynomials in the polynomial ring Z_8[x], we'll follow the standard multiplication process for polynomials, but we'll perform the arithmetic operations in Z_8.

f(x) = 4x - 5

g(x) = 2x^2 - 4x + 2

First, multiply each term of f(x) by each term of g(x):

(4x * 2x^2) + (4x * (-4x)) + (4x * 2) + ((-5) * 2x^2) + ((-5) * (-4x)) + ((-5) * 2)

Now, simplify the expressions using Z_8 arithmetic:

(8x^3) mod 8 = 0 (since 8 mod 8 = 0)

(-16x^2) mod 8 = 0 (since 16 mod 8 = 0)

(8x) mod 8 = 0 (since 8 mod 8 = 0)

(-10x^2) mod 8 = -2x^2 (since 10 mod 8 = 2)

(20x) mod 8 = 4x (since 20 mod 8 = 4)

(-10) mod 8 = 6 (since 10 mod 8 = 2)

Now, combine like terms:

0 + 0 + 0 - 2x^2 + 4x + 6

The resulting polynomial is:

-2x^2 + 4x + 6

However, this polynomial is in Z[x] (the ring of polynomials with integer coefficients). Since we're working in Z_8[x], we need to express the coefficients in Z_8.

In Z_8, the coefficients -2 and 6 can be expressed as 6 and 6, respectively.

So, the product of the given polynomials in Z_8[x] is:

6x^2 + 4x + 6

[jianc99]

Thanks for trying out DFlash on SGLang! In principle, speculative decoding is lossless with respect to the autoregressive target model, but in real systems you may still observe occasional output divergence due to non-determinism in GPU kernels and numerical effects, which can happen even with temperature=0; once a single token differs, the rest of the generation may diverge. This is a known thing in speculative decoding, and we generally expect the overall accuracy on a full benchmark to remain very close to the vanilla model despite rare single-example mismatches (see results in sgl-project/sglang#16818).

[shahizat]

@jianc99 Thanks for the detailed answer. So far, the best model I have discovered is Qwen/Qwen3-8B. I hope you guys add support for more models.

python -m sglang.launch_server \
    --model-path Qwen/Qwen3-8B \
    --speculative-algorithm DFLASH \
    --speculative-draft-model-path z-lab/Qwen3-8B-DFlash-b16 \
    --tp-size 2 \
    --dtype bfloat16 \
    --attention-backend flashinfer \
    --mem-fraction-static 0.75 \
    --trust-remote-code