Address possible vulnerability in preauth lengths in hybrid KEM

[alharrison]

Martin Ramkellyn (martin-r-dev) found during an AI-assisted exploration into crafting malformed packets to exploit an SSH server.

The use of the blob does not ensure proper lengths before being passed into the KEM handling code. This allows a possible underflow.

Example that would generate the underflow:
client_blob_len = 1567
kem->length_public_key = 1568
ecdh_part_len = 1567 - 1568 = 18446744073709551615 (size_t wrap)

Cases that Martin tested with his crafted script:
All cases below refer to message-30 (SSH2_MSG_KEX_ECDH_INIT) mutation for mlkem1024nistp384-sha384.

Case	Declared q_c_len	Actual q_c_blob bytes sent	Helper reached	Observed behavior
Declared/actual mismatch	Very large (e.g. 1000000)	Short (e.g. 32)	No	Parser rejects early as incomplete message [preauth]
Self-consistent short blob	799	799	Yes	OQS_KEM_encaps fails, handshake aborts, no underflow trace
Near-boundary short blob	1535 / 1536 / 1537 / 1540 / 1567	Same as declared	Yes	OQS_KEM_encaps may succeed; split underflow path observed; pre-auth allocation failure and connection abort
Expected size	1665	1665	Yes	Normal split length (97), handshake continues normally

Additional local observations:

• 799 repeatedly showed encapsulation failure.
• 1535 and 1540 repeatedly showed encapsulation success followed by downstream failure on malformed split handling.
• 1567 is a deterministic underflow demonstration (ec_len wrap).

What was observed:

• Connection-level pre-auth failures for malformed inputs.
• Listener process stayed alive across repeated mutated attempts in this setup.
• No reliable listener crash was reproduced from these packet mutations.