Skip to content

Address possible vulnerability in preauth lengths for hybrid KEM#190

Merged
alharrison merged 2 commits intoopen-quantum-safe:OQS-v10from
alharrison:bugfix/hybrid_preauth_length
Mar 16, 2026
Merged

Address possible vulnerability in preauth lengths for hybrid KEM#190
alharrison merged 2 commits intoopen-quantum-safe:OQS-v10from
alharrison:bugfix/hybrid_preauth_length

Conversation

@alharrison
Copy link
Copy Markdown

@alharrison alharrison commented Feb 26, 2026

The first commit covers updating the slh-dsa name updates. We can either merge this one in to accomplish both, or we can merge #189 and then I'll rebase this to only include the hybrid preauth issue.

Martin Ramkellyn (@martin-r-dev) found during an AI-assisted exploration into crafting malformed packets to exploit an SSH server.

The use of the blob does not ensure proper lengths before being passed into the KEM handling code. This allows a possible underflow.

Example that would generate the underflow:
client_blob_len = 1567
kem->length_public_key = 1568
ecdh_part_len = 1567 - 1568 = 18446744073709551615 (size_t wrap)

Cases that Martin tested with his crafted script:
All cases below refer to message-30 (SSH2_MSG_KEX_ECDH_INIT) mutation for mlkem1024nistp384-sha384.

Case Declared q_c_len Actual q_c_blob bytes sent Helper reached Observed behavior
Declared/actual mismatch Very large (e.g. 1000000) Short (e.g. 32) No Parser rejects early as incomplete message [preauth]
Self-consistent short blob 799 799 Yes OQS_KEM_encaps fails, handshake aborts, no underflow trace
Near-boundary short blob 1535 / 1536 / 1537 / 1540 / 1567 Same as declared Yes OQS_KEM_encaps may succeed; split underflow path observed; pre-auth allocation failure and connection abort
Expected size 1665 1665 Yes Normal split length (97), handshake continues normally

Additional local observations:

  • 799 repeatedly showed encapsulation failure.
  • 1535 and 1540 repeatedly showed encapsulation success followed by downstream failure on malformed split handling.
  • 1567 is a deterministic underflow demonstration (ec_len wrap).

What was observed:

  • Connection-level pre-auth failures for malformed inputs.
  • Listener process stayed alive across repeated mutated attempts in this setup.
  • No reliable listener crash was reproduced from these packet mutations.

Signed-off-by: Alexander Harrison <Alexander.R.Harrison@gmail.com>
Signed-off-by: Alexander Harrison <Alexander.R.Harrison@gmail.com>
@alharrison alharrison self-assigned this Mar 9, 2026
@alharrison alharrison marked this pull request as ready for review March 12, 2026 14:04
@alharrison alharrison changed the title Bugfix/hybrid preauth length Address possible vulnerability in preauth lengths for hybrid KEM Mar 12, 2026
@dstebila
Copy link
Copy Markdown
Member

Hi Alex! I read over the changes and they make sense to me. That being said, most of the builds are failing, seemingly related to a header file not showing up in the right spot, which seems orthogonal to these changes; I'm not sure what's going on there.

Up to you whether you prefer to merge this and close #189 or do them separately.

@alharrison
Copy link
Copy Markdown
Author

alharrison commented Mar 12, 2026

Hi Alex! I read over the changes and they make sense to me. That being said, most of the builds are failing, seemingly related to a header file not showing up in the right spot, which seems orthogonal to these changes; I'm not sure what's going on there.

Up to you whether you prefer to merge this and close #189 or do them separately.

Hey, I did see those builds failing. They seem to be new builds in the CI. The one that existed in the last PR (ubuntu_build) succeeded still. Are those somehow new? I can make an issue for one of us to investigate the failures so that it gets addressed.

I think maybe I'll merge this one and close #189 just to keep the noise and churn down.

EDIT: Added issue #191 for us to look into these build failures.

@alharrison alharrison mentioned this pull request Mar 12, 2026
@alharrison alharrison merged commit be2efdd into open-quantum-safe:OQS-v10 Mar 16, 2026
3 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants