[✨ Triage] dotnet/runtime#126334 by mconnew - [API Proposal]: Add support for replay detection and out of sequence d ...

[MihuBot]

Triage for dotnet/runtime#126334.
Repo filter: All networking issues.
MihuBot version: 246635.
Ping MihaZupan for any issues.

This is a test triage report generated by AI, aimed at helping the triage team quickly identify past issues/PRs that may be related.
Take any conclusions with a large grain of salt.

Tool logs

dotnet/runtime#126334: [API Proposal]: Add support for replay detection and out of sequence detection to NegotiateAuthentication by mconnew
Extracted 5 search queries: NegotiateAuthentication add DetectReplay and SequenceDetection options, ISC_REQ_REPLAY_DETECT ISC_REQ_SEQUENCE_DETECT support in NegotiateAuthentication SSPI SPNEGO, NegotiateAuthentication not setting replay or sequence detection flags current behavior, WCF SPNEGO replay and out of order detection using NegotiateStream and NegotiateAuthentication, Performance and memory overhead of replay detection in NegotiateAuthentication/NegotiateStream
Found 25 candidate issues

• Issue #70982 (June 20, 2022) - [API Proposal]: Expose additional flags and validation in NegotiateAuthentication

  • Motivation: surface more of the SSPI/GSSAPI context flags and server-side validation (SPN/channel-binding/extended protection, impersonation) that NegotiateStream and HttpListener currently implement internally.
  • Discussion/conclusion: authors avoided exposing the raw internal ContextFlags enum; instead they proposed higher-level options (e.g. RequireMutualAuthentication, AllowedImpersonationLevel, ExtendedProtectionPolicy/RequiredImpersonationLevel) and returning richer status codes (TargetUnknown, BadBinding, ImpersonationValidationFailed). The work was folded into the broader NegotiateAuthentication expansions (see #69920/#70909/#70720). This is the primary prior discussion around exposing additional control/validation (where replay/sequence flags would conceptually belong).

• Issue #69920 (May 27, 2022) - [API Proposal]: Expose a high level authentication API

  • Motivation: create a public, cross-platform NegotiateAuthentication primitive (replacing internal NTAuthentication) for HTTP, Kestrel, SMTP, NegotiateStream, etc.
  • Discussion/conclusion: the proposal defined GetOutgoingBlob, status codes, and options. It became the baseline API that subsequent proposals extended (Wrap/Unwrap, MIC, impersonation/extended protection). The core API shape was reviewed/approved and served as the place to add any new negotiation flags/features like replay/sequence detection.

• PR #70720 (June 14, 2022) - Implement NegotiateAuthentication API

  • What happened: initial implementation of the public NegotiateAuthentication type was merged. Review comments surfaced naming/behavior details and cross-platform concerns.
  • Relevance: this is where the runtime started having the concrete code/abstraction that a new replay/sequence-detection option would be wired into.

• Issue / PR #70909 (June 17 → July 11, 2022) - [API Proposal] Add Wrap/Unwrap methods to NegotiateAuthentication / merged as PR #71777

  • Motivation: expose signing/encryption primitives (GSS_Wrap/GSS_Unwrap) to enable SASL/SMTP/NegotiateStream-like scenarios.
  • Discussion/conclusion: API review accepted Wrap/Unwrap shape; allocation semantics were handled via IBufferWriter (to allow callers to reuse buffers while letting implementations allocate when necessary). This is relevant because replay- and sequence-detection semantics are tied to message wrap/unwrap and per-message sequence tracking; adding replay/sequence detection will likely be implemented alongside or inside the same Wrap/Unwrap/MIC primitives.

• PR #71777 (July 7, 2022) - NegotiateAuthentication: Implement additional API surface (merged)

  • What happened: implemented Wrap/Unwrap and integrated them into runtime components (e.g., System.Net.Mail), added bits of extended-protection/impersonation plumbing.
  • Relevance: shows the code paths where replay/sequence detection would be toggled and implemented.

• Issue #86950 (May 31, 2023) - [API Proposal]: Add public Compute/Verify IntegrityCheck (MIC) methods to NegotiateAuthentication

  • Motivation: expose GSSAPI GetMIC/VerifyMIC equivalents to support protocols that use MICs directly (e.g., NegotiateStream, some SASL uses).
  • Discussion/conclusion: naming and buffer API shape settled (ComputeIntegrityCheck / VerifyIntegrityCheck, use IBufferWriter for compute). Participants discussed QOP and sequence-number parameters: GSSAPI does not expose user-supplied sequence numbers (SSPI/NTLM behave differently), so the proposal avoided exposing sequence-number parameters. The API was accepted/merged later (labels show api-approved / closed). MIC + Wrap/Unwrap are the right places to implement/trigger sequence/replay checks.

• Issue #103461 (June 14, 2024) - NegotiateAuthentication.ComputeIntegrityCheck on Windows sets an invalid QOP

  • Problem: Windows implementation passed an SSPI QOP constant intended for EncryptMessage into MakeSignature, causing SEC_E_INVALID_TOKEN for Sign-only contexts.
  • Discussion/conclusion: platform differences between SSPI and GSSAPI were surfaced. Resolution was to always pass the default QOP (0) for GetMIC (consistent with GSSAPI/Linux behavior). This demonstrates two important points for replay/sequence feature work: (1) Windows vs Unix/GSSAPI semantics differ and must be handled carefully, and (2) MIC/Wrap/Unwrap calls are sensitive to provider-specific flags — replay/sequence flags will need per-platform mapping.

• dotnet/wcf PR 4248 (April 29, 2020) - Expose LocalClientSecuritySettings.DetectReplays (WCF)

  • What happened: WCF already had an API surface to enable replay-detection (DetectReplays) and it was exposed in WCF code. The PR was merged.
  • Relevance/conclusion: concrete consumer demand exists (WCF uses SSPI ISC_REQ_REPLAY_DETECT / ISC_REQ_SEQUENCE_DETECT today). This is direct evidence that adding DetectReplay/SequenceDetection options to NegotiateAuthentication*Options has real users and is not just theoretical.

• PR #87930 (June 22, 2023) - Restructure NegotiateAuthentication implementation (managed NTLM, PAL splitting)

  • What happened: implementation refactor introduced PAL indirection and an AppContext switch to enable managed NTLM/SPNEGO on some Unix builds.
  • Relevance/conclusion: platform-specific implementations and fallbacks matter. Any new flags (replay/sequence detection) will need to be mapped to SSPI on Windows and to the appropriate GSSAPI behavior (or the managed NTLM implementation) on Unix/macOS. Expect some platform-specific handling and testing needs.

Summary / implications for the new proposal

• There is a clear precedent and API-surface evolution path: NegotiateAuthentication was designed to be extended (GetOutgoingBlob → Wrap/Unwrap → MIC). Replay/sequence detection logically belongs with the additional context flags / protection options discussed in #70982 and the Wrap/Unwrap/MIC surfaces (#70909 / #86950).
• WCF already exposes a DetectReplays setting (PR 4248), which demonstrates a concrete consumer and functional expectations (SSPI ISC_REQ_REPLAY_DETECT / ISC_REQ_SEQUENCE_DETECT semantics).
• Implementation caution: GSSAPI (Unix) and SSPI (Windows) differ in semantics (QOP, sequence-number handling, what flags exist). Prior bugs (e.g., #103461) and the managed/native split (#87930) show that cross-platform behavior needs explicit attention and tests.
• Suggested next steps drawn from prior discussions: add options to NegotiateAuthenticationClientOptions/ServerOptions (DetectReplay, SequenceDetection) as you proposed, but: (a) decide and document platform behaviors (what gets set on SSPI vs GSSAPI vs managed-NTLM), (b) wire into Wrap/Unwrap/GetMIC implementations so bookkeeping is enabled only when requested, and (c) add tests exercising Windows SSPI, GSSAPI, and managed-NTLM paths (and call out performance/memory trade-offs in docs, per your comment).