Skip to content

feat(agents): add enablement dimension to Experiment Designer for code-with MVEs#1416

Open
eedorenko wants to merge 12 commits intomicrosoft:mainfrom
eedorenko:eedorenko/experiment-designer-enablement
Open

feat(agents): add enablement dimension to Experiment Designer for code-with MVEs#1416
eedorenko wants to merge 12 commits intomicrosoft:mainfrom
eedorenko:eedorenko/experiment-designer-enablement

Conversation

@eedorenko
Copy link
Copy Markdown
Contributor

In code-with engagements (ISE or similar), MVEs serve a dual purpose: validate feasibility and enable the customer to own the outcome independently. The Experiment Designer agent previously focused entirely on the validation side — hypothesis formation, vetting, and experiment design — without prompting the user to think about whether the customer would leave the engagement able to replicate the work.

This PR adds enablement-aware guidance throughout the agent's phases and companion instructions, drawn from a real ISE engagement designing an MVE for Azure Confidential Computing migration.

Changes

Agent: Experiment Designer (experiment-designer.agent.md)

  • Phase 1 (Discovery): added two probing questions — whether this is a code-with engagement and what the customer's current knowledge level is. Added guidance that code-with MVEs should reflect a dual purpose in the problem statement.
  • Phase 1 (Context tracking): added enablement goal as a captured field in context.md.
  • Phase 3 (Red Flag Checklist): added Show without teach — flags engagements where the customer watches but does not participate in building.
  • Phase 4 (Experiment Design): added Enablement Design section for code-with engagements covering pairing structure, ownership progression (ISE leads → joint → customer leads), knowledge transfer checkpoints, and enablement as a measurable success criterion.
  • Phase 5 (MVE Plan): added enablement plan to the mve-plan.md contents list.
  • Coaching Style: reinforced that the customer leaving unable to replicate the outcome is a failure mode even if all hypotheses are validated.

Instructions: Experiment Designer (experiment-designer.instructions.md)

  • Added MVE as Enablement section under "What is an MVE" defining the dual-purpose model with five principles: joint work from scratch, full-stack understanding, ownership progression, enablement as measurable outcome, and embedded knowledge transfer.
  • Added Show without teach to the Red Flags list — a demo disguised as an experiment.
  • Added Customer as passive observer to Common Pitfalls — designing the experiment so the customer watches instead of drives.

Notes

The enablement insight emerged from designing an MVE for a real customer migrating AI inference workloads from AWS TEVM to Azure Confidential Computing. The customer's engineering team needed to leave the engagement owning the full Azure CC stack (AKS, attestation, SKR, GPU CC mode), not just seeing a validated architecture. Prior ISE research was preparation, not scope reduction — all work was done jointly from scratch.

…-with MVEs

Adds dual-purpose (validate + enable) guidance for MVEs conducted as
code-with engagements. In these contexts, the customer must leave the
MVE owning the full technology stack, not just seeing a working demo.

Agent changes:
- Phase 1: added probing questions for code-with context and customer
  knowledge level
- Phase 3: added 'Show without teach' red flag
- Phase 4: added Enablement Design section with pairing structure,
  ownership progression, and knowledge transfer checkpoints
- Phase 5: added enablement plan to mve-plan.md contents
- Coaching Style: reinforced dual purpose and flagged passive customer
  as a failure mode

Instructions changes:
- Added 'MVE as Enablement' section defining the dual-purpose model
- Added 'Show without teach' to Red Flags
- Added 'Customer as passive observer' to Common Pitfalls
@eedorenko eedorenko requested a review from a team as a code owner April 20, 2026 22:34
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Apr 20, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.16%. Comparing base (57ea279) to head (ed24cfb).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1416      +/-   ##
==========================================
- Coverage   87.44%   87.16%   -0.28%     
==========================================
  Files          68       67       -1     
  Lines       10335    10067     -268     
==========================================
- Hits         9037     8775     -262     
+ Misses       1298     1292       -6     
Flag Coverage Δ
pester 84.79% <ø> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eedorenko
Copy link
Copy Markdown
Contributor Author

@mattdot

Copy link
Copy Markdown
Member

@bindsi bindsi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good additions to the agent/instructions to give more accuracy and constraints. Thanks

Comment thread .github/agents/experimental/experiment-designer.agent.md Outdated
Comment thread .github/agents/experimental/experiment-designer.agent.md Outdated
Replace ISE-specific references with 'forward deployed engineering (FDE)'
and generic 'engineering team' / 'guiding team' language so the agent is
applicable to any engineering team using the extension.

Addresses review feedback from @mattdot.
@eedorenko eedorenko requested a review from mattdot April 22, 2026 23:05
@WilliamBerryiii WilliamBerryiii changed the title feat(agent): add enablement dimension to Experiment Designer for code-with MVEs feat(agents): add enablement dimension to Experiment Designer for code-with MVEs Apr 23, 2026
Copy link
Copy Markdown
Contributor

@katriendg katriendg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, great additions. Only comments left are around generalizing more. Taking into account this could be used by any type of teams collaborating together.

Comment thread .github/agents/experimental/experiment-designer.agent.md Outdated
Comment thread .github/agents/experimental/experiment-designer.agent.md Outdated
Comment thread .github/agents/experimental/experiment-designer.agent.md Outdated
Replace 'code-with', 'FDE', and 'customer' with generic terms:
- 'code-with engagement' → 'collaborative engagement'
- 'customer' → 'partner team' (in enablement sections)
- 'guiding team' → 'advisory team'
- Section headers updated accordingly

Makes the enablement concepts applicable to any engineering team,
not predicated on a specific engagement model.

Addresses review feedback from @mattdot and @katriendg.
@eedorenko eedorenko requested a review from katriendg April 23, 2026 20:15
Copy link
Copy Markdown
Contributor

@katriendg katriendg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @eedorenko, this looks good to me other than the committed json files which I believe should be removed, they don't fit this PR.
Once that is done, all good for me.

Comment thread beval/results/identity-test.json Outdated
Comment thread beval/results/results.json Outdated
@katriendg
Copy link
Copy Markdown
Contributor

@mattdot could you please re-review this one? Thanks.

Remove beval/results/*.json artifacts that were accidentally committed.
Revert the cspell workaround that was only needed for those files.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants