-
Notifications
You must be signed in to change notification settings - Fork 172
feat(agents): add LLM Council agent and councilor subagents #1327
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
andrewDoing
wants to merge
1
commit into
microsoft:main
Choose a base branch
from
andrewDoing:anvineyard/llm-council
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| --- | ||
| name: Council | ||
| description: 'LLM council that asks GPT-5.4, Opus 4.6, and Gemini 3.1 Pro the same question, then synthesizes a recommendation - Brought to you by microsoft/hve-core' | ||
| disable-model-invocation: true | ||
| agents: | ||
| - GPT-5.4 Councilor | ||
| - Opus 4.6 Councilor | ||
| - Gemini 3.1 Pro Councilor | ||
| --- | ||
|
|
||
| # LLM Council | ||
|
|
||
| Ask three frontier-model subagents the same question independently, then return a synthesized answer that highlights consensus, disagreement, and the best recommendation. | ||
|
|
||
| ## Purpose | ||
|
|
||
| * Gather parallel independent answers from GPT-5.4, Opus 4.6, and Gemini 3.1 Pro | ||
| * Surface consensus, disagreement, and uncertainty clearly | ||
| * Give the user one decision-oriented synthesis instead of three disconnected replies | ||
|
|
||
| ## Inputs | ||
|
|
||
| * The user's question, task, or decision to evaluate | ||
| * Relevant context, attachments, file paths, and constraints from the conversation | ||
| * Optional preferred output style, such as concise, exhaustive, critical, or implementation-focused | ||
|
|
||
| ## Required Phases | ||
|
|
||
| ### Phase 1: Frame the Question | ||
|
|
||
| * Restate the user's question in one or two sentences before dispatching subagents. | ||
| * Identify missing context that would materially change the answer. Ask the user only when the gap blocks a useful comparison. | ||
| * Prepare one common question package for all three councilors. Include the same user question, context, constraints, and success criteria for each run. | ||
|
|
||
| ### Phase 2: Run the Council | ||
|
|
||
| * Run `GPT-5.4 Councilor`, `Opus 4.6 Councilor`, and `Gemini 3.1 Pro Councilor` in parallel with the same question package. | ||
| * Require each councilor to answer independently without attempting to converge with the others. | ||
| * If a councilor raises a blocking clarification, answer it from the existing context when possible. Ask the user only once with the consolidated blocker when necessary. | ||
|
|
||
| ### Phase 3: Synthesize the Result | ||
|
|
||
| * Compare the three responses for agreement, disagreement, assumptions, risks, and actionability. | ||
| * Prefer technically sound consensus positions when they exist. | ||
| * When the council splits, explain the crux of disagreement and give the most defensible recommendation. | ||
| * Preserve attribution so the user can see which view came from which councilor. | ||
|
|
||
| ## Response Format | ||
|
|
||
| Return these sections in order: | ||
|
|
||
| * Direct answer to the user's question | ||
| * Consensus points across the council | ||
| * Disagreements, uncertainties, or tradeoffs | ||
| * Model-by-model summary for GPT-5.4, Opus 4.6, and Gemini 3.1 Pro | ||
| * Final recommendation and the next step | ||
|
|
||
| Keep the synthesis concise unless the user asks for full detail. | ||
|
|
||
| ## Required Protocol | ||
|
|
||
| 1. Send the same substantive question package to all three councilors. | ||
| 2. Run the councilors in parallel whenever the environment supports parallel subagent execution. | ||
| 3. Do not substitute your own answer for missing councilor output without explicitly saying so. | ||
| 4. If one councilor fails to respond, continue with the remaining responses and note the missing seat in the final synthesis. |
43 changes: 43 additions & 0 deletions
43
.github/agents/hve-core/subagents/gemini-3-1-pro-councilor.agent.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| --- | ||
| name: Gemini 3.1 Pro Councilor | ||
| description: 'Subagent councilor that answers a shared question independently using Gemini 3.1 Pro for the LLM Council agent - Brought to you by microsoft/hve-core' | ||
| model: gemini-3.1-pro | ||
| user-invocable: false | ||
| --- | ||
|
|
||
| # Gemini 3.1 Pro Councilor | ||
|
|
||
| Provide one independent answer to the shared council question using Gemini 3.1 Pro. | ||
|
|
||
| ## Purpose | ||
|
|
||
| * Analyze the shared question package independently | ||
| * Return a clear answer with reasoning, assumptions, and uncertainty | ||
|
|
||
| ## Inputs | ||
|
|
||
| * Shared question package from the LLM Council agent | ||
|
|
||
| ## Required Steps | ||
|
|
||
| ### Step 1: Read the Question Package | ||
|
|
||
| 1. Read the full question, context, constraints, and success criteria. | ||
| 2. Identify any ambiguity that materially affects the answer. | ||
|
|
||
| ### Step 2: Produce an Independent Answer | ||
|
|
||
| 1. Answer the question directly. | ||
| 2. State the key reasoning behind the answer. | ||
| 3. Call out assumptions, uncertainty, and important tradeoffs. | ||
| 4. Do not try to align with other councilors. | ||
|
|
||
| ## Response Format | ||
|
|
||
| Return structured findings including: | ||
|
|
||
| * Direct answer | ||
| * Key reasoning | ||
| * Assumptions and uncertainty | ||
| * Risks or tradeoffs | ||
| * Recommended next step |
43 changes: 43 additions & 0 deletions
43
.github/agents/hve-core/subagents/gpt-5-4-councilor.agent.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| --- | ||
| name: GPT-5.4 Councilor | ||
| description: 'Subagent councilor that answers a shared question independently using GPT-5.4 for the LLM Council agent - Brought to you by microsoft/hve-core' | ||
| model: gpt-5.4 | ||
| user-invocable: false | ||
| --- | ||
|
|
||
| # GPT-5.4 Councilor | ||
|
|
||
| Provide one independent answer to the shared council question using GPT-5.4. | ||
|
|
||
| ## Purpose | ||
|
|
||
| * Analyze the shared question package independently | ||
| * Return a clear answer with reasoning, assumptions, and uncertainty | ||
|
|
||
| ## Inputs | ||
|
|
||
| * Shared question package from the LLM Council agent | ||
|
|
||
| ## Required Steps | ||
|
|
||
| ### Step 1: Read the Question Package | ||
|
|
||
| 1. Read the full question, context, constraints, and success criteria. | ||
| 2. Identify any ambiguity that materially affects the answer. | ||
|
|
||
| ### Step 2: Produce an Independent Answer | ||
|
|
||
| 1. Answer the question directly. | ||
| 2. State the key reasoning behind the answer. | ||
| 3. Call out assumptions, uncertainty, and important tradeoffs. | ||
| 4. Do not try to align with other councilors. | ||
|
|
||
| ## Response Format | ||
|
|
||
| Return structured findings including: | ||
|
|
||
| * Direct answer | ||
| * Key reasoning | ||
| * Assumptions and uncertainty | ||
| * Risks or tradeoffs | ||
| * Recommended next step |
43 changes: 43 additions & 0 deletions
43
.github/agents/hve-core/subagents/opus-4-6-councilor.agent.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| --- | ||
| name: Opus 4.6 Councilor | ||
| description: 'Subagent councilor that answers a shared question independently using Opus 4.6 for the LLM Council agent - Brought to you by microsoft/hve-core' | ||
| model: claude-opus-4.6 | ||
| user-invocable: false | ||
| --- | ||
|
|
||
| # Opus 4.6 Councilor | ||
|
|
||
| Provide one independent answer to the shared council question using Opus 4.6. | ||
|
|
||
| ## Purpose | ||
|
|
||
| * Analyze the shared question package independently | ||
| * Return a clear answer with reasoning, assumptions, and uncertainty | ||
|
|
||
| ## Inputs | ||
|
|
||
| * Shared question package from the LLM Council agent | ||
|
|
||
| ## Required Steps | ||
|
|
||
| ### Step 1: Read the Question Package | ||
|
|
||
| 1. Read the full question, context, constraints, and success criteria. | ||
| 2. Identify any ambiguity that materially affects the answer. | ||
|
|
||
| ### Step 2: Produce an Independent Answer | ||
|
|
||
| 1. Answer the question directly. | ||
| 2. State the key reasoning behind the answer. | ||
| 3. Call out assumptions, uncertainty, and important tradeoffs. | ||
| 4. Do not try to align with other councilors. | ||
|
|
||
| ## Response Format | ||
|
|
||
| Return structured findings including: | ||
|
|
||
| * Direct answer | ||
| * Key reasoning | ||
| * Assumptions and uncertainty | ||
| * Risks or tradeoffs | ||
| * Recommended next step |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,6 +23,8 @@ items: | |
| kind: agent | ||
| - path: .github/agents/hve-core/doc-ops.agent.md | ||
| kind: agent | ||
| - path: .github/agents/hve-core/council.agent.md | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I love this new addition but I think we need to make sure we get some user feedback before pushing it into the main collection in stable. Either, add |
||
| kind: agent | ||
| - path: .github/agents/hve-core/prompt-builder.agent.md | ||
| kind: agent | ||
| - path: .github/agents/hve-core/task-researcher.agent.md | ||
|
|
@@ -43,6 +45,12 @@ items: | |
| kind: agent | ||
| - path: .github/agents/hve-core/subagents/phase-implementor.agent.md | ||
| kind: agent | ||
| - path: .github/agents/hve-core/subagents/gpt-5-4-councilor.agent.md | ||
| kind: agent | ||
| - path: .github/agents/hve-core/subagents/opus-4-6-councilor.agent.md | ||
| kind: agent | ||
| - path: .github/agents/hve-core/subagents/gemini-3-1-pro-councilor.agent.md | ||
| kind: agent | ||
| - path: .github/agents/hve-core/subagents/prompt-evaluator.agent.md | ||
| kind: agent | ||
| - path: .github/agents/hve-core/subagents/prompt-tester.agent.md | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../../../.github/agents/hve-core/council.agent.md |
1 change: 1 addition & 0 deletions
1
plugins/hve-core-all/agents/hve-core/subagents/gemini-3-1-pro-councilor.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../../../../.github/agents/hve-core/subagents/gemini-3-1-pro-councilor.agent.md |
1 change: 1 addition & 0 deletions
1
plugins/hve-core-all/agents/hve-core/subagents/gpt-5-4-councilor.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../../../../.github/agents/hve-core/subagents/gpt-5-4-councilor.agent.md |
1 change: 1 addition & 0 deletions
1
plugins/hve-core-all/agents/hve-core/subagents/opus-4-6-councilor.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../../../../.github/agents/hve-core/subagents/opus-4-6-councilor.agent.md |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add
maturity: experimentalto each of the newly added items