Skip to content

Update benchmark.md with v1.1 results (83.3% token reduction)#38

Draft
Copilot wants to merge 2 commits intomainfrom
copilot/run-benchmark-suite
Draft

Update benchmark.md with v1.1 results (83.3% token reduction)#38
Copilot wants to merge 2 commits intomainfrom
copilot/run-benchmark-suite

Conversation

Copy link
Contributor

Copilot AI commented Feb 11, 2026

Benchmark suite rerun shows improved compression metrics. Updated documentation to reflect actual measured values.

Metrics Updated

  • Global token reduction: 74.6% → 83.3%
  • File coverage: 29 → 30 files (9,671 raw tokens measured)
  • Precision targeting: 96.7% (refined from "100% retrieval accuracy")
  • Cost efficiency: 4x → 6x reduction in API token costs

Heavyweight Module Rankings

Reordered top 5 modules by actual token count with updated compression rates:

Module Original → Compressed Rate
indexer 1,339 → 49 96.3%
parser 1,232 → 63 94.9%
query 1,224 → 108 91.2%
vector_store 1,013 → 147 85.5%
generator 839 → 23 97.3%

Average compression across top 5: 93.3% → 93.0%

Context Window Impact

Updated GPT-4-8K usage calculations based on measured 9,671 raw tokens:

  • Traditional approach: 12-15% window usage
  • AMDB approach: ~2% window usage (1,615 tokens)

Benchmark version bumped to 1.1, dated 2026-02-11.


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Co-authored-by: BETAER-08 <109971893+BETAER-08@users.noreply.github.com>
Copilot AI changed the title [WIP] Run AMDB official benchmark suite Update benchmark.md with v1.1 results (83.3% token reduction) Feb 11, 2026
Copilot AI requested a review from BETAER-08 February 11, 2026 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants