Skip to content

Prepare for v0.6 release #876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 43 commits into
base: dev
Choose a base branch
from
Draft

Conversation

fsschneider
Copy link
Contributor

@fsschneider fsschneider commented Jun 25, 2025

This PR prepares for the upcoming v0.6.0 release of the benchmark. Depends on #874.

Specifically, it changes:

  • Rework Readme to be clearer and easier to understand.
  • Remove deprecated COMPETITON_RULES and CALL_FOR_SUBMISSION docs.
  • Backfill the changelog with the most relevant changes and highlight it in more documents.
  • Increment rules version to 0.6.0 and detail new rule changes:
    • Reduce studies from 5 to 3.
    • Remove held-out workloads.
    • Update API in documentation.
  • Integrate and describe the new version policy (Proposal: Unify Versioning for Codebase, Leaderboard, and Rules #870).
  • Minor formatting improvements.

Todos

There are a few more rule changes to detail. I also want to give the entire DOCUMENTATION.md a thorough review, focusing on readability and removing deprecated statements. The changes include, for example:

  • Remove mentions of model track?
  • Rewrite section on qualification set -> Phrase it as a recommended protocol for reporting partial results if limited resources are available. An AlgoPerf light version?
  • Update note on "how many submissions"
  • Update FAQs (for rolling leaderboard format)
  • Integrated important requirements from the COMPETITON_RULES doc (e.g., Apache 2.0 license, defining software dependencies, CLA/membership/trademark agreement?)
  • Mention that you can change the dropout value in the forward pass now.

@fsschneider fsschneider self-assigned this Jun 25, 2025
Copy link

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant