Skip to content

Release to Prod [2.8.20]#353

Merged
johnwalz97 merged 50 commits intoprodfrom
main
Apr 16, 2025
Merged

Release to Prod [2.8.20]#353
johnwalz97 merged 50 commits intoprodfrom
main

Conversation

@johnwalz97
Copy link
Contributor

@johnwalz97 johnwalz97 commented Apr 16, 2025

Internal Notes for Reviewers

External Release Notes

AnilSorathiya and others added 30 commits March 11, 2025 11:57
* Remove ydata-profiling  library dependency

* poetry lock update

* update infer_datatypes function to infer Text type

* update test

* remove lint error

* remove print statements
* Fix index issue when using input_grid

* bug fix : show the results of all the features

---------

Co-authored-by: Anil Sorathiya <anil@validmind.ai>
* Remove ydata-profiling  library dependency

* poetry lock update

* update infer_datatypes function to infer Text type

* update test

* remove lint error

* remove print statements

* resolve warning message
* Oops, I fudged up a link in the templates

* New wording

* Changing out old links in prep pt1

* Changing out old links in prep pt2

* Changing out old links in prep pt3

* Changing out old links in prep pt4

* Adding ToC extension to template README

* Style guide tweak

* 101 +intro

* Update notebooks/tutorials/model_development/101-set_up_validmind.ipynb

Co-authored-by: Nik Richers <nik@validmind.ai>

* 2.8.13

---------

Co-authored-by: Nik Richers <nik@validmind.ai>
* support config option for logging test results

* remove commented code

* update the test

* pass config to log_metric

* remove usafe parameter from logging methods

* remove config from the log_metric
* Add initial Quarto docs assets from experimental branch

* Save point

* Save point

* Add API tree script

* Save point

* Save point

* Save point before switching to Griffe API

* Save point before splitting QMD template

* Interim commit with mostly working reimplementation of template

* Re-add more content, fix markup issues, and create sidebar template

* Save point

* Partially fix nested sidebar entries

* Save point

* Save ppoint docstring parsing

* Save point before error message fixes

* Save point

* Save point

* Save point

* Save point

* Save point for errors

* Save point before switching to vm_models

* Another save point

* Got vm_models working

* Save point with docstring issues

* Save point with partial sidebar fixes

* Save point with partial sidebar fixes

* Sidebar fragment contains correct file embeds

* Got sidebar working

* Save point

* Save point

* Save point with first complete sidebar

* Save point with first complete sidebar

* Sidebar works

* Add root-level classes

* Undo save point

* Sidebar fixes complete

* Add codeblock template comments

* Markup fixes

* Fix sidebar regression

* Fix sidebar regression

* Undo stray commit

* Save point

* Save point

* Save point

* Likely safe save point

* Halfway through switching to better signatures

* Halfway through switching to better signatures

* Continue to improve snignature blocks and remove duplication

* Remove dead code

* More signatures fine-tuning

* More signatures fine-tuning

* Save point for return annotations

* Save point with most return annotations fixed but still getting NoneNoneNoneTuple

* Save point with most return annotations fixed but still getting NoneNoneNoneTuple

* Fix docstrings

* Fix docstrings

* Fixed docstring defaults when set to None

* Save point for type annotations rework

* Undo new processing functions in script

* Fix docstring regression

* Fix docstrings in Python API

* Fix Python API docstring

* Fix Python API docstring

* Save point while fixing some docstrings

* Fixed short and long description regression

* Minor content update

* Fix type class annotations

* Fix type class annotations

* Save point types fine-tuning

* Save point types fine-tuning

* Add missing default to docstring and fix grammar

* Fix default handling in docstrings, fix class for type keywords

* Fix default handling in docstrings, fix class for type keywords

* Fix default handling in docstrings, fix class for type keywords

* Fix wording

* Output class methods on single line and append colon for classes and class methods

* Output class methods on single line and append colon for classes and class methods

* Remove colons for return annotations

* Remove colons for return annotations

* Fix variadic and positional parameters

* Save point

* Fix single line vs multiline signatures regression

* Reapply fix for ExprList

* Save point

* Save point with debug comments in types.jinja2

* Fixed final raw JSON output for Expr* types

* Save point before descending into experimental darkness

* Minor type fix

* Interim save point

* Fixed ExprAttribute rendering for inherited members

* Remove debug statements

* Save point for class names in signatures for inherited members

* Fix class for constructors that have multiple parameters

* Fix raw JSON issue with ExprCall

* Generalize Expr* type handling further

* Better decorator formatting when >1 decorator

* Add metadata and CSS files

* Fix return annotation None and improve handling of @ in decorators

* Fixed display of self in methods

* Fix Python API warnings

* Save point after fixing Python API, did it work?

* Switch from reference/ to validmind/ path to avoid breaking links

* CSS rename, latest version

* Docstring edits

* Generate QMD

* Fix CSS embed in metadata file

* Fix docstring handling bug with newlines

* Improve alias handling to catch all root-level aliases safely

* Clarify Torch TensorDataset is supported

* Remove ToC and sidebar decoration to help with long names

* Add special case for vm_models

* Fix return type annotations

* Improve type annotation coverage

* Fix for missing vm_model classes and methods

* Fix inherited members info

* Improve formatting for type_info in docstrings

* Fix ordering for class attributes

* Add special case for cleaning up tests

* Try to fix code test errors

* Fix more code test errors

* Fix yet more code test errors

* Interim save point on better control over ordering

* Remove unused _quarto.yml

* Fix return annotation for run_test_suite

* Fix some return annotations

* Minor punctuation fix

* Fix docstring for inspect()

* Remove API tree script I used to explain the API docstrings to myself

* Initial README.md draft

* Move diagram in README.md

* Start using template for errors

* Update Quarto source

* Save point for better error messages

* Better API error handling

* Error message formatting & fixes

* Interim save point for fixing inherited BaseError content

* Save point with signature and inherited member improvements

* Save point for error messages

* Interim save point for inherited member fixes

* Remove inherited members for errors from test descriptions

* Clean up inherited members

* Finally fixed the elusive description method for BaseError

* Fix minor formatting issue

* Remove hardcoding

* Save point before chewing out Sonnet

* Remove more hardcoded content after a frank discussion with Sonnet

* Consolidate signature handling

* Improve handling for class param vs params

* Improve sidebar

* Hide test suite class info from main page

* Improve filtering out non-public functions

* Add version number to sidebar & hide ToC for root function topic

* Add workflow for Quarto docs

* Add test building Quarto docs to python.yaml

* Ignore docs/** path in workflows

* Make version in sidebar fancier

* Fix missing RawData children in sidebar

* Simplify inherited member formatting

* Omit inherited members heading when there are none

* Fix return annotation for pandas.DataFrame

* Fix function signatures when async

* Handle module-level aliases like describe_test_suite

* Remove debug statement from script

* ValidMind CSS tweaks

* Update README.md

* Update mermaid diagram

* Update mermaid diagram, again

* Update mermaid diagram, take three

* Update mermaid diagram, take four

* Update mermaid diagram, take five

* Update mermaid diagram, take six

* Edit README.md

* Edit README.md

* Add linter to README.md

* Readme & CSS tweaks

* CSS tweak for codeblocks

* Fix flake8 whitespace error

* Update Quarto Markdown source

* Add dependencies to pyproject.toml and GitHub workflow

* Switch .html links in titles to .qmd

* Class prefix and function suffix CSS experiment

* More class prefix and function suffix CSS experiments

* Save point for sidebar fragment generation fixes after CSS experiments

* Fix sidebar navigation fragment after CSS changes

* Fix headings that link to themselves

* Remove class prefixes from return annotation links

* Improve handling for external types and return annotation linking

* Enable parameter type linking for internal types

* Fix case-sensitive external type link

* Add HTML to external types

* Improve TestID handling

* Improve unit metric handling in signatures

* Update docs/_metadata.yml

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>

* Fix decorators macro

* Update poetry.lock

* Fix unit tests, round 1

* Fix unit tests, round 2

* Fix unit tests, round 3

* Apply  # noqa: C901 to complex tests

* Fix invalid table format errors

* Try fixing pandas.io.formats error

* Fix TableOutputHandler to address invalid table format errors

* Fix C901 'TableOutputHandler.process' is too complex error

* Refactor test to reduce complexity and undo # noqa: C901

* Fix annotation warnings in utils.py

* Improve sidebar spacing

* Found correct global fix for smart quotes

---------

Co-authored-by: Beck <164545837+validbeck@users.noreply.github.com>
* Uploading Michael's notebook before testing

* + champion model file & moved draft to tutorials

* 111 draft

* 111 notes

* 111 adjusting

* 111 +intro

* 111 + intro

* 102 accidentally deleted a header somehow

* 111 new draft

* 111 editing intro to be validator-focused

* Edited the About ValidMind section to be more role agnostic

* Missed one

* 111 +Register a sample model

* 111 +Getting to know ValidMind

* 111 Set up for Import the champion model

* 111 Shuffling things around

* 111 Importing the champion model done?

* 111 Setup for custom context

* Moving LLM context stuff, changed my mind

* 111 Added section on finding the validation report in the platform

* 111 +ToC

* 112 setup

* 112 copied info over

* 111 more context

* 112 Setup for challenger models

* 112 Challenger models v1

* 112 Setup for custom context

* Changed dataset for custom context

* 113 setup

* 112 ToC

* 113 title adjust

* 114 setup

* 113 Prereqs

* 113 Setting up

* 113 Setting up edit

* 113 Setting up edit2

* 113 Setting up edit3

* 113 Setting up edit4

* 113 Set up for Identify qualitative tests

* 113 Identify qualitative tests editing

* 113 Run performance tests draft

* 113 warning about missing text-driven blocks

* 113 Evaluate performance of challengers WIP

* 113 Wrapping up

* 114 prereq

* 114 transfered rest of code over

* 114 transfered rest of code over edit

* 114 cleanup wip

* 114 cleanup wip2

* 114 cleanup wip3

* 114 Adjust a comparison test

* 114 Next steps draft

* 114 quick code cleanup

* 114 Run diagnostic tests done

* Implement a custom test done

* 114 Verify test runs done

* In summary done

* 114 Add test results to reporting draft

* 114 Finishing up

* LLM custom context - session info

* 114 quick edit to summary

* Removing the old notebook

* Tweak

* Tweak2

* Typo

* Comment on .gitignore

* Updating the old intro notebook just in case someone finds it

* Edits to validation key concepts + dataset transformations

* Removing test suites as well

* Tweak

* Setup for advanced notebook

* Moving the application scorecard advanced notebook

* Moving 111 to new notebook

* Moving 112 to new notebook

* Moving 113 to new notebook

* Moving 114 to new notebook

* Moving the champion model

* Shifting use case stuff around take 1

* Moving more stuff around

* Adding individual tests

* Editing

* More editing

* Adding ToC

* Ugh

* Cleaning up the What's next section

* Removing the tests causing issues

* Final edits?

* xgb_model > xgb_model_developer_champion

* Moving the introductory notebooks to own PR

* 2.8.14
bug: reverting changes to load.py that broke listing and loading tests
…-logistic-regression

[SC-7456] intro_for_model_developers.ipynb throws SHAPGlobalImportance.py error
…n and update Quarto documentation workflow to exclude specific branch.
… sections (#339)

* introduce log_text function to log qualitative text block

* introduce log_text function to log qualitative text block

* introduce log_text function to log qualitative text block

* validate parameter types in log_text interface

* convert str into text markup

* format files

* ux for log_text output

* handle html text

* checking if text is html

* support html text input
… ClassImbalance and DescriptiveStatistics tests for improved categorization.
* Remove some pdoc remnants

* Add GH_TOKEN again

---------

Co-authored-by: Andres Rodriguez <andres@validmind.ai>
juanmleng and others added 15 commits April 3, 2025 12:03
…g-in-rag-use-case

[SC-9395] Add RAG benchmarking demo notebook
* Moved the introductory notebooks to their own PR

* Renamed the old model development series

* Renamed the new model validation series

* Exporting the champion model

* Editing validation 1

* Adjusting

* Test

* WIP new validator 2

* copying some code into validator 2

* Ugh

* I give up for this week

* Validation 2 - Verify data quality assessments setup

* Validation 2 - Verify data quality assessments editing

* Validation 2 - Document test results WIP

* Validation 2 - Document test results - add

* Validation 3 - setup

* Validation 3 - Setting up WIP

* Validation 3 - Testing...

* Save point

* Save point

* Save point

* Testing something

* Testing more

* Applying Andres' fixes

* Validation 4 setup

* Testing

* Applying changes to subsequent notebooks

* Deleting old draft for validation 2

* Save point

* Save point

* Save point

* Save point

* Testing?

* Testing...

* Editing...

* Changing screencaps

* Save point

* Editing......

* Validation 2 cleanup

* Validataion 3 editing

* Save point

* Save point

* Save point

* Save point

* Validation 3 log a finding

* Validation 3 Adjusted test

* Validation 3 finding recommendation

* Validation 3 more tests

* Moving split dataset

* Validation 4 Setting up

* Validation 4 Summary & next steops

* Copied custom test stuff over

* Copied over verify test runs, adjusted next steps

* Deleting old notebook

* Validation 4 Editing custom tests

* Validation 4 Custom inline test pt1

* Validation 4 Custom inline test pt2

* Validation 4 Custom inline test pt3

* Validation 4 Custom inline test pt4

* Validation 4 Custom inline test pt5

* Validation 4 Custom inline test pt6

* Validation 4 Custom test provider pt1

* Validation 4 Custom test provider pt2

* Validation 4 Next steps typo

* Validation 4 Verify test runs setup

* Validation 4 Verify test runs edit

* Forgot the raw dataset

* Fixing broken API links

* Adding ToCs

* Next steps - Work with your validation report

* 2.8.18
* Adjustments for validation notebooks docs side

* Updating the template

* Test?

* Update docs/templates/module.qmd.jinja2

Co-authored-by: Nik Richers <nik@validmind.ai>

* Editing terminology

* 2.8.19

---------

Co-authored-by: Nik Richers <nik@validmind.ai>
* Add text support in test result object

* add custom test description example in the notebook
…-too-many-plots-for-test-description

John6797/sc 9452/add check for too many plots for test description
@CLAassistant
Copy link

CLAassistant commented Apr 16, 2025

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
5 out of 6 committers have signed the CLA.

✅ AnilSorathiya
✅ validbeck
✅ johnwalz97
✅ juanmleng
✅ cachafla
❌ nrichers
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions
Copy link
Contributor

PR Summary

This pull request enhances the documentation generation process by integrating Quarto for generating API documentation. The changes include:

  1. GitHub Actions Workflow Updates:

    • Updated .github/workflows/integration.yaml and .github/workflows/python.yaml to ignore all paths under docs/ instead of just docs/_build/.
    • Added a new step in the python.yaml workflow to generate Quarto documentation using the make quarto-docs command.
  2. New Quarto Documentation Workflow:

    • Introduced a new workflow file .github/workflows/quarto-docs.yaml to automate the generation of Quarto documentation using Griffe for API extraction and Jinja2 templates.
  3. Makefile Enhancements:

    • Added a new target quarto-docs in the Makefile to handle the generation of Quarto documentation.
    • The quarto-docs target includes steps to clean old files, generate an API JSON dump, and create Quarto documentation from templates.
  4. Documentation Directory:

    • Added a new docs/README.md file explaining the process of generating Quarto Markdown for the Python API.
    • The docs/ directory now includes templates, macros, CSS, and Quarto Markdown output for generating the ValidMind Library reference documentation.
  5. .gitignore Updates:

    • Updated .gitignore to include docs/validmind.json for Quarto docs.
  6. README Updates:

    • Updated links in README.md and README.pypi.md to point to the new documentation paths.
  7. Removal of Old Documentation:

    • Removed old HTML documentation files from docs/_build/.

These changes aim to streamline the documentation process, making it more efficient and maintainable by leveraging Quarto's capabilities.

Test Suggestions

  • Verify that the Quarto documentation is generated correctly by running make quarto-docs locally.
  • Ensure that the GitHub Actions workflows execute successfully and generate the expected documentation artifacts.
  • Check that the generated Quarto documentation is correctly integrated into the documentation site.
  • Test the new Quarto documentation generation process with different branches and ensure it ignores the specified paths.
  • Validate that the .gitignore updates prevent unnecessary files from being tracked.

@johnwalz97 johnwalz97 changed the title Generate quarto docs Release to Prod Apr 16, 2025
Copy link
Contributor Author

@johnwalz97 johnwalz97 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@cachafla @AnilSorathiya @juanmleng can you take a look and see if this looks good

@johnwalz97 johnwalz97 changed the title Release to Prod Release to Prod [2.8.20] Apr 16, 2025
Copy link
Contributor

@cachafla cachafla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice 👌

@cachafla cachafla added the internal Not to be externalized in the release notes label Apr 16, 2025
@cachafla
Copy link
Contributor

BTW, next time remember you can use the Create Prod Release action that automates this :P

@johnwalz97
Copy link
Contributor Author

BTW, next time remember you can use the Create Prod Release action that automates this :P

Ah okay... I always forget about that 😬. Will do

@johnwalz97 johnwalz97 merged commit f5bfbed into prod Apr 16, 2025
9 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal Not to be externalized in the release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants