From c71e4f3bfafb6df3820929e25117c3cd41a81ec2 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 7 Apr 2026 03:06:38 +0000 Subject: [PATCH 1/8] Update EvoTrees requirement to 0.18.5 in /docs Updates the requirements on [EvoTrees](https://github.com/Evovest/EvoTrees.jl) to permit the latest version. - [Release notes](https://github.com/Evovest/EvoTrees.jl/releases) - [Changelog](https://github.com/Evovest/EvoTrees.jl/blob/main/NEWS.md) - [Commits](https://github.com/Evovest/EvoTrees.jl/compare/v0.2.1...v0.18.5) --- updated-dependencies: - dependency-name: EvoTrees dependency-version: 0.18.5 dependency-type: direct:production ... Signed-off-by: dependabot[bot] --- docs/Project.toml | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Project.toml b/docs/Project.toml index f1728a43e..8f607a63b 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -25,4 +25,5 @@ TypedTables = "9d95f2ec-7b3d-5a63-8d20-e2491e220bb9" [compat] Documenter = "1" +EvoTrees = "0.18.5" julia = "1.6" From 212a2d38bab9bf2b43fae2ec1085288a272f7b27 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 7 Apr 2026 03:06:45 +0000 Subject: [PATCH 2/8] Update TypedTables requirement to 1.4.6 in /docs Updates the requirements on [TypedTables](https://github.com/JuliaData/TypedTables.jl) to permit the latest version. - [Release notes](https://github.com/JuliaData/TypedTables.jl/releases) - [Commits](https://github.com/JuliaData/TypedTables.jl/compare/v0.0.1...v1.4.6) --- updated-dependencies: - dependency-name: TypedTables dependency-version: 1.4.6 dependency-type: direct:production ... Signed-off-by: dependabot[bot] --- docs/Project.toml | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Project.toml b/docs/Project.toml index f1728a43e..f149a06f6 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -25,4 +25,5 @@ TypedTables = "9d95f2ec-7b3d-5a63-8d20-e2491e220bb9" [compat] Documenter = "1" +TypedTables = "1.4.6" julia = "1.6" From b6a3dbd120e6b0fc2d8a27ca6e44d0d4fd4288f1 Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Mon, 13 Apr 2026 11:54:51 +1200 Subject: [PATCH 3/8] re-export DataAPI.describe and update docs --- Project.toml | 2 ++ docs/src/evaluating_model_performance.md | 35 ++++++++++++++++++++++-- src/MLJ.jl | 1 + test/exported_names.jl | 4 +++ 4 files changed, 40 insertions(+), 2 deletions(-) diff --git a/Project.toml b/Project.toml index a216dce2f..f8ded0e16 100644 --- a/Project.toml +++ b/Project.toml @@ -6,6 +6,7 @@ authors = ["Anthony D. Blaom "] [deps] CategoricalArrays = "324d7699-5711-5eae-9e2f-1d82baa6b597" ComputationalResources = "ed09eef8-17a6-5b46-8889-db040fac31e3" +DataAPI = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a" Distributed = "8ba89e20-285c-5b6f-9357-94700520ee1b" Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f" FeatureSelection = "33837fe5-dbff-4c9e-8c2f-c5612fe2b8b6" @@ -32,6 +33,7 @@ Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c" [compat] CategoricalArrays = "1" ComputationalResources = "0.3" +DataAPI = "1.16" Distributions = "0.21,0.22,0.23, 0.24, 0.25" FeatureSelection = "0.2" MLJBalancing = "0.1" diff --git a/docs/src/evaluating_model_performance.md b/docs/src/evaluating_model_performance.md index 1af22243d..c186455de 100644 --- a/docs/src/evaluating_model_performance.md +++ b/docs/src/evaluating_model_performance.md @@ -48,8 +48,8 @@ machine potentially change. ) Multiple measures are specified as a vector: ```@repl evaluation_of_supervised_models -evaluate!( - mach, +performance_evaluation = evaluate( + model, X, y; resampling=cv, measures=[l1, rms, rmslp1], verbosity=0, @@ -58,6 +58,37 @@ evaluate!( [Custom measures](@ref) can also be provided. + +## Multiple models + +To create a short named tuple summary of a performance evaluation, one can apply the +`describe` method: + +```@repl evaluation_of_supervised_models +describe(performance_evaluation) +``` + +This is useful when tabulating performance evaluations for multiple models, which you can +do by providing a vector of models in place of `model` in your `evaluate` command. The +models can also include tags to appear in the final table, as shown in the following +example: + +```@repl evaluation_of_supervised_models +performance_evaluations = evaluate( + ["const" => ConstantRegressor(), "ridge" => model], X, y; + resampling=cv, + measures=[l1, rms, rmslp1], + verbosity=0, +) +table = describe.(performance_evaluations); +pretty(table) +``` + +!!! info + + The `describe` method assumes you have at least MLJBase 1.13.0 installed. + + ## Specifying weights Per-observation weights can be passed to measures. If a measure does not support weights, diff --git a/src/MLJ.jl b/src/MLJ.jl index 963357e16..02afb3271 100644 --- a/src/MLJ.jl +++ b/src/MLJ.jl @@ -84,6 +84,7 @@ import Random: AbstractRNG, MersenneTwister using ProgressMeter using ComputationalResources using ComputationalResources: CPUProcesses +@reexport import DataAPI.describe # extended by MLJBase >= 1.13.0 # to be extended: import MLJBase: fit, update, clean!, fit!, predict, fitted_params, diff --git a/test/exported_names.jl b/test/exported_names.jl index b0c05c92e..e95dda078 100644 --- a/test/exported_names.jl +++ b/test/exported_names.jl @@ -47,4 +47,8 @@ log_score @test Transformer <: Unsupervised +# DataAPI + +describe + true From 8347224ba7a6f6607e35aa4a7cedc897fa4e90b3 Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Mon, 13 Apr 2026 12:39:45 +1200 Subject: [PATCH 4/8] further doc updates --- docs/src/evaluating_model_performance.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/src/evaluating_model_performance.md b/docs/src/evaluating_model_performance.md index c186455de..fb6a48621 100644 --- a/docs/src/evaluating_model_performance.md +++ b/docs/src/evaluating_model_performance.md @@ -5,12 +5,8 @@ selected losses or scores, using the [`evaluate`](@ref) or [`evaluate!`](@ref) m For more on available performance measures, see [Performance Measures](performance_measures.md). -In addition to hold-out and cross-validation, the user can specify -an explicit list of train/test pairs of row indices for resampling, or -define new resampling strategies. - -For simultaneously evaluating *multiple* models, see "[Comparing models of different type -and nested cross-validation](@ref explicit)". +In addition to hold-out and cross-validation, the user can specify an explicit list of +train/test pairs of row indices for resampling, or define new resampling strategies. For externally logging the outcomes of performance evaluation experiments, see [Logging Workflows](@ref) @@ -62,7 +58,7 @@ performance_evaluation = evaluate( ## Multiple models To create a short named tuple summary of a performance evaluation, one can apply the -`describe` method: +[`describe`](@ref) method: ```@repl evaluation_of_supervised_models describe(performance_evaluation) @@ -84,9 +80,14 @@ table = describe.(performance_evaluations); pretty(table) ``` +One can also wrap a collection of models as a single "metamodel", which always represents +the model with the best performance evaluation estimate for the data it is trained. See +"[Comparing models of different type and nested cross-validation](@ref explicit)". + + !!! info - The `describe` method assumes you have at least MLJBase 1.13.0 installed. + The [`describe`](@ref) method assumes you have at least MLJBase 1.13.0 installed. ## Specifying weights @@ -196,5 +197,6 @@ MLJBase.evaluate! MLJBase.evaluate MLJBase.PerformanceEvaluation MLJBase.CompactPerformanceEvaluation +describe default_logger ``` From c5e6f552eb63413d1d4391016c7ecd2db2a43a77 Mon Sep 17 00:00:00 2001 From: "Anthony D. Blaom" Date: Mon, 13 Apr 2026 12:41:16 +1200 Subject: [PATCH 5/8] bump 0.23.2 --- Project.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Project.toml b/Project.toml index f8ded0e16..15746fbd9 100644 --- a/Project.toml +++ b/Project.toml @@ -1,6 +1,6 @@ name = "MLJ" uuid = "add582a8-e3ab-11e8-2d5e-e98b27df1bc7" -version = "0.23.1" +version = "0.23.2" authors = ["Anthony D. Blaom "] [deps] From 4a9599b6fb9c1abebe23a2b7a9c6f317b3ef832b Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 Apr 2026 00:55:55 +0000 Subject: [PATCH 6/8] Update ScientificTypesBase requirement to 3.1.0 in /docs Updates the requirements on [ScientificTypesBase](https://github.com/JuliaAI/ScientificTypesBase.jl) to permit the latest version. - [Release notes](https://github.com/JuliaAI/ScientificTypesBase.jl/releases) - [Commits](https://github.com/JuliaAI/ScientificTypesBase.jl/compare/v1.0.0...v3.1.0) --- updated-dependencies: - dependency-name: ScientificTypesBase dependency-version: 3.1.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] --- docs/Project.toml | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Project.toml b/docs/Project.toml index f149a06f6..b9d1158e0 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -25,5 +25,6 @@ TypedTables = "9d95f2ec-7b3d-5a63-8d20-e2491e220bb9" [compat] Documenter = "1" +ScientificTypesBase = "3.1.0" TypedTables = "1.4.6" julia = "1.6" From b6d6df305481a41ee0e380fd186ea0a5eab958f4 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 Apr 2026 00:55:58 +0000 Subject: [PATCH 7/8] Update IterationControl requirement to 0.5.4 in /docs Updates the requirements on [IterationControl](https://github.com/JuliaAI/IterationControl.jl) to permit the latest version. - [Release notes](https://github.com/JuliaAI/IterationControl.jl/releases) - [Commits](https://github.com/JuliaAI/IterationControl.jl/compare/v0.1.0...v0.5.4) --- updated-dependencies: - dependency-name: IterationControl dependency-version: 0.5.4 dependency-type: direct:production ... Signed-off-by: dependabot[bot] --- docs/Project.toml | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Project.toml b/docs/Project.toml index f149a06f6..95508715e 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -25,5 +25,6 @@ TypedTables = "9d95f2ec-7b3d-5a63-8d20-e2491e220bb9" [compat] Documenter = "1" +IterationControl = "0.5.4" TypedTables = "1.4.6" julia = "1.6" From f4cb409a3a4de436e878cf27b9afeaacbbdc1933 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 13 Apr 2026 00:56:01 +0000 Subject: [PATCH 8/8] Update MLJClusteringInterface requirement to 0.1.13 in /docs Updates the requirements on [MLJClusteringInterface](https://github.com/JuliaAI/MLJClusteringInterface.jl) to permit the latest version. - [Release notes](https://github.com/JuliaAI/MLJClusteringInterface.jl/releases) - [Commits](https://github.com/JuliaAI/MLJClusteringInterface.jl/compare/v0.1.0...v0.1.13) --- updated-dependencies: - dependency-name: MLJClusteringInterface dependency-version: 0.1.13 dependency-type: direct:production ... Signed-off-by: dependabot[bot] --- docs/Project.toml | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/Project.toml b/docs/Project.toml index 39fa6ab31..045dd9de9 100644 --- a/docs/Project.toml +++ b/docs/Project.toml @@ -26,5 +26,6 @@ TypedTables = "9d95f2ec-7b3d-5a63-8d20-e2491e220bb9" [compat] Documenter = "1" EvoTrees = "0.18.5" +MLJClusteringInterface = "0.1.13" TypedTables = "1.4.6" julia = "1.6"