Add alpha and lambda (L1/L2 regularization) parameters to xgbTree and xgbDART#1415
Open
dgenio wants to merge 2 commits intotopepo:masterfrom
Open
Add alpha and lambda (L1/L2 regularization) parameters to xgbTree and xgbDART#1415dgenio wants to merge 2 commits intotopepo:masterfrom
dgenio wants to merge 2 commits intotopepo:masterfrom
Conversation
… xgbDART Adds alpha (L1 regularization) and lambda (L2 regularization) as tunable parameters to the xgbTree and xgbDART model definitions, matching the existing pattern in xgbLinear. Changes: - models/files/xgbTree.R: added lambda/alpha to parameters, grid, loop, fit, sort, tags - models/files/xgbDART.R: same changes - RegressionTests/Code/xgbTree.R: updated test grid with lambda=1, alpha=0 - RegressionTests/Code/xgbDART.R: updated test grid with lambda=1, alpha=0 - pkg/caret/inst/models/models.RData: regenerated via parseModels.R Closes topepo#1187
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Adds
alpha(L1 regularization) andlambda(L2 regularization) as tunable parameters to thexgbTreeandxgbDARTmodel definitions. These regularization parameters are valid for the tree booster in XGBoost (confirmed in xgboost#6763) but were previously only exposed inxgbLinear.Files modified:
models/files/xgbTree.R— addedlambda/alphatoparameters,grid,loop,fit(all 3 objective branches),sort, andtagsmodels/files/xgbDART.R— same changesRegressionTests/Code/xgbTree.R— updated test grid withlambda = 1, alpha = 0(XGBoost defaults)RegressionTests/Code/xgbDART.R— updated test grid withlambda = 1, alpha = 0pkg/caret/inst/models/models.RData— regenerated viaparseModels.RWhy
Closes #1187. XGBoost's
alphaandlambdaregularization parameters apply to the tree booster, not just the linear booster. Users want to tune these forxgbTree/xgbDARTviatrain()and random search.How verified
grid()produces correct columns for both grid and random searchmodels.RDatacontains updated parameter definitions (239 models total, all intact)xgbLinearwas not affectedlambda = 1, alpha = 0)Tradeoffs / risks
lambda = 1, alpha = 0) to avoid grid explosion. Full regularization range is explored via random search (10^runif(len, -5, 0)). Users can explore custom values viatuneGrid.tuneGriddata frames that omitlambda/alphawill need to add them. This is expected caret behavior when new tuning parameters are added.Scope notes (Mode A)
Changes are strictly limited to adding
alpha/lambdatoxgbTreeandxgbDART. Also addedL1 Regularization ModelsandL2 Regularization Modelstags to both models for consistency withxgbLinear.