Goodness of fit test?

[jarbet]

I know that plot(fit) can visually compare the estimated parametric survival curve with a nonparametric Kaplan-Meier curve, but I'm wondering if it's possible to do a type of "goodness of fit test" to check whether a parametric AFT model fits the data well?

For example, the afttest package provides a global goodness of fit test for semiparametric AFT models fit from the aftgee R package (small pvalue means there's evidence that your model does not fit the data well). I'm wondering if it would be possible to make a similar goodness of fit test for flexsurvreg?

I know these types of tests are not super useful when the sample size is massive as the GOF pvalue is almost always < 0.05.... but could still be useful for more moderately sized datasets.

[chjackson]

I'm not very familiar with methods like these. I've never seen much use for goodness-of-fit tests, partially for the reason you say about them being strongly influenced by sample size.

For inclusion in flexsurv, I think such a test would have to be usable for any parametric survival models, not just accelerated failure time families.

The principle of comparing survival to nonparametric estimates like KM curves has limited use for models with covariates - where we'd have to do the nonparametric estimation for subgroups of the data that might be small.

I think any useful measure of goodness-of-fit has to account for the purpose the model will be used for. For models used for clinical risk prediction, I think the more common practice now is to use measures of calibration and discrimination, in a cross-validation framework (rather than goodness-of-fit hypothesis testing). Things like C-index, Brier scores, etc. See e.g. SurvMetrics for an R package that does these, though I have not used it.

If the model is then used for clinical decision making, I think decision curve / net benefit analysis is particularly well suited.

Good question anyway, and I will leave the issue open for discussion!

[jarbet]

I think any useful measure of goodness-of-fit has to account for the purpose the model will be used for. For models used for clinical risk prediction, I think the more common practice now is to use measures of calibration and discrimination, in a cross-validation framework (rather than goodness-of-fit hypothesis testing). Things like C-index, Brier scores, etc. See e.g. SurvMetrics for an R package that does these, though I have not used it.

Yep this is true when the goal is prediction.

However, when the goal is inference on predictor effects, then it's unclear how to best evaluate whether a given parametric model is appropriate or not (i.e. whether the model's assumptions are roughly satisfied). I can use AIC to choose between different competing parametric models, but that doesn't really answer whether the model fit is adequate.

I suppose one option would be to compare the AIC of my parametric AFT to the AIC of a CoxPH. If they are close or the parametric AIC is better, then I guess that suggests my model fit is okay, at least okay relative to the most widely used survival model. EDIT: actually I don't think AIC is comparable between parametric AFT and CoxPH?

Lastly, another option is I could just use a flexible parametric model from your package but use robust standard errors to help ensure the inference is valid. Does your package support robust standard errors? Issue #208 suggests it is implemented in the development version of the package?

[chjackson]

I think I'd approach this by identifying particular questionable assumptions, then comparing AICs with an equivalent model where those assumptions are relaxed. For example, more flexible parametric families, or models with additional covariates.... I don't think AICs would be comparable between a parametric and Cox model, as the Cox model is based on a partial likelihood. Even in a Cox model, there is the assumption of proportional hazards (flexsurvspline can relax this though!)

Then I'd do one or two comparisons with KM curves for an overall check of practical goodness-of-fit. For models with covariates, I'd suggest comparing the KM curve with predictions standardised over the observed distribution of covariates. flexsurv has a function standsurv for generating standardised predictions. If you are interested in particular covariates, you might standardise over particular sub-populations of interest.

I think this approach is clearer than goodness-of-fit significance testing. It isn't clear to me what p-value threshold would be appropriate to declare adequacy, given the sample size dependence - if it's 5%, then at what sample size do you judge that close fits are close enough in practice despite the rejection of the null?

The robust standard errors in the development version use the formula from Lin and Wei. These were added for use in weighted estimation, and are only available for the models that have analytic likelihood derivatives, that is the Weibull, Gompertz and the hazard spline model.