Likelihood of models with constant vs piece-wise constant rates

[ellessenne]

Hi,

I hope all is well.
I have a question about the likelihood of models with constant rates vs the likelihood of models with piece-wise constant rates:

library(msm)
twoway4.q <- rbind(
  c(-0.5, 0.25, 0, 0.25),
  c(0.166, -0.498, 0.166, 0.166),
  c(0, 0.25, -0.5, 0.25),
  c(0, 0, 0, 0)
)
cav.msm0 <- msm(
  state ~ years,
  subject = PTNUM,
  data = cav,
  qmatrix = twoway4.q,
  obstype = 1
)
cav.msm1 <- msm(
  state ~ years,
  subject = PTNUM,
  data = cav,
  qmatrix = twoway4.q,
  obstype = 1,
  pci = fivenum(cav$years)[2:4]
)
logLik.msm(cav.msm0)
#> 'log Lik.' -1993.044 (df=7)
logLik.msm(cav.msm1)
#> 'log Lik.' -2009.05 (df=28)
lrtest.msm(cav.msm0, cav.msm1)
#>          -2 log LR df p
#> cav.msm1 -32.01333 21 1

^{Created on 2025-09-18 with reprex v2.1.1}

I was expecting model cav.msm0 to be nested within cav.msm1, but the likelihood values do not seem comparable. Is this because of the expanded dataset constructed by {msm} under the hood to implement the pci argument?

Many thanks,

Alessandro

[chjackson]

It looks like your cav.msm1 model fit just hasn't converged to a maximum likelihood. Look at the confidence intervals in the printed model output - the likelihood appears to be flat with respect to the intensities and covariate effects for the 2-4, 3-4 and 3-2 transitions. If I just set pci=5 here, I get finite confidence limits and an increase in the likelihood over cav.msm0.

Expanding the dataset to include censored observations at the change times should not affect the likelihood, because we are not adding any new state data - we are just saying the state is unknown at these times.

[ellessenne]

Hm, interesting, thanks for your feedback!

Expanding the dataset to include censored observations at the change times should not affect the likelihood, because we are not adding any new state data - we are just saying the state is unknown at these times.

That's what I was expecting, thanks for confirming that.

I didn't check the confidence intervals in that example because in the actual data that motivated this question we are getting a reasonable output (CIs for the time periods coefficients are finite and relatively narrow) and no non-convergence warnings, but the likelihood for the pci model was still smaller than the likelihood of the model assuming constant rates.

Does this imply that the model with piece-wise constant intensities had actually not converged to a maximum likelihood?

[chjackson]

I'd suspect so. In any case it's usually sensible to check with different initial values and optimization settings.

[ellessenne]

Ok, thanks for the feedback. I'll keep looking into this.