numerical overflow in calculating likelihood

sppb.msm <-msm(state_1 ~ days, subject = symbol, data = data,

•            qmatrix = qmat0, death = 3, method = "BFGS", 
  

•            control = list(fnscale = 4000, maxit = 10000),
  

•            covariates = ~ PB)
  

Error in Ccall.msm(params, do.what = "lik", ...) :
numerical overflow in calculating likelihood

[chjackson]

Again, a reproducible example might help (PB isn't in the data you sent me). Have you tried all the tricks in the "convergence failure" section of the PDF manual? Perhaps try rescaling the covariates to have SD around 1, or rescaling the time variable in months instead of years?

The error message is not helpful so I'll leave the issue open, but this kind of error message is common and really difficult to prevent in a general case.

[AlexKram]

Have the same problem with different data sets. You can get a reproducible example even with provided cav data set:

> cavsex.msm <- msm(state ~ years, subject=PTNUM, data=cav, qmatrix=Q, deathexact=4, covariates = ~age)
Error in Ccall.msm(params, do.what = "lik", ...) : 
  numerical overflow in calculating likelihood

[chjackson]

Try setting control=list(fnscale=4000) in the call to msm. Since the -2 log likelihood is around 4000 in this example, this normalises the -2 log-likelihood during optimisation to a scale of around 1, so that overflow is less likely to occur.

[chjackson]

I have experimented with doing this rescaling automatically for all model fits, but it seemed to cause worse convergence for as many examples as it improved.

[hbg26]

I'm facing the same problem when I use continuos covariates. The fnscale option doesn't seem to work with the cav dataset or with others datasets.

[chjackson]

Could you give a reproducible example please @hbg26

[hbg26]

Using the cav dataset:

cav.age.msm <- msm( state ~ years, subject=PTNUM, data = cav, qmatrix = Q, deathexact = 4, covariates = ~age, obstype = 1, control=list(fnscale=4000))

I tried using different values for fnscale and maxit but didn't work.

[chjackson]

That works for me, using

Q<-rbind(c(0,0.25,0,0.25), c(0.166,0,0.166,0.166),c(0,0.25,0.25,0), c(0,0,0,0))

and the latest CRAN msm, R 3.4.4 on Linux. If it hadn't worked I would have tried rescaling the covariate:

age10 <- (cav$age - mean(cav$age))/10

[28sophiesworld]

The code to set the allowable transitions has proven essential for me. Try playing with those numbers, e.g. change the values within
Q<-rbind

[seairis]

Great. The code of control=list(fnscale=4000) solved my problem of numerical overflow and control =list(reltol = 1e-16) solved false convergence.

[Heart-wang]

numerical overflow in calculating likelihood,,, How can i correct it？

[Heart-wang]

wsex.msm<-msm(state~years,subject=id,data=w,qmatrix=Q,covariates=~sex,method="BFGS",control=list(fnscale=4000,maxit = 10000))
Warning message:
In msm(state ~ years, subject = id, data = w, qmatrix = Q, covariates = ~sex, :
Optimisation has probably not converged to the maximum likelihood - Hessian is not positive definite.

[chjackson]

@Heart-wang A reproducible example would help others to give advice. Though please first try some of the tricks above in the thread, and in the "Convergence failure" section of the PDF manual

[Heart-wang]

When there are many states of the disease we are studying, such as the progress of the metabolic syndrome , eight states are set according to its diagnostic criteria. How should we set Q properly?