a list of papers I have been reading so far.

[hkim89]

I have trimmed out a list of papers I have focused on reading so far although there were more than these sorted out.

• MLE based variational approximation : to understand whether I can apply to a frequentist approach to glmm as well as to understand how VA is constructed by introducing Variational parameters

  • Hall, P., Ormerod, J. T., and Wand, M. P. (2011), “Theory of Gaussian Variational Approximation for a Poisson Linear Mixed Model,” Statistica Sinica, 21, 369–389.
  • Hall, P., Pham, T., Wand, M. P., & Wang, S. S. J. (2011b). Asymptotic normality and valid inference for Gaussian variational
    approximation. Annals of Statistics, 39, 2502–2532.
  • Ormerod, J. T., & Wand, M. P. (2012). Gaussian variational approximate inference for generalized linear mixed models.
    Journal of Computational and Graphical Statistics, 21, 2–17.
  • Ormerod, J. T., and Wand, M. P. (2008), “Variational Approximations for Logistic Mixed Models,” in Proceedings
    of the Ninth Iranian Statistics Conference, Department of Statistics, University of Isfahan, Isfahan, Iran, pp. 450–467.

• EM with penalized likelihood: to see how to apply penalty terms to EM

  • M. J. Fadili and J. -. Starck, "EM algorithm for sparse representation-based image inpainting," IEEE International Conference on Image Processing 2005, 2005, pp. II-61, doi: 10.1109/ICIP.2005.1529991.

• Bayesian penalization or related: to think about how to build up penalization to fixed terms in glmm/glmm-susie. More papers relevant to this will be read together

  • Tung, Dao Thanh, Minh-Ngoc Tran, and Tran Manh Cuong. "Bayesian adaptive lasso with variational Bayes for variable selection in high-dimensional generalized linear mixed models." Communications in Statistics-Simulation and Computation 48.2 (2019): 530-543.
  • Jeon, Minjeong, Frank Rijmen, and Sophia Rabe-Hesketh. "A variational maximization–maximization algorithm for generalized linear mixed models with crossed random effects." Psychometrika 82.3 (2017): 693-716.

[gaow]

Thanks @hkim89. Do you have the PDF file for Ormerod, J. T., and Wand, M. P. (2008), “Variational Approximations for Logistic Mixed Models,” that you could share with me via slack? It claims to be a better approach than Jaakkola + Jordan (as I read from their 2012 paper) but I cannot seem to find it .. .

[gaow]

Also,

to think about how to build up penalization to fixed terms in glmm/glmm-susie. More papers relevant to this will be read together

I'm wondering what's the goal here -- this can potentially handle lots of correlated, fixed effect covariates in the model? Not sure if we ever need to do that in the context of genetic association studies because typically we control for age, sex, a few other things, and a few genotype PC. Not too much covariates to control. Am I understanding it correct?

[gaow]

Clarification from @hkim89 on slack :I try to control fixed effects including intercepts to avoid the domain error. However, this could help control other covariates you mentioned.

[hkim89]

Jakkola's VA extension:

Rijmen, Frank, and Jiří Vomlel. "Assessing the performance of variational methods for mixed logistic regression models." Journal of statistical computation and simulation 78.8 (2008): 765-779.

Bayesian penalization: how to use L2 to glmm
Fahrmeir, Ludwig, Thomas Kneib, and Susanne Konrath. "Bayesian regularisation in structured additive regression: a unifying perspective on shrinkage, smoothing and predictor selection." Statistics and Computing 20.2 (2010): 203-219.

Tung, Dao Thanh, Minh-Ngoc Tran, and Tran Manh Cuong. "Bayesian adaptive lasso with variational Bayes for variable selection in high-dimensional generalized linear mixed models." Communications in Statistics-Simulation and Computation 48.2 (2019): 530-543.

[hkim89]

Another Ormerod paper with his 2012 one I am reading. This would also be good to introduce VA for Yin & people in the meeting.
Ormerod, John T., and Matt P. Wand. "Explaining variational approximations." The American Statistician 64.2 (2010): 140-153.

[hkim89]

They have how to incorporate flat priors for covariates or fixed effects to VA for glm/glmm

Carbonetto & Stephens 2017 : varbvs: Fast Variable Selection for Large-scale Regression
Carbonetto, Peter, and Matthew Stephens. "Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies." Bayesian analysis 7.1 (2012): 73-108.
&
Starting with Ormerod et al 2008

I focus on the papers to make derivation for SuSiE-GLMM this week.

[hkim89]

With Carbonetto's papers reading, I found this to understand his flat prior for covariates since his math derivation seems to be intuitive but too brief to understand the detail:

Ibrahim, Joseph G., and Purushottam W. Laud. "On Bayesian analysis of generalized linear models using Jeffreys's prior." Journal of the American Statistical Association 86.416 (1991): 981-986.

[hkim89]

Found Micheal Jordan's lecture notes for Bayesian modeling and inference to study priors (lecture 5~10), reading to 8 (conjugate priors, Jefferys, reference priors)
https://people.eecs.berkeley.edu/~jordan/courses/260-spring10/

Seems to be helpful, but will focus on the tutorial tomorrow.

[hkim89]

Read Jordan's lecture notes above continuously to 13 (reference priors, nuisance parameters, multivariate regression, g-priors)
, revisited my old textbook, matrix analysis and applied linear algebra by Carl Meyer (orthogonal projections) and

Rakitsch, Barbara, et al. "It is all in the noise: Efficient multi-task Gaussian process inference with structured residuals." Advances in neural information processing systems 26 (2013)
to check log-likelihood evaluation. Get to understand Carbonettos' papers better than before.

[hkim89]

Read Ormerod et al 2012 in detail (methods ). Will delve into its Appendix about all math details. From time to time look back to Carbonetto's papers (getting better and better)

[hkim89]

The reference for Ormerod et al 2012- appendix: the core numerical method

Liu, Qing, and Donald A. Pierce. "A note on Gauss—Hermite quadrature." Biometrika 81.3 (1994): 624-629.

[hkim89]

I have figured out controlling covariates in Carbonetto's papers. The formulas turns out to be correct, but the model intro in the appendix for logistic regression (Peter Carbonetto, Xiang Zhou, and Matthew Stephens 2017) including typos leads me to thinking a wrong way to go and taking much time to figure out. I will check the rest of the papers and move to susie-glmm to solve for covariates' control. Hopefully this will resolve the susie-glmm's numerical instability.

[hkim89]

I wonder if you got any response from Peter. There is one last part to try to figure out: adjusting \xi (variational parameter) in the M-step by covariates which were integrated out in the earlier step. While thinking about it, I move to read Ormerod with the reference (A note on Gauss-Hermite quadrature).