Notice also that in this post p(.) is used to denote either probability, probability density or probability distribution depending on the context. The subsection marked by a (∞) are pretty mathematical and can be skipped without hurting the global understanding of this post. Finally in the third section we will introduce Variational Inference and see how an approximate solution can be obtained following an optimisation process over a parametrised family of distributions. Then in the second section we will present globally MCMC technique to solve this problem and give some details about two MCMC algorithms: Metropolis-Hasting and Gibbs Sampling. In the first section we will discuss the Bayesian inference problem and see some examples of classical machine learning applications in which this problem naturally appears. In this post we will discuss the two main methods that can be used to tackle the Bayesian inference problem: Markov Chain Monte Carlo (MCMC), that is a sampling based approach, and Variational Inference (VI), that is an approximation based approach. In large problems, exact solutions require, indeed, heavy computations that often become intractable and some approximation techniques have to be used to overcome this issue and build fast and scalable systems. Meanwhile, it can be noticed that Bayesian inference problems can sometimes be very difficult to solve depending on the model settings (assumptions, dimensionality, …). For example, Gaussian mixture models, for classification, or Latent Dirichlet Allocation, for topic modelling, are both graphical models requiring to solve such a problem when fitting the data. Bayesian inference is a major problem in statistics that is also encountered in many machine learning methods.

Writing challenge for inferences