Understanding Variational Inference and Visualization of Its Challenges

VI attempts to find an optimal surrogate posterior by maximizing the Evidence Lower Bound (=ELBO). The surrogate posterior acts as a replacement for the intractable true posterior. Let's look at some details. Here are the notes: https://github.com/Ceyron/machine-learning-and-simulation/blob/main/english/probabilistic_machine_learning/vi_challenges.pdf

In this video, we will look at the simple example of the Exponential-Normal Model with a latent and an observed variable. Even in this simple example with one-dimensional random variables, the marginal and therefore also the posterior is intractable, which motivates the usage of Variational Inference.

We are going to compare the probability distributions we have access to: Prior, Likelihood and Joint, as well as the ones we do not have access to (due to intractable integrals): Marginal and Posterior. This should show that latent does not necessarily have to mean not computable.

Finally, we will analyze a visualization you can also access here: https://share.streamlit.io/ceyron/machine-learning-and-simulation/main/english/probabilistic_machine_learning/vi_joint_of_exponential_normal_model_visualization.py

If you want, you can also check out the corresponding Python code: https://github.com/ceyron/machine-learning-and-simulation/blob/main/english/probabilistic_machine_learning/vi_joint_of_exponential_normal_model_visualization.py

-------

📝 : Check out the GitHub Repository of the channel, where I upload all the handwritten notes and source-code files (contributions are very welcome): https://github.com/Ceyron/machine-learning-and-simulation

📢 : Follow me on LinkedIn or Twitter for updates on the channel and other cool Machine Learning & Simulation stuff: https://www.linkedin.com/in/felix-koehler and https://twitter.com/felix_m_koehler

💸 : If you want to support my work on the channel, you can become a Patreon here: https://www.patreon.com/MLsim

-------

Timestamps:
00:00 Recap VI and ELBO
00:30 Agenda
00:52 Example: Exponential-Normal model
02:26 (1) We know the prior
04:15 (2) We know the likelihood
05:36 (3) We know the joint
06:34 (1) We do NOT know the marginal
08:15 (2) We do NOT know the (true) posterior
08:53 Why we want the posterior
09:51 Remedy: The surrogate posterior
10:31 Example for the ELBO
10:58 Fix the joint to the data
11:37 Being able to query the joint
12:56 Visualization
14:52 Outro