Mixture models

Let us introduce a mixture of two gaussian distributions with density \[f(x) = \omega_1 \varphi(x; \mu_1, \sigma_1^2) + \omega_2 \varphi(x; \mu_2, \sigma_2^2)\] where \(\omega_1 = 1 - \omega_2 \in [0,1]\), \(\mu_1,\mu_2 \in \mathbb{R}\), \(\sigma_1^2, \sigma_2^2 \in \mathbb{R}^+\) and \(\varphi\) is the Gaussian density \[\varphi(x; \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left\{-\frac{1}{2\sigma^2}(x-\mu)^2\right\}\]

How to draw a sample from the mixture distribution?

sample_mixture <- function(size, omega, mean, sd) {
    # First sample indices which determine the Gaussian distribution used for
    # each sample.
    allocations <- sample(x = c(1, 2), size = size, replace = TRUE, prob = omega)
    
    # Sample normal variates using the indices into the mean and variance
    # vectors.  Here we are giving rnorm vectors of length n for the mean and
    # standard deviation. In this case it generates each sample wit the mean and
    # variance at the corresponding index.
    rnorm(n = size, mean = mu[allocations], sd = sigma[allocations])
}

Let us set \(\omega_1 = 0.3 = 1 - \omega_2\), \(\mu_1 = -2\), \(\mu_2 = 2\), \(\sigma_1^2 = \sigma_2^2 = 1\) and run our sampler.

num_samples <- 10000
omega <- c(0.3, 0.7)
mu <- c(-2, 2)
sigma <- c(1, 1)

samples <- sample_mixture(num_samples, omega, mu, sigma)

# Plot a histogram of the samples
library(ggplot2)
theme_set(theme_bw())
qplot(x = samples, y = ..density.., geom = "histogram", binwidth = 0.15)