Posts

Why Do We Have Seasons?

Earth’s 23.5° tilt is the secret — not distance from the Sun.

Correlation Modeling with LKJ Priors

Modeling correlations using LKJ priors and Cholesky decomposition.

Gaussian Mixture Models with ADVI

2D Gaussian mixtures with custom logp, Metropolis sampling, and mini-batch ADVI.

variational-inference

Marginalizing Over Discrete Variables

The log-sum-exp trick for marginalizing out cluster assignments in mixture models.

variational-inference

Mixture Models and MCMC

Gaussian mixtures, label-switching, and identifiability in MCMC sampling.

Two-Component Gaussian Mixture

Identifiability and label-switching in two-component Gaussian mixtures.

variational-inference

Variational Inference

Approximating posteriors through optimization instead of sampling.

variational-inference

Variational Autoencoder

Generative modeling with VAEs on MNIST using PyTorch.

neural-networks

variational-inference

Variational Inference with Neural Networks

Bayesian neural network classification using ADVI in PyMC.

neural-networks

variational-inference

Automatic Differentiation Variational Inference

From CAVI to scalable black-box VI with automatic differentiation.

variational-inference

Bayesian Workflow: Zero-Inflated Poisson

Prior selection, simulation-based calibration, and posterior predictive checks for GLMs.

Model Comparison Continued

WAIC, cross-validation, LOOCV, and practical tips for choosing between models.

information-theory

Model Comparison

From KL-divergence to deviance — AIC, DIC, and WAIC for comparing models.

information-theory

GP Recap and Salmon Example

A condensed recap of GP theory applied to sockeye salmon recruitment.

gaussian-processes

Levels of Bayesian Analysis

From MLE to Full Bayes — each level trades optimization for integration.

The Idea Behind the Gaussian Process

From multivariate normals to kernel-based function priors and predictive distributions.

gaussian-processes

Poisson Regression — Model Comparison and Hierarchical Overdispersion

WAIC model comparison, ensemble averaging, and varying-intercepts for overdispersed counts.

Inference for Gaussian Processes

Hyperparameter learning via empirical Bayes and MCMC with PyMC.

gaussian-processes

Gaussian Processes and Non-parametric Bayes

From kernel trick to infinite basis functions — the theory behind GP regression.

gaussian-processes

Geographic Correlation and Oceanic Tools

Gaussian process covariance captures spatial correlation between island societies.

gaussian-processes

Choosing Priors: From Flat to Weakly Informative

Uninformative, Jeffreys, and regularizing priors for Bayesian inference.

Bayesian Regression with Normal Models

From heights to height-weight relationships using conjugate priors and posterior predictives.

Poisson Regression: Modeling Tool Diversity Across Islands

Centering, correlations, and counterfactual posteriors in Oceanic tool kit evolution.

Formal Convergence Tests for MCMC Chains

Gewecke, Gelman-Rubin, and Effective Sample Size diagnostics.

Identifiability in Bayesian Models

How parameter redundancy breaks MCMC convergence — and how priors can help.

Hierarchical Bayesian Modeling: The 8 Schools Example

Learning to pool information across groups with non-centered parametrization.

HMC/NUTS Tuning and Diagnostics

Why centered models diverge and non-centered models don’t — a hands-on HMC exploration.

Gelman Schools and Hierarchical Pathology

Diagnosing and fixing funnel pathology in hierarchical models with reparameterization.

Exploring Hamiltonian Monte Carlo

From typical sets to leapfrog integration: a detailed exploration of HMC and NUTS.

A Tetchy Gibbs Sampler

Gibbs sampling on a bimodal, highly-correlated posterior with lots of autocorrelation.

Data Augmentation

Iterative sampling via hidden variables using the Tanner-Wong algorithm and Gibbs.

The Idea of Hamiltonian Monte Carlo

Using momentum and energy conservation to glide efficiently through the typical set.

Imputation and Convergence

A coal-mine disaster switchpoint model for convergence testing and posterior predictive checks.

Metropolis and Support Mismatch

Why combining rejection with Metropolis sampling wastes efficiency.

Gibbs Sampling with Conjugate Conditionals

Exploiting conjugate prior pairs for efficient Gibbs sampling.

Gibbs from Metropolis-Hastings

Proving that Gibbs sampling is a special case of MH with acceptance probability one.

The Metropolis-Hastings Algorithm

Sampling from complex distributions with asymmetric proposals and limited support.

From Annealing to Metropolis

Understanding detailed balance and the Metropolis algorithm for sampling probability distributions.

Markov Chains and MCMC

Stationarity, reversibility, and the mathematical foundation for Metropolis sampling.

Discrete MCMC

Metropolis sampling from discrete distributions using proposal matrices.

Introduction to Gibbs Sampling

MCMC via alternating conditional sampling from a complex joint distribution.

Hierarchical Bayesian Models

From partial pooling to shrinkage estimation with the rat tumors example.

The EM Algorithm

Inferring parameters from incomplete data through iterative expectation and maximization.

Mixture Models and Types of Learning

From generative classifiers to clustering: supervised, unsupervised, and semi-supervised approaches.

Utility, Risk, and Decision Theory

From utility functions to Bayes actions: applying decision theory to regression and model comparison.

decision-theory

Generative vs Discriminative Models

Understanding discriminative boundaries vs. generative class models through logistic regression and LDA.

How Sigmoids Combine

Visualizing neural network function approximation through hidden layer activations.

neural-networks

Multi-Layer Perceptron for Classification

Training neural networks to classify synthetic data with decision boundaries.

neural-networks

Regression in PyTorch

Building and training multi-layer perceptrons as universal function approximators.

neural-networks

Logistic Regression and Backpropagation

From MLE classification to computing gradients through layers.

Gradient Descent and SGD

Optimization by following the slope downhill.

From the Normal Model to Regression

Building a Bayesian regression from Kung San census data.

Bayesian Regression

Putting priors on regression coefficients and updating with data.

The Normal Model

Conjugate priors for the workhorse of statistics.

The Inverse Transform

Turning uniform random numbers into any distribution you want.

Importance Sampling

Sample where it matters most to compute integrals efficiently.

Lab: The Beta-Binomial Globe Model

Grid approximation, quadratic approximation, and MCMC in practice.

Rejection Sampling

Accept or reject: a simple algorithm for hard distributions.

The Beta-Binomial Globe Model

Conjugate priors, Bayesian updating, and decision theory on a globe toss.

Sufficient Statistics and Exchangeability

When less data tells you everything, and order doesn’t matter.

Bayesian Statistics

Treating parameters as random variables changes everything.

Entropy and Maximum Entropy

Quantifying uncertainty and the distributions it favors.

information-theory

Convexity and Jensen’s Inequality

Why the average of a function isn’t the function of the average.

Divergence and Deviance

Measuring how far your model is from the truth.

information-theory

Understanding AIC

An information-theoretic shortcut for model selection.

information-theory

The Significance and Size of Effects

When a drug works, how much does it matter?

Learning With Noise

Bias, variance, and the tradeoff that haunts every model.

Regularization

Taming complexity by penalizing parameters.

Learning Bounds and the Test Set

How to honestly evaluate what your model has learned.

Learning Without Noise

What happens when you fit a model to perfect data.

Validation and Cross-Validation

Why one split is never enough.

Frequentist Statistics

Fixed parameters, random data — the frequentist creed.

Maximum Likelihood Estimation

Find the parameters that make your data most probable.

Monte Carlo Integration

When calculus is hard, sample instead.

Basic Monte Carlo

Let randomness do the heavy lifting.

Expectations and the Law of Large Numbers

What you expect is what you get — eventually.

Sampling and the Central Limit Theorem

Why everything looks normal in the limit.

Distributions Example: Elections

Simulating a presidential election with coin flips.

Distributions

The shapes that randomness takes.

Probability

From coin flips to Bayes’ theorem.

Box’s Loop

Build, compute, critique, repeat.

Some Data Analysis about Congress

Does the president’s party always lose seats in congress?

The LLN

Flip enough coins and the truth emerges.

Visualization As Story

Don’t make your audience think.

Discuss this post

X / Twitter Bluesky LinkedIn