Uncertainty Quantification

Knowing what you don't know — calibration, conformal prediction, deep ensembles, and Bayesian neural networks for reliable chemical ML.

Courses

3 resources

A Gentle Introduction to Conformal Prediction — Angelopoulos & Bates

The most accessible introduction to conformal prediction theory: split conformal, cross-conformal, and risk control.

arXiv:2107.07511 · Free

Uncertainty in Deep Learning — Yarin Gal (PhD Thesis)

Foundational work connecting dropout to variational inference and Bayesian deep learning.

University of Cambridge · Free PDF

Probabilistic ML — Kevin Murphy

Two-volume textbook covering probabilistic modelling, Bayesian inference, and uncertainty quantification. Free PDFs available.

MIT Press · Free PDF

Tutorials

2 resources

MAPIE Tutorial Notebooks

Official notebooks demonstrating split conformal prediction for regression, with coverage guarantees on chemistry datasets.

MAPIE docs · Free notebooks

Uncertainty Quantification with Deep Ensembles — PyTorch

Implementations of MC-Dropout, deep ensembles, and Bayes-by-backprop for uncertainty estimation in PyTorch.

Papers

4 resources

Lakshminarayanan et al. — "Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles" (2017)

Establishes deep ensembles as the gold standard for UQ, outperforming Bayesian approximations in practice.

NeurIPS 2017 · arXiv:1612.01474

Guo et al. — "On Calibration of Modern Neural Networks" (2017)

Shows that deep networks are overconfident and introduces temperature scaling as a simple post-hoc calibration fix.

ICML 2017 · arXiv:1706.04599

Hirschfeld et al. — "Uncertainty Quantification Using Neural Networks for Molecular Property Prediction" (2020)

Benchmarks UQ methods (ensembles, MC-Dropout, evidential learning) on molecular property prediction datasets.

Gal & Ghahramani — "Dropout as a Bayesian Approximation" (2016)

Shows that dropout at test time approximates Bayesian inference, enabling cheap uncertainty estimates from any dropout network.

ICML 2016 · arXiv:1506.02142

Tools

3 resources

scikit-learn-compatible library for conformal prediction. Wraps any regressor or classifier to produce valid prediction intervals with coverage guarantees.

Uncertainty Toolbox

Python library for evaluating and calibrating predictive uncertainty: calibration plots, sharpness, proper scoring rules.

Scalable Gaussian process library in PyTorch providing exact uncertainty estimates for regression.

Datasets

3 resources

Standardised molecular property benchmarks (ESOL, FreeSolv, BBBP, Tox21) widely used to evaluate calibration and conformal prediction for chemistry.

DeepChem · Free

134k small molecules with 19 quantum chemical properties. Useful for testing UQ methods on continuous regression targets with high dynamic range.

Alchemy Dataset

119k organic molecules with 12 quantum mechanical properties from DFT. Larger-scale alternative to QM9 for benchmarking UQ and GNN models.

Tencent · Free