Transformers for Chemistry

The most widely recommended visual walkthrough of the transformer architecture: attention heads, positional encoding, and the encoder-decoder stack.

Free course teaching transformer fine-tuning, tokenisation, the Trainer API, and HuggingFace datasets. Directly applicable to ChemBERTa fine-tuning.

2-hour live coding session building a GPT-style transformer from scratch in pure PyTorch. Best way to develop mechanical understanding of attention.

Line-by-line PyTorch implementation of the original transformer paper with inline commentary.

Step-by-step guide for fine-tuning pre-trained BERT-style models for classification and regression using the Trainer API.

Model card and usage example for ChemBERTa-zinc-base-v1, including how to load the tokeniser and fine-tune on a molecular property dataset.

The original transformer paper introducing scaled dot-product attention and multi-head attention.

Pre-trains RoBERTa on 77M SMILES from ZINC and evaluates on MoleculeNet property prediction benchmarks.

MolFormer: transformer pre-trained on 1.1 billion SMILES, outperforming GNNs on several benchmarks.

Applies the transformer architecture to reaction SMILES for forward prediction and retrosynthesis.

Library with 200k+ pre-trained models including ChemBERTa, MolFormer, and ESM. Simple API for fine-tuning and inference.

Ready-to-use pre-trained chemical language model for molecular property prediction.

2.4M standardised SMILES with associated bioactivity data. Used to pre-train ChemBERTa and other chemical language models.

250k drug-like molecules in SMILES format. Standard corpus for training and evaluating chemical language models.

ChEMBL-derived benchmark for evaluating chemical language models on distribution learning and goal-directed generation tasks.