Model Interpretability

Recording

Learning Objectives

• Compute SHAP values for fingerprint-based models and map them back to molecular substructures
• Apply LIME to explain individual predictions from any black-box model
• Visualise attention weights in a GNN and critically evaluate what they do and do not tell you
• Distinguish local explanations (per prediction) from global explanations (model-wide) and choose appropriately

Key Takeaways

• Takeaway 1. SHAP values are grounded in cooperative game theory and satisfy desirable axioms (efficiency, symmetry, dummy). They are the most principled attribution method for feature-based models.

• Takeaway 2. Attention weights are not explanations — high attention on an atom does not mean it causes the predicted property. Treat attention visualisations as hypotheses to test, not ground truth.

• Takeaway 3. Interpretability methods can reveal when a model has learned a spurious correlation (e.g., a solvent artifact rather than a true structure–activity relationship), making them a quality-control tool.

• Takeaway 4. The right level of explanation depends on the audience: chemists want highlighted substructures on a 2-D drawing; regulators may need counterfactual explanations ("what would need to change to flip this prediction?").

Further Reading & Resources