Complete results: six architectures on one learning curve

This page carries every table behind A head start from physics beats a bigger AI model. It is generated from one committed data file, so the figures in the article and the numbers here come from the same source.

The article calls the two setups learning from scratch and working with a physics head start. This page uses the names the data files carry, direct and delta. All errors are mean absolute error in kcal/mol, on the identical held-out test set of 49,235 molecules. The training pool holds 998,112 molecules with zero overlap against that test set.

Learning curves

Direct target, or learning from scratch

The wB97M-V total electronic energy minus a per-element composition fit. With no learning at all the reference sits at 22.6 to 23.3 kcal/mol.

Delta target, or the physics head start

The same quantity after a g-xTB baseline has been subtracted as well. The reference alone sits at 2.74 to 2.99 kcal/mol.

Cost of one training run

Core-hours for a single training run against the accuracy it bought. The two targets use different accounting, deliberately: on the direct target chembricks-phy-AI-2 alone is charged its g-xTB featurisation, because it is the only model that needs g-xTB there. On the delta target nobody is charged, because all six need it.

Direct target, learning from scratch: all 36 points

Frontier

Delta target, the head start: all 36 points

Frontier

Measured study cost

Process CPU accounting on production-length runs, not wall-clock estimates. MACE never early-stopped, so part of its total is protocol rather than architecture.

Interpolation against extrapolation

The training pool is built from small fragments, so molecules above 13 heavy atoms are outside the training size distribution entirely.

Methods

This is the detail the article leaves out. It is here so that every claim on that page can be checked rather than taken on trust.

The two targets, precisely

Both targets are residuals after a linear per-element composition fit, fitted on the training draw only. The direct target is the wB97M-V energy minus that fit. The delta target subtracts a g-xTB single point as well.

The composition term is not cosmetic on the delta arm. g-xTB is all-electron while this DFT uses ccECP pseudopotentials, so the raw difference between the two is dominated by core-electron energy the DFT never computed, and has a wider spread than the DFT energy itself. For HBr the g-xTB energy is minus 2,574.60 Hartree against a DFT value of minus 13.93. The per-element fit absorbs that offset before any learning starts.

The split, and the leak in the first version of it

The split originally intended for this study had complete test leakage: the file meant for testing was a strict subset of the file meant for training. The partition was rebuilt on data origin instead.

Zero overlap between training and test is asserted when the split is built and re-asserted on every load. A further 55 conformers containing arsenic, selenium or tellurium occur only in the test set, where a composition reference cannot place them, so they are excluded and all six models score the identical 49,235 molecules. The training pool is drawn as nested prefixes of one shuffle per seed, so a curve moves with data rather than with luck, and a fixed 5,000-molecule validation set is held out of every draw.

Shared protocol

Five of the six models run through one harness with identical split, reference, targets, training loop and metrics. chemprop and chembricks-phy-AI-2 use their own harnesses on the same split and the same test set, verified molecule for molecule.

Per-site loss normalisation matters here: without it a 76-atom structure would dominate a 3-atom one, and this corpus spans exactly that range. Molecules are split for reporting at 13 heavy atoms, since the training pool is built from small fragments and anything above that is genuinely outside the training size distribution.

The six models

Two of the six are ours. chembricks-phy-AI-1 is the message-passing network the study calls cmbx-nn. chembricks-phy-AI-2 is the kernel method it calls SORF, structured orthogonal random features over the FCHL19 representation with g-xTB charges, solved as ridge regression. Those internal identifiers are what the source data files use, so every number here can be traced back to them.

Physics terms in SpookyNet (repulsion, electrostatics and dispersion) were disabled, because they contribute on the absolute energy scale while the target here is a kcal/mol residual. Its nonlocal attention, the more distinctive part of the architecture, is untouched. MACE ran at its own defaults, with atomic energies zeroed because the harness already subtracts the composition reference.

The one protocol deviation, and its control

SpookyNet does not train under the shared protocol. It zero-initialises its nuclear embedding, so its output starts at exactly zero, and the shared exponential moving average keeps dragging the evaluated weights back to that state. At 1,000 molecules on the delta target it returned a 0.3% gain over the reference; with weight averaging off and a higher learning rate it returned 24.9%.

Since SpookyNet then took the best direct result and was the only model given any hyperparameter attention, the obvious worry is that its win is tuning rather than architecture. The control was to run SchNet at 5,000 molecules on the direct target, same seed, under SpookyNet's exact settings.

SpookyNet's settings make SchNet 3.2% worse, so the deviation is not a general advantage. Its result stands as a finding about the architecture.

---

Back to the article: A head start from physics beats a bigger AI model. A subset of the underlying data is public under an MIT license at chembricks/WB96MV-ORGANIC.