---
abstract: 'A common bottleneck for materials discovery is synthesis. While recent methodological advances have resulted in major improvements in the ability to predicatively design novel materials, researchers often still rely on trial-and-error approaches for determining synthesis procedures. In this work, we develop a model that predicts the major product of solid-state reactions. The cardinal feature of this approach is the construction of fixed-length, learned representations of reactions. Precursors are represented as nodes on a ‘reaction graph’, and message-passing operations between nodes are used to embody the interactions between precursors in the reaction mixture. Through an ablation study, it is shown that this framework not only outperforms less physically-motivated baseline methods but also more reliably assesses the uncertainty in its predictions.'
author:
- 'Shreshth A. Malik'
- 'Rhys E. A. Goodall'
- 'Alpha A. Lee'
bibliography:
- 'references.bib'
title: Materials Graph Transformer predicts the outcomes of inorganic reactions with reliable uncertainties
---

The discovery of new materials and control of their properties is fundamental to the advancement of technology. Adjusting the structure and composition of crystals allows fine-tuning of properties such as conductivity and band gap [@DInnocenzo:2014aa; @Kalantari2011; @Gong:2014aa]. Advances in first-principles calculations and increased computational power have revolutionised the availability of materials information [@MatProj2013; @OQMD]. Simultaneously, advances in computational materials design [@Jansen:2015aa] have enabled high-throughput prediction and screening in silico [@Ludwig:2019aa].

However, due to the challenges associated with material synthesis, only a small proportion of candidates are produced in the laboratory and experimentally validated. The most common route to the formation of polycrystalline materials is solid-state synthesis [@Kanatzidis:2017aa], which typically involves calcining a mixture of solid reactants. Reactions occur through solid-state diffusion of ions, thus reactants are often milled and mixed to improve the reaction kinetics. However, the interplay between thermodynamically driven energy minimisation and kinetic factors results in products which are often difficult to predict; the field lacks the well-understood reaction mechanisms in organic chemistry. Particular procedures or reactants result in the formation of crystals with unique morphologies and compositions as they form through metastable states. Thus researchers frequently rely on a trial-and-error process of synthesis and evaluation. The development of tools to propose and evaluate the most promising synthetic pathways is, therefore, one of the biggest challenges facing materials discovery [@Aykol:2019aa; @kim2019machinelearned].

A recently released inorganic synthesis dataset has now enabled the possibility of using a data-driven approach to predict the outcomes of inorganic synthesis [@Kononova:2019aa]. To our knowledge, there exists only one study which explores this opportunity [@kim2018inorganic], which addresses the inverse problem of retrosynthesis – predicting precursors that can react to yield a target product – with an architecture that cannot be readily extended to solve the forward problem. However, save for large scale experimental validation, the accuracy of retrosynthesis models cannot be quantitatively benchmarked because there are almost infinitely many ways to synthesize a material, and reactions reported in the literature are not necessarily even the best synthetic route.

Here, we report an accurate forward reaction prediction model with reliable uncertainties. Methodologically, we leverage ideas from graph representation learning to learn the optimal representation of sets of inorganic reactants directly from data. Our work significantly outperforms baselines, and we show that model uncertainty can be robustly estimated using an ensemble. Our work is a building block towards an inorganic retrosynthesis planner that can interface directly with computational tools to close the materials design-make-test cycle.

Data and Model
==============

Solid-State Syntheses Dataset {#sec:data}
-----------------------------

We can define a generalised solid-state synthesis procedure for a target material as a sequence of processes (*actions*) performed on a set of starting materials (*precursors*). A recent study has extracted detailed information for over 19,000 such synthesis procedures from academic literature [@Kononova:2019aa]. Each synthesis procedure has four relevant fields for this problem,

1.  **Target:** The stoichiometric formula of the target material for the synthesis.

2.  **Precursors:** Defined as starting materials which explicitly share at least one element with the target material, excluding ‘abundant’ materials, i.e. those found in the air.

3.  **Processing Actions:** The sequence of synthesis actions performed on the precursors, including the relevant conditions for each action where available.

4.  **Balanced Chemical Equation:** Balanced chemical equations for the formation of the target compound provide the relevant molar ratios of precursors in the reaction mixture.

This dataset is the most extensive and highest-fidelity available for inorganic syntheses. Chemical information (precursors, targets and balanced equations) has been extracted with high accuracy (93%).

However, there are some notable limitations that must be considered when utilising it for modelling. Firstly, some procedures have actions and conditions with missing or incorrect data. Over 10% of reactions have no, or only one recorded action. This is primarily because standard procedures are often referenced rather than explicitly stated. The accuracy of the complete procedures (where all actions, conditions, and chemical information are fully correct) is 51%. Further, there is no structural information for the products. This limits physical analysis of the materials and hence a purely stoichiometric approach must be used in this work.

Nevertheless, this dataset provides a wealth of structured information on successful solid-state syntheses. Machine learning (ML) frameworks provide a unique way to assimilate empirical patterns from data. One can thus envisage the target material as being an abstract but learnable function of the precursors and processing procedure.

![image](figures/model_v2.png){width="100.00000%"}

Representation of Inorganic Reactions {#sec:rep-mat}
-------------------------------------

ML models generally operate on fixed-length inputs whilst typically our understanding of inorganic reactions is expressed in terms of variable-sized sets of precursors and sequences of processing steps. Bridging this discrepancy is a key challenge that needs to be addressed in the construction of ML algorithms for inorganic reaction prediction.

Here this process is broken down into 2 stages: 1) obtaining fixed-length representations for the precursors and the processing sequence, and 2) performing set regression on this set of precursors to predict the product given the processing sequence.

### Representation of Precursors

A naïve approach would consider representing materials as sparse vectors with components proportional to the relative amounts of its constituent elements (its stoichiometry). However, such a representation fails to capture critical correlations between different elements. In contrast, *Magpie* embeddings offer powerful general-purpose descriptors for bulk inorganic materials in the absence of structural information [@Ward:2016aa]. These are highly engineered, 145-dimensional vector representations which include as many scientific priors derived from its chemical formula as possible. These features fall into 4 groups: stoichiometric properties, elemental properties, electronic structure, and ionic compound features. We utilise these embeddings as initial precursor feature vectors.

### Representation of Processing Actions {#sec:rep-seq}

Sequence representation is a well studied problem in ML research [@seq2seq; @vaswani2017attention; @devlin2018bert]. Here we utilise single-layer Long Short-Term Memory Units (LSTMs) [@lstm] in an autoencoder architecture to obtain fixed-length representations of the processing sequences.

Reaction Graph Model {#sec:model}
--------------------

In solid-state syntheses, precursors are inherently ‘mixed’ together to form a product. Here we make use of an architecture for set regression based on the *Roost* model introduced in [@goodall2019predicting]. The model treats this process through a series of message-passing stages in which precursors appear as nodes on a dense, weighted graph. A fixed-length representation for the reaction is derived from this graph which allows for generalisation to arbitrary numbers of precursors.

This learned reaction embedding is then used to predict the elements present in the target material. The same embedding is then used to separately predict the product stoichiometry given the elements in the product. If the stoichiometry was directly predicted, an arbitrary threshold stoichiometry would have to be used to select which precursor elements are actually present in the product. The two stage approach used here is therefore necessary to differentiate between products with and without trace elements.

### Reaction Representation Learning {#sec:model-comp}

The crucial first step is to produce a fixed-length representation of the reaction mixture. Figure \[fig:model\]**a.** shows how precursors are represented as nodes on a dense, weighted graph and action sequences are transformed into a fixed-length representation using an LSTM encoder. The initial precursor feature vectors are transformed into a trainable embedding through a learnable affine transformation, $$\label{eq:embedding}
\vec{v}_i = \textbf{W}\vec{v}_{0,i} ,$$ where $\textbf{W}$ is a learnable weight matrix, and $\vec{v}_{0,i}$ is the initial precursor representation.

A series of message-passing operations (Figure \[fig:model\]**b.**) then update the precursor representations $$\vec{v}_{i}^{t+1} = U^{t}(\vec{v}_{i}^{t}, \sum\nolimits_{j\neq i}\vec{v}_{j}^{t}, \vec{u}_{seq}),$$ where $\vec{v}_i^{t}$ is the $i$th precursor feature vector at message-passing layer $t$, $\vec{v}_j^{t}$ are the other precursor features, $\vec{u}_{seq}$ is the fixed-length representations of the action sequences and $U^{t}$ is the update function.

At the heart of the update function is the soft-attention mechanism, which allows the model to capture the importance of the other precursors in the reaction to the precursor in question. *Attention* is in general a method for assigning the importance of certain features in a machine learning task. A *soft-attention* mechanism extends this idea by allowing the model to learn the attention coefficients itself [@vaswani2017attention]. This has been shown to improve graph representation learning [@velikovi2017graph]. Here, unnormalised attention coefficients are first calculated across pairs of precursors $$\epsilon_{ij} = f\left( \vec{v}_{i}^{t} \oplus \vec{v}_{j}^{t} \oplus \vec{u}_{seq}\right),$$ where $f$ is a single hidden-layer neural network, $\oplus$ is the concatenation operation, and $\vec{u}_{seq}$ is the action sequence embedding which is used as a ‘global’ state in all update steps. This gives procedural context to enable differentiation between products formed through different synthesis pathways. $\epsilon_{ij}$ are then normalised by a weighted softmax function,

$$\label{eq:weighted-softmax}
\alpha_{ij} = \frac{\displaystyle w_{ij} \exp(\epsilon_{ij})}{ \displaystyle \sum\nolimits_{k} w_{ik} \exp(\epsilon_{ik})},$$

where the weight $w_{ij}$ is the molar amount of precursor $j$ in the balanced chemical equation. This gives the model context of the quantities of each precursor in the reaction mixture. The node feature is then updated with the pairwise interactions between nodes, weighted by their attention coefficients $$\vec{v}_{i}^{t+1}= \vec{v}_{i}^{t} + \frac{1}{K} \sum_{k=1}^{K} \sum\nolimits_{j\neq i}\alpha_{ij}^{k}g^{k}\left( \vec{v}_{i}^{t} \oplus \vec{v}_{j}^{t} \oplus \vec{u}_{seq}\right).$$ Here $g$ is again a single hidden-layer neural network, and we average over $K$ attention heads, which has been shown to stabilise performance [@velikovi2017graph].

A similar attention-mediated pooling operation is then used to create a fixed-length, learned representation of the reaction $\vec{r}$ from the final graph. The model learns how much attention to pay to each precursor given its final representation and relative quantity.

### Product Prediction

The learned reaction embedding is then fed through a feed-forward neural network trained to perform multi-label element classification for the target product of the reaction. In materials synthesis, the target material cannot contain elements that are not present in the precursors. We add this inductive bias in the form of a binary *‘alchemy mask’* on the output of the network. Oxygen is assumed to be abundant and is additionally included in all masks irrespective of its presence in the precursors. Once the elements in the product have been predicted, the final task is to predict the relative amounts of each element in the product. Here, we use *Matscholar* element embeddings, $\vec{e}_i$, to represent and differentiate between constituent elements [@Tshitoyan:2019aa]. These are concatenated with the reaction representation, $\vec{r}$, to provide a contextualised query and fed through another feed-forward neural network, $h$, to predict the relative amount of element $i$, $$a_i=  h\left( \vec{e}_i \oplus \vec{r} \right).$$ These are then normalised using a softmax function to arrive at a fractional stoichiometry $$s_i= \frac{1}{N} \sum_{k=1}^{N} \text{softmax}_{i}(a^k_i),$$ where $s_i$ is the normalised stoichiometry of element $i$ in the product, and we average over $N$ predictions to increase stability and accuracy. In addition, using an ensemble of $N$ models also allow us to estimate the epistemic uncertainty by taking the variance in predictions across the models in the ensemble. We refer to this ensembled model as the reaction graph model.

Results and Discussion {#sec:results}
======================

Product Prediction {#sec:results-main}
------------------

The reaction graph model was tested on solid-state reactions with up to ten precursors. Product elemental composition was predicted with a subset accuracy of 0.940. This can be compared to a null baseline accuracy of 0.338 which would be achieved if all elements in the precursors were assumed to be present in the product. There is evidently a bias towards positive element labelling; 83.9% of elements in the precursors are also present in the product. Thus another metric of interest is the F1 score. The average F1 score achieved was 0.9910, where the score for each element is weighted by the number of true occurrences to account for the elemental imbalance.

The L1 and L2 distances between composition vectors can provide metrics for measuring material similarity. Here, the fractional stoichiometry vector of predicted products was accurate to a mean L1 distance of 0.1259, and L2 distance of 0.0729 to their true stoichiometries. An $R^2$ score of 0.9354 was achieved, which measures the correlation between actual and predicted stoichiometries.

Interpreting Model Predictions {#sec:incorrect-preds}
------------------------------

### Product Elements Prediction

In general, through the inclusion of the *alchemy mask* which prevents chemical inconsistencies, the prediction of which elements are present in the product is very accurate. The small number of erroneous predictions made by the model can typically be attributed to two areas:

\[tab:carbon-false-neg\]

                 **Carbon False Negatives**
  -------------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  **Example**    0.19 K~2~CO~3~ + 0.31 Li~2~CO~3~ + 0.5 CO~2~ + 0.25 O~2~ $\Rightarrow$ Li~0.62~K~0.38~CO~3~
  **Analysis**   Products containing carbon are rare in the dataset and ones that do are mostly carbonates. The model has a structure-agnostic approach and treats each element as a simple ratio of presence in the product. Carbonates are covalently bonded polyatomic ions, and thus do not have the same characteristics as independent elements in a crystal. This could be why a lower accuracy is seen for these reactions.

\[tab:additives\]

                 **Data Processing of Dopants**
  -------------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  **Example**    3 MoO~3~ + La~2~O~3~ $\Rightarrow$ La~2~(MoO~4~)~3~; with additives (Eu^3+^) via Eu~2~O~3~
  **Analysis**   Some products in the dataset have additives/dopants which do not appear in the target stoichiometry. However, the relevant precursors are still present in the data (Eu~2~O~3~ in this case). The model in these cases accurately predicts the presence of these elements in the product, but the additive is not accounted for in the product stoichiometry data.

### Stoichiometry Prediction

The inclusion of precursor amounts as weights is seen to provide vital information in differentiating between product stoichiometries. For example, the model distinguished well between two otherwise equivalent ceramic syntheses:

\[tab:prec-weights\]

  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                     **Utility of Precursor Weighting**
  ------------------ ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  **Reaction (A)**   $\text{PbO + 0.03 TiO\textsubscript{2} + 0.97 ZrO\textsubscript{2}}
                         \Rightarrow  \text{Pb(Zr\textsubscript{0.97}Ti\textsubscript{0.03})O\textsubscript{3}}$

  **Reaction (B)**   $\text{PbO + 0.48 TiO\textsubscript{2} + 0.52 ZrO\textsubscript{2}} 
                         \Rightarrow \text{Pb(Zr\textsubscript{0.52}Ti\textsubscript{0.48})O\textsubscript{3}}$

  **Analysis**       Here, (A) was used to train the model but (B) was in the unseen test set and was predicted with an L1 distance of less than 0.0056 from its true value. The learned reaction representation has thus effectively encoded information about how the relative amounts of precursors affects stoichiometry.
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

However, the model does have some failure modes. As many of these cases can be identified via estimation of the epistemic uncertainty we are primarily interested in explaining the worst predictions – those which have large error but low uncertainty.

Analysis of such erroneous predictions shows that they can generally be grouped into two main areas:

\[tab:over-ox\]

                   **Oxygen Over-prediction**
  ---------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  **Example**      1.5 Na~2~CO~3~ + 0.333 Co~3~O~4~ $\Rightarrow$ Na~3~CoO~2~ + 1.5CO~2~ + 0.417O~2~
  **Target**       Na: 0.5000 Co: 0.1667 O: 0.3333
  **Prediction**   Na: 0.1985 Co: 0.2692 O: 0.5323
  **Analysis**     Many errors arise from the over-prediction of oxygen content. This may be due to the over-representation of oxygen in the dataset. A large subset of products are perovskite materials where oxygen typically is present in a fractional amount of 0.6 (e.g. BaTiO~3~).

\[tab:pure-prec\]

                   **Pure Precursors and Alloys**
  ---------------- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  **Example**      8 Ni + 86 Al + 6 La $\Rightarrow$ Al~86~Ni~8~La~6~
  **Target**       Ni: 0.0800 La: 0.0600 Al: 0.8600
  **Prediction**   Ni: 0.6859 La: 0.1497 Al: 0.1642
  **Analysis**     This reaction is for the formation of an amorphous aluminium alloy anode through arc melting. Although it looks as if product prediction should be simple, *Magpie* features are not designed to work with pure elements. This could be the reason for the large errors in these cases.

Overall, many of the incorrect predictions come from either misrepresentation in the dataset or from the lack of structural information. A material is defined not only by its stoichiometry but crucially by its crystal structure. The dataset and therefore this framework is completely structure-agnostic and thus is limited in its predictive abilities. The use of predicted crystal structures from stoichiometry could be an avenue of further investigation [@Zhao:2020aa].

Benchmarking and Ablation Study {#sec:ablation}
-------------------------------

To place the accuracy of our model into context, we benchmark our reaction graph model against a baseline approach and conduct an ablation study on its key features.

The baseline model we use takes the *Magpie* precursor embeddings and concatenates them to be used as the input to a neural network with the same architecture as the output network of the reaction graph model. As with the reaction graph model, we make use of an ensemble of $N$ individual models for this baseline.

While the reaction graph model can handle variable numbers of precursors, this *Magpie* baseline approach is inherently restricted to a certain input size. Thus we limit our model to reactions with up to three precursors only, and zero-pad the input in the *Magpie* baseline model where there are less than three precursors to enable comparison.

The ablation study compares results across reaction graph models with/without the processing actions and the *alchemy mask* to discern which parts of the model design are significant. A summary of the key accuracy metrics is shown in Table \[tab:elempred\] for models in the benchmarking and ablation study. A null baseline result is also given for comparison where we assume all elements in the precursors are present in the product, and are present in their average relative amounts derived from the training data.

  -------------------------------- -------------- ---------- ---------- -------- -------- --------
  Reaction Representation             Alchemy       Subset    F1 Score    Mean     Mean    $R^2$
                                        Mask       Accuracy                L1       L2    
  *Magpie* Baseline                                 0.625      0.902                      
  Reaction Graph without Actions                    0.773      0.950                      
  Complete Reaction Graph                           0.770      0.948                      
  Null Baseline                     $\checkmark$    0.416      0.988     0.2233   0.1333   0.8406
  *Magpie* Baseline                 $\checkmark$    0.933      0.990     0.1568   0.0948   0.9050
  Reaction Graph without Actions    $\checkmark$    0.960      0.994     0.1298   0.0790   0.9148
  Complete Reaction Graph           $\checkmark$    0.960      0.994     0.1326   0.0805   0.9131
  -------------------------------- -------------- ---------- ---------- -------- -------- --------

### Reaction Representation Learning {#reaction-representation-learning}

For product element prediction, the reaction graph models are seen to perform significantly better than using *Magpie* features alone. For stoichiometry prediction, Figure \[fig:mae-threshold\] shows the proportion of predictions that are within given L1 error tolerances. A clear improvement is seen through incorporating the graph representation learning framework.

Figure \[fig:stoich\] shows plots of actual against predicted fractional stoichiometry. This figure shows how predictions are distributed about the equivalence line, and their relative confidence as measured through ensemble variance. Similar to the null baseline, the *Magpie* baseline model predictions show a lack of parity about the equivalence line.

![Threshold plot for product stoichiometry prediction. The cumulative proportion of correctly predicted stoichiometries is plotted against an L1 error threshold. A clear improvement is seen through using the learned reaction graph embedding compared to fixed *Magpie* features. No apparent improvement is observed through the inclusion of the action sequences.[]{data-label="fig:mae-threshold"}](figures/threshold_L1.png){width="0.98\linewidth"}

In contrast, the learned reaction graph embeddings show a significant improvement, both in absolute terms and in better prediction of the model uncertainty. In general, the most certain predictions are closest to the equivalence line and there is more parity. This indicates the model has assimilated some chemical knowledge. The improvement in uncertainty prediction can be seen more clearly in the confidence-error plot in Figure \[fig:confidence\]. Including the learned reaction graph embedding shows a clear decrease in the mean error of predicted products as the most uncertain predictions are removed. The almost flat line observed when using fixed *Magpie* embeddings indicates that there is little calibration between error and uncertainty for the baseline model.

![image](figures/stoich_scatter_r2.png){width="1\linewidth"}

![Confidence-error plots for stoichiometry prediction showing the mean L1 error in product stoichiometry prediction as the most uncertain predictions are removed. Uncertainty estimation is greatly improved through using the reaction graph embedding – more erroneous predictions are in general more uncertain. []{data-label="fig:confidence"}](figures/confidence_plot_L1.png){width="0.98\linewidth"}

### Alchemy Mask

The results in the lower half of the table are achieved through the inclusion of the *alchemy mask*, which effectively prevents the model from making predictions which are chemically unfeasible. This inductive bias provides the most notable increase in accuracy across the board for element prediction. This highlights the utility of incorporating physical principles into ML models.

### Processing Action Sequences {#sec:actions}

The synthesis actions are crucial in determining the major product of an inorganic reaction; different procedures on the same precursors can lead to differing products due to the interplay between thermodynamics and kinetics. However, Table \[tab:elempred\] shows the addition of the action sequences in the reaction embedding has a detrimental effect – if any – on model performance.

The level of detail of the action data used in the model can provide an explanation. Different product compositions and morphologies can be formed from even slight changes in temperature or pressure. Thus one cannot expect that using the generic, coarse terms to represent action sequences can lead to significant improvements in performance. The inaccuracies of the processing sequences in the dataset only serve to compound this effect. Thus a model that includes more granular processing procedures is required for practical applications.

Despite this, when the models were extended to the full dataset by including reactions with up to ten precursors, there was a significant improvement in accuracy through incorporating the processing actions sequences. An L1 error of 0.1357 was achieved without the actions, and an L1 error of 0.1259 with the actions. This is likely due to the greater diversity of reactions in the extended dataset, meaning the actions are more important in providing contextual information.

Conclusion
==========

In this work, a data-driven approach to inorganic reaction prediction has been investigated for the first time. A physically-motivated model has been developed which can predict major products of solid-state reactions from precursors and synthesis procedures. The key feature of this work is the generation of a learned reaction embedding which effectively encodes the reaction context for product prediction. Through an ablation study, this framework is shown to predict product stoichiometries more accurately than less physically-motivated models. Importantly, it also more reliably assesses the uncertainty in its predictions.

The reaction representation used here can also be transferred to further problems. One can envisage using it to predict the probability of success of a given synthesis procedure. This would further aid retrosynthesis; precursors and synthesis pathways which have the highest probability of success can be preferentially chosen. This would require supervision in the form of both successful and unsuccessful reactions [@Raccuglia:2016aa].

This work provides a basis for evaluating a future retrosynthesis planner for inorganic materials. For practical use, however, further work is required to differentiate between morphologies of products and more granular synthesis action steps with relevant conditions are needed. Access to greater quality and quantity of synthesis data and a concerted effort between ML practitioners and experimentalists will allow these advances.

Methods {#methods .unnumbered}
=======

Data Processing {#sec:comp-data}
---------------

Reactions with non-stoichiometric or organic precursors and targets, and reactions with pure products or only one precursor were removed from the dataset. This reduced the dataset size to 16,231 reactions with up to ten precursors, and 11,083 reactions for the ablation study with up to three precursors. This was randomly split into training and test sets with an 80:20 ratio.

Processing action sequences ranged from 0 to 16 steps in length, and were represented by 7 unique action types: {Dry Mixing, Mixing in solution, Quenching, Shaping, Heating, Drying, Liquid Grinding}.

Precursor stoichiometries were extracted along with their molar ratios from the balanced chemical equation for each reaction. Where precursors or targets were themselves a mixture of different materials/phases, their stoichiometries were added, weighted by their amounts.

Targets were extracted as 81-dimensional stoichiometric vectors. The set of elements present in the precursors was used to construct the *alchemy mask*.

Implementation Details {#sec:comp-imp}
----------------------

The model was implemented in PyTorch [@NEURIPS2019_9015] and *Matminer* was used for generating *Magpie* features from precursor stoichiometries [@matminer].

An embedding dimension of 8 and a hidden state dimension of 32 were used for the processing actions autoencoder. This lead to a sequence reconstruction accuracy of $\sim87\%$. Reconstruction accuracies of over 95% were achieved using a hidden dimension of 64 but did not significantly improve product prediction accuracy.

The learnable precursor embedding size was set to 128, and 5 message-passing layers were used, each with 3 attention heads. The hidden layer dimension was chosen to be 256 for both $f$ and $g$. The output network of the product element prediction model had hidden layer dimensions of 256, 512, 512 and 256 respectively with ReLU activation. The threshold output probability for elemental presence was chosen through maximising accuracy on a validation set. The stoichiometry prediction model had hidden layer dimensions of 256, 256, 256, 256, 128, 128 and 64 respectively with ReLU activation. These were averaged over 5 iterations. Skip connections were used to reduce the vanishing gradient effect [@resnet].

Learning rates were chosen using a heuristic search where we choose a rate one magnitude lower than that which causes the training loss to diverge. A mini-batch size of 256 and the Adam optimiser was used to train the model [@kingma2014adam]. Early stopping using an 80:20 train:validation split was used in all models to avoid over-fitting.

The ensembles used to estimate the epistemic uncertainty in both the *Magpie* baseline and reaction graph models were constructed using 5 individual models trained on the same data with different initialisation.

Data availability {#data-availability .unnumbered}
-----------------

The exact data used in this work can be found at <https://doi.org/10.6084/m9.figshare.9722159.v3>. More recent versions of the synthesis dataset are released by the original authors at <https://github.com/CederGroupHub/text-mined-synthesis_public>.

Code availability {#code-availability .unnumbered}
-----------------

An open-source repository of the code used in this work is available at <https://github.com/s-a-malik/inorg-synth-graph>.

Acknowledgements {#acknowledgements .unnumbered}
================

R.E.A.G. and A.A.L. acknowledge the support of the Winton Programme for the Physics of Sustainability.
