Accurate, Interdisciplinary and Transparent Structure-property Understanding with Deep Native Structural Reasoning

Overview§

SciReasoner is a multimodal scientific foundation model that performs native structural reasoning across proteins, small molecules, and inorganic crystals. The key innovation is a unified structure-aware vocabulary that discretizes domain-specific spatial, topological, and periodic information into addressable tokens, allowing the model to treat structural elements as evidence units during autoregressive reasoning.

Methodology§

Unified Structure-Aware Vocabulary§

The model defines a token set $\mathcal{V} = \mathcal{V}_{coord} \cup \mathcal{V}_{topo} \cup \mathcal{V}_{periodic}$. Structural coordinates are discretized via a learned vector quantization (VQ) over local neighborhoods, producing tokens from $\mathcal{V}_{coord}$. Topologies (bond graphs, connectivity) are encoded as sequences of edge tokens from $\mathcal{V}_{topo}$. For crystals, periodic boundary conditions are captured via offset tokens in $\mathcal{V}_{periodic}$. All tokens share an embedding space.

Model Architecture§

SciReasoner is a Transformer decoder that processes concatenated token sequences from different modalities. The attention mechanism uses relative positional biases to capture spatial relationships. The model is trained to autoregressively predict both property labels and intermediate reasoning traces in a chain-of-thought manner.

Training Objective§

Given a structural input $X$ (tokenized) and target property $y$, the model maximizes: $$\mathcal{L} = \mathbb{E}_{X,y} \left[ -\log p(y | X) - \lambda \sum_{t} \log p(r_t | X, r_{<t}) \right]$$ where $r_t$ are reasoning trace tokens. The first term is the property prediction loss, and the second encourages structured reasoning.

Implementation Details (PyTorch-style)§

import torch
import torch.nn as nn

class StructuralTokenizer:
    def __init__(self, vocab_size, coord_vocab, topo_vocab, periodic_vocab):
        self.emb = nn.Embedding(vocab_size, d_model)
    
    def tokenize_molecule(self, coords, bonds, offsets=None):
        # VQ on coordinates
        coord_tokens = self.vq_coords(coords)  # shape: (n_atoms,)
        # Encode bond graph as sequence of edge tokens
        topo_tokens = self.encode_bonds(bonds)  # shape: (n_edges,)
        tokens = torch.cat([coord_tokens, topo_tokens])
        if offsets is not None:
            periodic_tokens = self.encode_periodic(offsets)
            tokens = torch.cat([tokens, periodic_tokens])
        return tokens

class SciReasoner(nn.Module):
    def __init__(self, vocab_size, d_model=768, n_layers=12):
        super().__init__()
        self.tokenizer = StructuralTokenizer(vocab_size, ...)
        self.transformer = nn.TransformerDecoder(
            nn.TransformerDecoderLayer(d_model, nhead=12), n_layers)
        self.output_head = nn.Linear(d_model, vocab_size)
    
    def forward(self, tokens, mask=None):
        x = self.tokenizer.emb(tokens)
        x = self.transformer(x, memory=x, tgt_mask=mask)
        logits = self.output_head(x)
        return logits

Key Results§

• Gene Ontology prediction: increases Cellular Component $F_{\max}$ from 0.42 to 0.55 for low-homology proteins.
• Retrosynthesis: single-step accuracy from 0.63 to 0.72 with interpretable disconnection traces.
• Materials: separates elemental/compound phases and band-gap regimes.
• Overall: SOTA on 67/86 benchmarks; expert evaluation prefers reasoning traces over frontier LLM in 98% of cases.

Significance§

By making structural tokens inspectable, SciReasoner bridges accurate prediction with interpretable scientific inference, enabling verification and discovery of structure-property mechanisms.