onepot-Bench 0: towards lab-aware in silico chemistry benchmarks

Overview§

onepot-Bench 0 is a benchmark suite designed to evaluate LLMs on tasks directly relevant to synthetic chemistry in a laboratory setting. It addresses two key issues in existing benchmarks: they often do not measure wet-lab-specific reasoning, and they rely on public data that may be contaminated via model pretraining. The suite comprises three complementary evaluations: ChemAbacus, SynthRefusal, and SynthBench.

Benchmark Components§

ChemAbacus§

ChemAbacus probes tool-free cheminformatics literacy and numerical reasoning. Tasks likely include calculating molecular weights, stoichiometric ratios, percent yields, and interpreting chemical formulas without external tools. This tests whether a model can perform precise arithmetic and domain-specific symbolic manipulation from memory.

SynthRefusal§

SynthRefusal characterizes safety and refusal behavior across three target classes: benign, controlled, and designer-drug. This evaluates how well models distinguish legitimate chemical synthesis requests from harmful ones, measuring both over-refusal (false positives) and under-refusal (false negatives). The benchmark likely uses graded refusal policies and reports metrics such as refusal accuracy and calibration.

SynthBench§

SynthBench evaluates reaction-outcome prediction and catalyst selection using proprietary experimental data generated in the authors' laboratory. This is a binary or multi-class classification problem: given reactants and conditions, predict whether a reaction yields the expected product, and optionally select the best catalyst. The use of private data prevents benchmark contamination and tests generalization to novel chemical space.

Implementation Details§

The benchmark is designed to be model-agnostic: any chat-based LLM can be evaluated through a standard prompting interface. Each task is formulated as a question-answer pair, and models are scored with task-specific metrics. For example, ChemAbacus uses exact-match accuracy, SynthRefusal uses a confusion-matrix-based score, and SynthBench uses top-1 accuracy and ROC-AUC.

The evaluation pipeline likely applies a temperature of 0 for reproducibility and parses model outputs with regex or structured prompting (e.g., answer in JSON). To prevent data leakage, all validation and test items are proprietary and are not released publicly.

Example Evaluation Code§

The following Python snippet shows how a model might be evaluated on SynthBench using the HuggingFace transformers library:

from transformers import pipeline
from sklearn.metrics import accuracy_score, roc_auc_score

# Load the benchmark items (reaction SMILES, catalyst, outcome)
with open('synthbench_test.json') as f:
    items = json.load(f)

llm = pipeline('text-generation', model=model_name, device=0)

def evaluate(item):
    prompt = f"""Given the reaction: {item['reactants']} \
catalyst {item['catalyst']}, predict if yield > 50%. Reply yes/no."""
    out = llm(prompt, max_new_tokens=5, temperature=0)[0]['generated_text']
    return 'yes' in out.lower()

preds = [evaluate(item) for item in items]
print("Accuracy:", accuracy_score([i['label'] for i in items], preds))

Metrics and Analysis§

Let $Y = \{y_i\}$ be ground-truth labels and $\hat{Y} = \{\hat{y}_i\}$ model predictions. For binary tasks like reaction outcome prediction, accuracy is defined as:

$$ \text{ACC} = \frac{1}{N}\sum_{i=1}^{N} \mathbb{1}[\hat{y}_i = y_i] $$

For refusal behavior, the false refusal rate (FRR) and false compliance rate (FCR) can be computed:

$$ \text{FRR} = \frac{\#\text{benign requests refused}}{\#\text{benign requests}}, \quad \text{FCR} = \frac{\#\text{harmful requests completed}}{\#\text{harmful requests}} $$

These quantities allow a balanced assessment of safety versus usability.

Significance§

onepot-Bench 0 is a step toward lab-aware LLM evaluation. By combining fundamental numeracy, safety behavior, and private reaction data, it provides a more realistic picture of model readiness for wet-lab deployment. The proprietary nature of the data also introduces a moving target that reduces benchmark saturation from pretraining contamination.