OmniScientist: An Omni-Modal Omni-Discipline AI Scientist

Overview§

OmniScientist is an end-to-end AI scientist that directly ingests raw, heterogeneous scientific evidence (images, signals, audio, video, 3-D structures, trajectories, tables, formulae, graphs) and autonomously completes the full research lifecycle—from hypothesis generation to compiled manuscript—through a deterministic pipeline of specialized agents. The central claim is that lifecycle-wide perception—giving the system access to raw evidence rather than precomputed summaries—is essential for evidence-grounded discovery.

Architecture§

The system consists of three core components:

1. Perception layer: A multimodal encoder stack that converts raw data from different modalities into a unified latent representation. For each modality $m$, an encoder $E_m$ maps raw input $\mathbf{x}_m$ to a shared embedding $mathbf{z}_m = E_m(\mathbf{x}_m)$, then a cross-modal attention mechanism fuses all modalities into a single context vector:

$$ \mathbf{z} = \text{Attention}(\mathbf{Q} = \mathbf{z}_{query}, \mathbf{K} = [\mathbf{z}_1, \ldots, \mathbf{z}_M], \mathbf{V} = [\mathbf{z}_1, \ldots, \mathbf{z}_M]) $$

This representation is continuously accessible to all agents.

1. Three autonomous agents:

• Ideation Agent: Generates research questions based on patterns observed in raw data.
• Experiment Agent: Designs and executes experiments, manipulating variables and gathering results.
• Writeup Agent: Produces the final manuscript with proper reasoning and citations.

These agents operate in a deterministic pipeline: ideation → experiment → writeup, but with feedback loops that allow observations to reshape earlier decisions.

1. Automated checks: The system enforces novelty (idea checks), statistical validity (rigour checks), and numerical traceability (claim checks) directly in code. For example, hypothesis testing uses:

$$ t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{s_1^2/n_1 + s_2^2/n_2}} $$

and results must be reproducible from execution logs with full provenance.

Implementation Sketch§

The following PyTorch-like code outlines the perception layer and agent pipeline:

import torch
import torch.nn as nn

class PerceptionLayer(nn.Module):
    def __init__(self, modalities, d_model=768):
        super().__init__()
        self.encoders = nn.ModuleDict({
            m: nn.Sequential(nn.Linear(in_dim, d_model), nn.GELU(), nn.Linear(d_model, d_model))
            for m, in_dim in modalities.items()
        })
        self.cross_attn = nn.MultiheadAttention(d_model, num_heads=8)

    def forward(self, raw_data_dict):
        z_list = [self.encoders[m](raw_data_dict[m]) for m in raw_data_dict]
        z_stack = torch.stack(z_list)  # (M, B, d_model)
        z_fused, _ = self.cross_attn(query=z_stack, key=z_stack, value=z_stack)
        return z_fused.mean(dim=0)  # aggregate over modalities

class IdeationAgent:
    def propose(self, z):
        # Generate research question from latent context
        pass

class ExperimentAgent:
    def run(self, z, hypothesis):
        # Execute experiments, collect outcomes
        pass

class WriteupAgent:
    def compile(self, z, results):
        # Generate manuscript with tables, figures, and claims
        pass

# Full pipeline
def omniscient_pipeline(raw_data, modality_dims):
    perception = PerceptionLayer(modality_dims)
    z = perception(raw_data)
    ideation = IdeationAgent()
    experiment = ExperimentAgent()
    writeup = WriteupAgent()

    hypothesis = ideation.propose(z)
    results = experiment.run(z, hypothesis)
    manuscript = writeup.compile(z, results)
    return manuscript

Evaluation§

OmniScientist was evaluated on 36 real-data cases across 5 discipline families and 4 evidence families. It completed the full raw-data-to-manuscript pipeline in all 36 cases, achieving a mean paper score of 6.3 with its reference reasoning backbone. In paired comparisons, a blind variant that received only precomputed scalar features was outperformed on all 7 evaluation dimensions, with direct perception winning 85% of head-to-head judgments. This demonstrates that retaining raw perceptual information throughout the research lifecycle is critical for scientific correctness and insight.

Key Insight§

The system's novelty is not just in automating more workflow steps, but in ensuring that every stage of the scientific method is grounded in the original evidence, rather than in lossy summaries. This lifecycle-wide perception enables the AI scientist to notice spatial, temporal, and cross-channel relationships that would otherwise be invisible.