Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation

Core Methodology§

The paper proposes IdeaGene-Bench (IG-Bench), a benchmark for evaluating AI systems on scientific lineage reasoning and lineage-grounded idea generation. It is built on the IdeaGene framework, where each paper or proposal is represented as a set of minimal, typed, evidence-grounded objects called Idea Genome objects. A GenomeDiff operation aligns these objects to record evolutionary dynamics: inheritance, mutation, loss, external import, and novel insertion, governed by six operational evolutionary dynamics.

Benchmark Components§

• 1,961 golden lineage traces across 10 scientific domains.
• 1,085 curated Idea Genome objects.
• 920 pairwise GenomeDiff records.

Evaluation Tasks§

1. IG-Exam: 42 task types, 1,029 instances testing closed-form lineage reasoning across four capacities:

• Idea Genome abstraction
• Inheritance tracing
• Evolutionary reasoning
• Lineage verification

2. IG-Arena: Evaluates generation via a Population-Evolution Score (PES), measuring whether a proposal can be inserted as a coherent descendant of a given lineage population. PES rewards:

• Correct inheritance of Idea Genome objects
• Meaningful variation from nearby work
• Selection value for future research

Technical Details§

Each Idea Genome object is a tuple $(t, g, e, c)$ where:

• $t$: type (mechanism, limitation, result, etc.)
• $g$: grounding citation
• $e$: evidence snippet
• $c$: confidence score

GenomeDiff operations are formalized as:

• Inherit: $\text{GenomeDiff}(o_i, o_j) = \text{inherit}$ if $o_i$ and $o_j$ share the same type and grounding.
• Mutate: $\text{GenomeDiff}(o_i, o_j) = \text{mutate}$ if $o_i$ and $o_j$ share type but differ in grounding or evidence.
• Novel: $\text{GenomeDiff}(\emptyset, o_j) = \text{novel}$.
• Loss: $\text{GenomeDiff}(o_i, \emptyset) = \text{loss}$.
• Import: $\text{GenomeDiff}(o_i^{(A)}, o_j^{(B)}) = \text{import}$ for external domain $B$.

Implementation Code§

Below is a simplified Python snippet illustrating how to compute a GenomeDiff between two Idea Genome sets:

from typing import List, Tuple, Dict

class IdeaGenome:
    def __init__(self, type: str, grounding: str, evidence: str, confidence: float):
        self.type = type
        self.grounding = grounding
        self.evidence = evidence
        self.confidence = confidence

def compute_genome_diff(prev: List[IdeaGenome], curr: List[IdeaGenome]) -> Dict[str, List]:
    diff = {'inherit': [], 'mutate': [], 'loss': [], 'novel': [], 'import': []}
    prev_set = {(g.type, g.grounding) for g in prev}
    curr_set = {(g.type, g.grounding) for g in curr}
    
    for p in prev:
        key = (p.type, p.grounding)
        if key in curr_set:
            diff['inherit'].append(p)
        else:
            diff['loss'].append(p)
    
    for c in curr:
        key = (c.type, c.grounding)
        if key in prev_set:
            continue
        # check if same type but different grounding -> mutate
        if any(p.type == c.type for p in prev):
            diff['mutate'].append(c)
        else:
            # check if grounding from external domain (simplified)
            if 'extern' in c.grounding:
                diff['import'].append(c)
            else:
                diff['novel'].append(c)
    return diff

Mathematical Formulation§

PES (Population-Evolution Score) for a proposal $P$ given lineage population $L$:

$$ \text{PES}(P|L) = \alpha \cdot \text{InheritScore}(P, L) + \beta \cdot \text{VariationScore}(P, L) + \gamma \cdot \text{SelectionValue}(P, L) $$

where $\alpha,\beta,\gamma$ are hyperparameters tuned per domain.

Results§

Experiments on 14 LLM-based scientists (GPT-4, Claude, Gemini, etc.) show:

• Best exact accuracy on IG-Exam: 27.3%.
• PES scores reveal that structured lineage context reshuffles rankings; no single model uniformly benefits.
• A compositional bottleneck exists: models fail to combine multiple evolutionary dynamics simultaneously.