Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains

Overview§

This paper presents a complete adaptation of NVIDIA's Nemotron retrieval stack to Modern Greek, covering corpus mining, synthetic supervision, retriever training, reranking, reader fine-tuning, and the introduction of a new evaluation benchmark, HERA. The work targets specialist domains (legal, energy, financial, medical) where Greek-language RAG is under-served.

Corpus Mining and Synthetic Supervision§

The authors mine a large Greek corpus and generate 65,773 retrieval pairs using synthetic supervision. This involves constructing (query, positive, negative) triples, likely by using existing multilingual models or rule-based heuristics to derive training signal from documents.

Retrieval Model Training§

A Nemotron 1B embedder is fine-tuned on these pairs using a contrastive objective. The training loss is typically the InfoNCE loss:

$$ \mathcal{L} = -\log \frac{e^{s(q, d^+)/\tau}}{\sum_{j=1}^{N} e^{s(q, d_j)/\tau}} $$

where $s(q,d)$ is the similarity score (e.g., dot product or cosine similarity) between query $q$ and document $d$, and $\tau$ is a temperature hyperparameter. This pushes positive pairs closer while pushing negatives apart. The fine-tuned model improves nDCG@10 from 0.362 (unadapted) to 0.835 on specialist Greek corpora, significantly surpassing both BM25 and off-the-shelf multilingual dense retrievers. Interestingly, the learned competence transfers to general-domain Greek, but the advantage over BM25 becomes domain-dependent.

Reranker Adaptation§

A cross-encoder reranker is also adapted. Cross-encoders jointly encode query and passage, producing a relevance score. The reranker is trained on the same Greek data, yielding consistent improvements across domains when applied to the top-k retrieved candidates.

Reader Fine-Tuning with LoRA§

The reader is a Nemotron 30B-A3B mixture-of-experts (MoE) model. They apply Low-Rank Adaptation (LoRA) to fine-tune it for grounded generation. LoRA updates the weight matrix by a low-rank product:

$$ W' = W + \Delta W = W + BA $$

where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and $r \ll \min(d,k)$. This drastically reduces trainable parameters. After LoRA tuning, judged answer correctness improves from 29.4% to 66.9%, with significant gains in faithfulness and citation quality.

HERA Benchmark§

HERA is introduced as the first large-scale Greek retrieval-augmented generation benchmark, enabling reproducible evaluation for future research. The models and benchmark are released publicly.

Code Sketch§

The following snippet illustrates a contrastive fine-tuning loop for the embedder, using a standard sentence-transformers style approach:

import torch
from torch import nn
from sentence_transformers import SentenceTransformer, InputExample
from torch.utils.data import DataLoader

model = SentenceTransformer("nvidia/Nemotron-1B-Embedding")
# Use a temperature-scaled contrastive loss
tau = 0.05

def contrastive_loss(q_emb, pos_emb, neg_embs):
    pos_sim = (q_emb * pos_emb).sum(dim=-1) / tau
    neg_sims = torch.matmul(q_emb, neg_embs.transpose(0, 1)) / tau
    logits = torch.cat([pos_sim.unsqueeze(1), neg_sims], dim=1)
    labels = torch.zeros(logits.size(0), dtype=torch.long, device=logits.device)
    return nn.functional.cross_entropy(logits, labels)

# Assume you have triples: (query, positive, negatives)
for batch in DataLoader(triples, batch_size=32, shuffle=True):
    q_emb = model.encode(batch["query"], convert_to_tensor=True)
    pos_emb = model.encode(batch["positive"], convert_to_tensor=True)
    neg_embs = torch.stack([model.encode(n, convert_to_tensor=True) for n in batch["negatives"]])
    loss = contrastive_loss(q_emb, pos_emb, neg_embs)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Key Results§

• Retriever: nDCG@10 rises from 0.362 to 0.835 after fine-tuning.
• Reranker: consistent improvements across all specialist domains.
• Reader: correctness 29.4% → 66.9%, with better faithfulness and citation quality.

This work provides a blueprint for adapting multilingual retrieval and generation stacks to low-resource or specialized languages, while benchmarking shows the importance of domain-specific fine-tuning over generic multilingual models.