Evaluating Open-Weight LLMs for Generating Structured Threat Information for Autonomous Vehicle Vulnerabilities
By Md Erfan, Ahmed Ryan, Md Kamal Hossain Chowdhury, Md Rayhanur Rahman
"Evaluates open-weight LLMs for converting CAV vulnerability descriptions into structured STIX threat intelligence, achieving high F1 for SDO/CWE but poor MITRE ATT&CK mapping."
Abstract
Connected and Autonomous Vehicles (CAVs) rely on interconnected software and hardware components, including sensors, Electronic Control Units, in-vehicle infotainment systems, and telematics units, where vulnerabilities can compromise assets, users, and vehicle operations. These vulnerabilities are commonly documented as plain text in the Common Vulnerabilities and Exposures (CVE) database; however, security practitioners require structured information about affected assets, types of weaknesses, and attack behaviors to effectively mitigate the risks from these vulnerabilities. To this end, we evaluate open-weight Large Language Models (LLMs) for generating Structured Threat Information Expression (STIX), a well-known structured format for representing threat information, for CAV-related CVEs. We construct a dataset called CAV-STIXGen that maps CAV vulnerability descriptions to STIX domain objects (SDO), STIX relationship objects (SRO), Common Weakness Enumeration (CWE), and MITRE ATT&CK techniques mappings. Using this dataset, we evaluated 11 open-weight LLMs (4B to 120B parameters) across various prompting strategies and temperatures. Single-model configurations achieve F1 scores of 0.94 for SDO, 0.63 for SRO, and 0.99 for CWE mapping, while complete MITRE ATT&CK mapping remains challenging. In a multi-agent setup, Gemma-4-31B and Codestral-22B achieve F1 scores of 0.91 for SDOs and 0.43 for SROs, respectively. Lastly, we analyze CWE and MITRE ATT&CK co-occurrences to identify recurring threat patterns in the CAV domain, demonstrating how AI-assisted vulnerability-to-STIX translation can automate threat intelligence and prioritize defense in transportation security.
Technical Analysis & Implementation
Summary§
This paper evaluates 11 open-weight LLMs (4B-120B parameters) on generating Structured Threat Information Expression (STIX) from Common Vulnerabilities and Exposures (CVE) descriptions for Connected and Autonomous Vehicles (CAVs). The authors construct CAV-STIXGen, a dataset mapping vulnerability descriptions to STIX Domain Objects (SDO), STIX Relationship Objects (SRO), Common Weakness Enumeration (CWE), and MITRE ATT&CK techniques. They compare single-model and multi-agent configurations across prompting strategies and temperatures.
Core Methodology§
Dataset Construction§
CAV-STIXGen comprises CAV-related CVEs with manual annotations for SDO, SRO, CWE, and ATT&CK mappings. SDOs capture entities (e.g., vulnerability, asset), SROs capture relationships, and CWEs/ATT&CK provide taxonomy labels.
Evaluation Setup§
- Models: 11 open-weight LLMs: Gemma-2-27B, Gemma-4-31B, Codestral-22B, Llama-3.1-70B, etc.
- Prompts: Zero-shot and few-shot (1-shot, 3-shot) with chain-of-thought (CoT) reasoning.
- Temperatures: 0.0, 0.3, 0.7.
- Metrics: Exact match (EM) and F1 for SDO/SRO/CWE/ATT&CK.
Single-Model Results§
- Best SDO F1=0.94 (Gemma-4-31B, 1-shot CoT).
- Best SRO F1=0.63 (Codestral-22B, zero-shot CoT).
- CWE mapping nearly perfect (F1=0.99).
- MITRE ATT&CK mapping low (best F1=0.29), highlighting difficulty.
Multi-Agent Setup§
A coordinator agent decomposes the task: one agent generates SDOs, another SROs, a third extracts CWEs, and a fourth maps ATT&CK. Best multi-agent F1: SDO=0.91 (Gemma-4-31B), SRO=0.43 (Codestral-22B). For SROs, single-model outperforms multi-agent.
Implementation Details§
A simple PyTorch-style inference code snippet for single-model evaluation:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "google/gemma-2-27b"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
def generate_stix(cve_text: str, prompt_template: str) -> str:
prompt = prompt_template.format(cve=cve_text)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=512,
temperature=0.0,
do_sample=False
)
return tokenizer.decode(outputs[0], skip_special_tokens=True)
# Example usage
cve = "Buffer overflow in the telematics unit leads to remote code execution."
prompt = "Convert the following CVE description into STIX 2.1 JSON:\n{cve}\nSTIX:"
result = generate_stix(cve, prompt)Key Equations§
The evaluation uses standard classification metrics: \[ \text{F1} = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} \] For SDO and SRO, exact match of STIX object types and properties is required.
Analysis of Co-occurrences§
Co-occurrence analysis of CWE and MITRE ATT&CK reveals recurring patterns: memory corruption (CWE-119) often linked to initial access (T1190) and remote code execution (T1210). This aids in prioritization of defenses.
Conclusion§
The work demonstrates feasibility of using open-weight LLMs for automated STIX generation, but MITRE ATT&CK mapping remains a challenge. Multi-agent setups did not outperform single models for SROs. Future work may improve via fine-tuning or retrieval-augmented generation.