- F1
- 0.642
- Training examples
- 0
- Compute
- No GPU
- Per record
- ~$0.0003
Research question
Can a team of LLM agents, with no training data and no GPU, reliably detect when a retrieval-augmented system says something its sources don’t support?
In plain language
RAG systems answer questions using retrieved documents, but they still make things up. This work breaks each answer into individual factual claims, checks every claim against the retrieved passages, and marks the parts that aren’t supported, all without training a model.
Method
The pipeline has three stages. A claim-extraction agent decomposes the response into atomic factual claims. A verification agent checks each claim against the source passages. Task-specific decision rules then aggregate the verdicts and flag unsupported spans. It was evaluated zero-shot on RAGTruth across question answering, summarization and data-to-text.
Results
The system reached 0.642 F1 with no training examples and no GPU, at roughly $0.0003 of processing cost per record.