Agentic RAG Knowledge Assistant

The problem

A standard RAG pipeline retrieves the top few chunks and answers straight away. When the question is out of scope or rests on a false premise, nothing stops the model from answering from memory.

The project tests whether a closed loop of plan, retrieve, audit, retry and then generate or refuse reduces unsupported answers, and measures what that costs in answer coverage, latency and API spend.

Approach

A Python pipeline over a corpus of 10 research papers. Dense vector search and BM25 are fused with Reciprocal Rank Fusion (k = 60). An evidence auditor checks claims against the retrieved chunks, and the system retries up to twice before it answers or refuses. Embeddings are cached by SHA-256 content hash.

Architecture

Plan, retrieve, audit, retry, then generate or refuse. Each stage is a separate module in the repository.

Design choices

• Fusing two retrievers by rank

  What it does: Dense and BM25 results are combined with Reciprocal Rank Fusion (k = 60), which uses each result's rank rather than its raw score, so the two scoring scales do not need normalising.

  What it costs: Two indices to maintain and two searches per query.

• Auditing evidence before generating

  What it does: An LLM-based audit step, with deterministic quote verification, decides whether the retrieved evidence is enough before an answer is written.

  What it costs: The auditor accounts for about 88.2% of average latency (15.60 s of 17.69 s) and about 81.4% of benchmark API spend ($0.237 of $0.291).

In the product

Code excerpt

    def _retrieve_hybrid(
        self,
        query: str,
        top_k: int,
        filters: Optional[Dict[str, Any]],
        dense_weight: float,
        sparse_weight: float
    ) -> List[SearchResult]:
        """
        Executes Reciprocal Rank Fusion over candidate pools from Dense and BM25 retrievers.
        """
        # Fetch wider candidate pools (e.g. 2x top_k or at least 20 candidates) for robust fusion
        candidate_k = max(top_k * 3, 20)

        dense_hits = self.vector_store.search(query, top_k=candidate_k, filters=filters)
        sparse_hits = self.bm25_engine.search(query, top_k=candidate_k, filters=filters)

        # Lookup maps: chunk_id -> (score, rank)
        dense_map: Dict[str, Tuple[float, int]] = {c.chunk_id: (score, rank) for c, score, rank in dense_hits}
        sparse_map: Dict[str, Tuple[float, int]] = {c.chunk_id: (score, rank) for c, score, rank in sparse_hits}

        # Union of all candidate chunk IDs
        all_chunk_ids = set(dense_map.keys()) | set(sparse_map.keys())

        # Compute RRF score for every candidate
        scored_candidates: List[Tuple[str, float]] = []
        for cid in all_chunk_ids:
            rrf_score = 0.0
            if cid in dense_map:
                _, d_rank = dense_map[cid]
                rrf_score += dense_weight * (1.0 / (self.rrf_k + d_rank))
            if cid in sparse_map:
                _, s_rank = sparse_map[cid]
                rrf_score += sparse_weight * (1.0 / (self.rrf_k + s_rank))
            scored_candidates.append((cid, rrf_score))

        scored_candidates.sort(key=lambda x: x[1], reverse=True)
        top_candidates = scored_candidates[:top_k]

Results and facts

Eight configurations were run on the 32-question held-out test set of a 40-question dataset (8 development, 32 held-out). The test set has 24 answerable, 4 unanswerable and 4 false-premise questions, and answer accuracy is calculated over the 24 answerable ones. The figures here are the frozen results from before later fixes to the auditor, which the repository labels "frozen pre-fix". The full system had 55.2% citation precision and 52.1% evidence recall. The figures above come from small samples: four unanswerable and 24 answerable questions.

Hallucinations on unanswerable questions

0 / 4Four unanswerable questions in the 32-question held-out set. The hybrid RAG baseline hallucinated on 4 / 4.

Correct refusals

4 / 4All four unanswerable questions were refused.

Answer accuracy

45.8%A human-reviewed score of 11.0 / 24 on the answerable questions. The hybrid baseline scored higher, 60.4% (14.5 / 24).

Wrongly refused

12.5%3 of the 24 answerable questions were refused when they should have been answered.

Average latency

17.7 sAgainst 1.86 s for the hybrid baseline, roughly 8.5 times slower.

Automated tests

9090 test functions; 87 run offline by default with no API calls, and 3 live integration tests are opt-in.

Limitations

• The audit loop is slow: about 17.7 s per question against 1.86 s for the single-pass baseline.
• The conservative evidence gate costs coverage. It wrongly refused 12.5% of answerable questions and scored lower accuracy than the baseline.
• The test sample is small, with four unanswerable and 24 answerable questions, so the 0 / 4 result is a single small test, not a rate.
• On one false-premise question (Q38) the system tried to answer instead of challenging the premise. A contradiction check was added after the evaluation run, so the published numbers do not include it.

Related essays

• 10 Aug 2026Hybrid Retrieval Systems in Production: Combining BM25, Dense Embeddings, and Reciprocal Rank Fusion
• 25 Aug 2026Evaluating Hallucination and Citation Faithfulness in Retrieval-Augmented Generation