AI Data Analyst Agent

The problem

An agent that writes and runs arbitrary Python can crash, can behave differently from run to run, and is a security risk to execute.

To see whether a language model can be limited to choosing a tool and filling in its parameters, with the computation done by fixed code that can be inspected.

Approach

An LLM router (gpt-4o-mini) picks one of four tools (query_data, plot_chart, summary_stats, clarify) and extracts its parameters. SQL runs in DuckDB behind a 35-keyword regex filter, results are capped at 500 rows, and a guard compares the numbers in the written answer with the tool output.

Architecture

A dataset profiler, an LLM router, four fixed tools over an in-memory DuckDB table, and a numeric check on the answer.

Design choices

• Four fixed tools instead of generated code

  What it does: The model chooses a tool and its parameters. The analysis itself runs in pre-written tools backed by DuckDB and Plotly.

  What it costs: No custom analysis outside those tools. Each question runs as a single step, with no multi-step planning or memory.

• A regex filter for SQL

  What it does: Queries must start with SELECT or WITH, contain a single statement, and avoid 35 blocked keywords (mutation, file-system and administrative commands).

  What it costs: It is a keyword blacklist, not a parser and not an allowlist, so it is weaker than parser-based validation. It is not engine-enforced read-only access, and it does not guarantee that every external file read is blocked.

• Reference years from the data

  What it does: The repository records a bug where "this year" resolved to the model's training cutoff. The profiler now computes reference years from the dataset and puts them in the prompt.

  What it costs: It needs a profiling pass whenever a dataset is loaded.

In the product

Code excerpt

    def validate_sql(self, sql_query: str) -> None:
        """
        Validates that the SQL query is strictly read-only and safe.
        Raises ValueError on any violation.
        """
        clean_sql = sql_query.strip()

        # Remove comments (-- or /* */) before validation
        clean_sql = re.sub(r"--.*?(\n|$)", " ", clean_sql)
        clean_sql = re.sub(r"/\*.*?\*/", " ", clean_sql, flags=re.DOTALL).strip()

        # Must start with SELECT or WITH (for CTEs)
        upper_sql = clean_sql.upper()
        if not (upper_sql.startswith("SELECT") or upper_sql.startswith("WITH")):
            raise ValueError("Security violation: Only read-only SELECT and WITH (CTE) queries are permitted.")

        # Disallow statement chaining with semicolons
        semicolon_count = clean_sql.count(";")
        if semicolon_count > 1 or (semicolon_count == 1 and not clean_sql.endswith(";")):
            raise ValueError("Security violation: Multiple chained SQL statements are not allowed.")

        # Check for disallowed mutating / DDL / administrative keywords
        for forbidden in config.disallowed_sql_keywords:
            pattern = rf"\b{forbidden}\b"
            if re.search(pattern, upper_sql):
                raise ValueError(f"Security violation: SQL keyword '{forbidden}' is prohibited in read-only mode.")

Results and facts

A 20-question ground-truth benchmark on the bundled Superstore sales data, covering lookups, aggregations, regional comparisons, charts, descriptive statistics and ambiguous prompts. It is a small, single-dataset test. The repository also lists "industry baseline" figures next to these results; they are not measured comparisons, so they are not repeated here.

Tool selection

100.0%20 / 20 questions routed to the expected tool.

Execution success

100.0%20 / 20 tool runs completed without an error.

Answer correctness

85.0%17 / 20 answers matched the expected value.

Numerical faithfulness

80.0%16 / 20 answers had every number match the tool output.

Average latency

4,142 msAverage end-to-end time per question on the 20-question benchmark.

Limitations

• SQL safety rests on a keyword blacklist, not a parser or an allowlist. It is not engine-enforced read-only access and does not guarantee that external file reads are blocked.
• Routing is done by an LLM, so it can vary between runs.
• Each session registers one dataset as a single in-memory table.
• Questions run as one step. Multi-step planning and conversation memory are not implemented.
• The faithfulness check looks at numbers only, not wording or qualitative claims.

Related essays

• 14 Aug 2026Deterministic Guardrails for LLM Agents: Restricting SQL and Pydantic Schemas without Open-Ended Code Execution
• 18 Aug 2026In-Process Columnar OLAP with DuckDB: Architecture, Vectorized Execution, and Analytics Engineering