PYCON 2026 · 10 MIN + 5 Q&A

Benchmarking
offline LLM agents
in bioinformatics

Nine agent architectures, two models, 41 verified questions, one 48 GB GPU — and a scoring harness built because none of the standard evaluation methods fit.

Kayla Queenazima · @kaylaque · github.com/matinnuhamunada/chatbgc_agentic_rag

THE SHAPE OF THIS TALK
01Design thinking → the system
02Why the standard evals don't fit
03How I built one that does
04Results, findings, what's still open
Every number is checkable
Raw traces are public on the Hub.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202601 / 15
§0 · WHO'S TALKING · 0:25

Hi, I'm Kayla! An engineer who loves the
intersection of AI and bioinformatics.

◆What I do
AI Engineer, Hysn Technologies · Research assistant, BEAMS Lab, Faculty of Biology, UGM · Organizer, PyLadies Yogyakarta.
◆On ChatBGC I own the measurement layer
The agent harness, the benchmark runner and the evaluation stack. Matin Nuhamunada owns the biology and the bioinformatics.
◆Every problem in this talk is a harness problem, not a model problem
That's not modesty — it's the finding.
FIND ME
Kayla Queenazima
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202602 / 15
§1 · WHERE THE QUESTION COMES FROM · 0:50

The data isn't the bottleneck. Access is.

THE ACTUAL WORKFLOW

Typical request, in the corridor:
"Do we already have anything on X?"

Answering it means writing SQL across 65 tables. The bioinformatician stops, writes the query, and returns a few hours later.

→Each question costs an expert hours
Low-value questions therefore go unasked, and the lab's own data stays unqueried.
→The data isn't the bottleneck. Access is.
Every required fact is already in the database. The cost is translating a domain question into a correct query.
THE BIOLOGY, IN THREE LINES
01Bacteria make useful chemistry
Antibiotics, antifungals, pigments — the source of a large share of clinical drugs.
02The recipes sit together in the genome
A biosynthetic gene cluster — genes for one compound, physically adjacent. Find the cluster, and you have a candidate compound.
03A tool finds them; a database stores them
antiSMASH scans genomes and writes what it found into DuckDB. That database is the lab's memory of everything it has ever sequenced.
Why an LLM does not solve this directly
The model has not seen this schema, and a wrong-but-valid join returns rows — no error, no warning. A slow expert is replaced by a fast, confident, occasionally wrong one.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202603 / 15
§2 · DESIGN THINKING · 1:35

Four constraints decide the design. Only one is about biology.

The biology question is simple: "which gene clusters are in these genomes, and what do they encode?" Everything hard comes from the constraints around it.

CONSTRAINTWHY IT'S TRUE HERE THE DESIGN DECISION IT FORCESSAME FOR YOU IF
Data can't leaveUnpublished genomes. A hosted API is not an option. Self-hosted inference. Everything runs on one box. Health records, financial data, NDA work
Low concurrencyA research group, or one person. Not a public service. Optimise single-request latency, not throughput. No batching to hide prefill behind. Internal tools, CLIs, notebooks
The model has never seen this schema 65 tables. regions and bgc_types join only through a junction table. Domain knowledge must be retrieved, not assumed — look the schema up before writing SQL (that’s the RAG part). Any internal schema or API
Wrong answers look right A wrong-but-valid join returns rows. No error, no warning. Correctness cannot be judged by "did it run". → the whole evaluation stack. Anything generating queries or code
The through-line
Three of the four are domain-independent. They determine the architecture, and the absence of a suitable evaluation method.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202604 / 15
§3 · THE SYSTEM · 2:15

One fixed pipeline. Everything else is a swappable axis.

INPUT Direct user Typer CLI · chat MCP client MCP — any LLM client can call it QUERY HARNESS — FIVE PHASES, 11 SELECTABLE TOPOLOGIES Capture question in Retrieve schema + examples Plan tables, joins Fetch generate + run SQL Interpret answer from rows CONTROL --arch · 11 topologies t0–t5 · h1/h2 · g1 · t3e --route · multi-router small / medium / large tier pool OUTPUT Answer + SQL + rows back to CLI or MCP client TRAIN MODE Schema docs + Q→SQL pairs Embeddings nomic-embed-text RETRIEVAL — RAG ChromaDB tool + text memory Hybrid retriever keyword + meaning search, fused Cross-encoder rerank bge-reranker-base EXECUTION — SQL AGENT SQL validator EXPLAIN + deny-list DuckDB — antiSMASH 65 tables, read-only CONFIGURES
Train mode runs offline and fills ChromaDB. At query time the harness is fixed; --arch swaps the topology over it and --route picks a model tier per call. Both are benchmark axes, not user-facing settings.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202605 / 15
§4 · EVALUATION FUNDAMENTALS · 3:05

Four standard evaluation methods. One fits text-to-SQL.

Sebastian Raschka's four-way map of LLM evaluation. Identifying which category this task falls into determined the design.

APPROACHWHAT IT ACTUALLY SCORES ITS FAILURE MODEFIT FOR A TEXT-TO-SQL AGENT
Multiple choice
MMLU-style
 Picking from options Tests recall, not use NO  There are no options — the output is a query
Verifiers
deterministic check
 Free output, checked by code Needs a checkable domain, and ignores how you got there YES  SQL returns the right rows or it doesn't. The backbone.
Leaderboards
Elo / Bradley–Terry
 Which one people prefer Preference, not correctness NO  No user pool, and preference isn't the question
LLM-as-a-judge
rubric scored
 Another model grades it Only as good as the judge, and not reproducible ONE LAYER  Phrasing only, and kept optional
The gap I had to fill myself
Verifiers check the destination only. None of the four scores the route — junction table, schema prefix. A deterministic layer was added for it.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202606 / 15
§5 · BENCHMARK DESIGN · 3:55

Seven layers. Six of them cost nothing to run.

LAYERCHECK A SCORE LOOKS LIKECOSTWHAT IT CATCHES
1 Does it runEXPLAIN + deny-list 1 or 0FREESyntax errors, writes, unknown columns
2 Right rowsgold vs predicted result sets 1 or 0FREEWrong joins, wrong filters
3 Right nothingdeclared negative cases only 1 or 0FREERows invented for entities that don't exist
4 Right methodschema prefix · junction join · SELECT-only 0.75 = 3 of 4FREERight rows by luck — the trajectory layer the four standard methods don't have
5 Answer matches rowsexact numbers + named entities 0.60FREECorrect query, invented summary
6 Answer is sensibleLLM judge, rubric + reference 0.831 CALLPhrasing the deterministic checks can't score
7 Nothing crashedno pipeline error 1 or 0FREETimeouts, rate limits — kept off the accuracy axis
Why layer 3 — "right nothing" — exists
Previously an empty gold set and an empty prediction both scored 1.0. Fabrication was unpunished and unmeasurable.
Cost discipline
The judge is the only paid layer and is disabled by default. All figures here are deterministic.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202607 / 15
§5 · TEST VARIATION DESIGN · 4:40

Four axes. Vary one, hold the rest.

A1Architecture — 9 of 11 run
Ordered by autonomy: write SQL in one shot → think first, retry on failure → look up examples first → let the model pick its next tool → plan, then execute step by step → a second model critiques the first.
A2Model — three of them
A local 27B generalist, a hosted free tier, and a local 7B SQL specialist (XiYanSQL-7B-AWQ). The privacy constraint makes local-vs-hosted the decisive comparison.
A3Question type — the segments
27 positive / 14 negative. 6 Easy / 29 Medium / 6 Hard. Tagged simple / join / aggregation, so failure has a shape and not just a rate.
A4Routing — orthogonal to all of it
A tier pool the router picks from per call, so accuracy-vs-cost is measurable independently of topology.
41
Questions
every gold query verified to run
1,107
Runs · 3 models
41 × 9 × 3
14
Negative cases
real identifiers, no matching data
1
Replicate
the known weakness
Which gaps are interpretable
Each cell is a single run; single runs scatter by ±0.41–0.46.
Per architecture (41 runs): gaps below 0.27 are not interpretable.
Per model (363 runs): gaps above 0.03 are.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202608 / 15
§6 · SETUP · 5:20

The entire stack runs on one machine.

ROLEUSED HEREWHY THIS ONE
ServingvLLMContinuous batching, strong prefix-cache support
Local modelQwen3.6-27B-FP8FP8 fits the card with room for context
Comparison modeldeepseek-v4-flash-freeHosted free tier — the realistic alternative
SQL specialistXiYanSQL-7B-AWQLocal, 8k context — a purpose-built text-to-SQL model
OrchestrationLangGraphTyped state, swappable topologies — the --arch axis
RetrievalLlamaIndex + ChromaDBBM25 + vector fused; keyword alone misses paraphrase
RerankerBAAI/bge-reranker-baseRe-scores the shortlist by reading query and doc together
Embeddingsnomic-embed-text-v1.5Runs alongside the 27B on the same card
WarehouseDuckDBSingle-file, analytical, no server
Tracing / evalLangfuse · promptfoo + runnerOne span per phase, one per model call
THE HARDWARE

NVIDIA RTX 6000 Ada, 48 GB

Holds simultaneously
A 27B model in FP8, a long context window, the embedding model and the cross-encoder reranker — no second machine, no network hop. The Ada generation makes FP8 native, and that is what buys the context headroom.
Sizing, not checkpoints
Checkpoints date quickly. The sizing method does not.
Operating cost
No per-token cost, no rate limits, no data egress.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202609 / 15
§7 · RESULTS · 6:00

Both local models beat the hosted free tier.

MODEL EXECUTION ACCURACY ANSWER QUALITY TOK / RUN UNSCORED
Qwen3.6-27B-FP8
local · 27B generalist
0.767
 0.69510.8k6 / 369
deepseek-v4-flash-free
hosted · free tier
0.645
 0.70729.1k4 / 369
XiYanSQL-7B-AWQ
local · 7B SQL specialist
0.612
 0.39811.8k 54 / 369
THE FAIR COMPARISON — THE 5 ARCHITECTURES ALL THREE CAN RUN
MODELEXECP50 LATENCY
Qwen 27B0.79622.6 s
XiYanSQL 7B0.7583.0 s
deepseek0.59833.2 s
Summary
The 7B specialist lands 4 points behind a 27B generalist on SQL, at 7.5× lower latency — and less than half its answer quality.
Confidence
Single replicate · judge off · 54 of XiYanSQL's runs did not complete — see next slide.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202610 / 15
§7 · FINDINGS · 6:45

The segments disagree with the headline.

F1"Which architecture is best" is not a well-formed question.
· Same graph: 0.512 on deepseek, 0.841 on Qwen
· A 0.33 swing — bigger than any gap between the models
· Qwen benefits from planning steps; deepseek from shorter graphs
F2The specialist writes the query and then cannot say what it means.
· Restricted to runs where the SQL was perfect, answer quality is
  0.765 (Qwen) · 0.747 (deepseek) · 0.469 (XiYanSQL)
· Same correct rows in front of it, half the answer quality
F3Context window decides the architecture comparison before reasoning does.
· 54 XiYanSQL runs never completed — all context overflow at 8k
· t1 fails 41 of 41: it injects the whole schema, no retrieval
· The 5 retrieval architectures all complete, in a flat 0.74–0.79 band
EXECUTION ACCURACY BY SQL STRUCTURE
STRUCTUREDEEPSEEK QWENXIYAN
simple0.6730.8270.683
join0.5770.543 0.213
aggregation0.487 0.4860.400
Aggregation is model-independent
Two generalists within 0.001 of each other, ~34 points below their own simple-query score. That points at planner design.
The specialist fails differently
Joins collapse to 0.213. A model trained on single-table benchmarks does not transfer to biology that is 3–4 joins deep.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202611 / 15
§8 · OPEN PROBLEMS · 7:30

Four results this campaign cannot explain.

O136 runs followed every rule I wrote and still got the wrong rows.
No run produced correct rows while breaking a rule, so the check has no false positives — and 36 false negatives. The missing criterion is unidentified.
O2A third of correct queries still gave a wrong sentence.
76 of Qwen's 233 correct queries produced an answer inconsistent with the rows just retrieved. Prompt design or model limitation is unresolved.
O3Joins are the only thing the local model loses at.
Qwen leads on simple queries by 15 points and trails on joins (0.543 vs 0.577); the SQL specialist collapses to 0.213 on the same shape. No mechanism for a join-specific reversal is known.
O4My hardest question is hard for the wrong reason.
"What products are recorded for type II PKS clusters?" scores 0.22 because the table is empty from an import defect, not because the answer is negative. Whether this is a fair test of caution is undecided.
Status
All four are open. Answers or prior work on any of them are welcome after the session.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202612 / 15
§9 · RECOMMENDATIONS · 8:15

Eight things worth doing.

FOR YOUR OWN SYSTEM
01Declare the negative cases.
Refusal and fabrication are otherwise indistinguishable.
02Score the route, not only the outcome.
A correct result by chance otherwise scores as a correct method.
03Report segments, not means.
The aggregate conceals the actionable difference.
04Evaluate local inference first.
One 48 GB card exceeded the hosted model on accuracy and token cost.
FOR THIS PROJECT, NEXT
05Add replicates to the leading architectures.
Three repeats over four architectures costs less than one pass over nine, and yields error bars.
06Split the pipeline across two models.
Specialist for the SQL, generalist for the answer — the evidence points straight at it.
07Bound the ReAct loop's context growth.
Unbounded growth accounts for all six lost runs.
08Target aggregation queries.
The largest gap, and model-independent.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202613 / 15
§10 · THANK YOU · 9:05

A benchmark isn't a
score. It's how you learn
precisely what's broken.

And it can't stand still. The product changes, the schema changes, the questions change — so the benchmark moves too, or it starts measuring the past.

Thanks to Matin Nuhamunada and the BEAMS Lab, Faculty of Biology, UGM.

CHECK MY NUMBERS
Code + benchmark harness
github.com/matinnuhamunada/
chatbgc_agentic_rag
Every run, raw
huggingface.co/datasets/
kaylazima/trace-igf600-2026-08-05
Each run record holds
The SQL, the answer, the reasoning, timings, tokens and all seven scores.
kayla queenazima · @kaylaque · github.com/matinnuhamunada/chatbgc_agentic_rag14 / 15
§11 · ONE MORE THING · 9:40

Ladies — come to
this one next.

Python for Ladies:
Why Communities Matter?
Today at 2:05 PM · Parallel Room 1
Binus University, Campus Anggrek
Whatever your level
You don't need to have spoken, or built anything yet. Come and meet the chapters — Indonesia, Bandung and Yogyakarta will all be there.

I organise PyLadies Yogyakarta — find me afterwards if you'd like to get involved.

PyLadies Session — Python for Ladies: Why Communities Matter? Saturday 8 August 2026, 2:05 PM, Parallel Room 1, Binus University Campus Anggrek
BENCHMARKING OFFLINE LLM AGENTS · PYCON 202615 / 15
APPENDIX · NOT PRESENTED · FOR Q&A

What one plain-English sentence actually costs.

gold SQL — difficulty: hard
SELECT r.*, m.*, bgc.*
FROM antismash.regions r
JOIN antismash.modules m ON r.region_id = m.region_id
JOIN antismash.rel_regions_types rt ON r.region_id = rt.region_id
JOIN antismash.bgc_types bgc ON rt.bgc_type_id = bgc.bgc_type_id
WHERE bgc.term ILIKE '%PKS%'
  AND m.trans_at = true;
question: "Which PKS regions have trans-AT modules?"
◆Three joins are not optional
regions and bgc_types have no direct foreign key. Go through the junction table or get nothing.
◆The schema prefix is mandatory
Drop antismash. and the query fails outright — which is the good case.
◆The bad case is worse
A wrong-but-valid join returns rows. No error, no warning, just a plausible answer. This is what the "right method" scoring layer exists to catch.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 2026A1 / A2
APPENDIX · NOT PRESENTED · FOR Q&A

Two memories, filled offline, fused at query time.

WHAT'S STORED — BOTH IN CHROMADB

1 · Tool memory

Holds
Past question → SQL pairs, plus the tables each one used.
Used for
Few-shot examples. Cosine similarity, top 5. Successful queries are written back, so the corpus grows from real use.

2 · Text memory

Holds
Domain knowledge — schema docs, what the antiSMASH terms mean.
Used for
Semantic search with a category filter, top 5.
HOW A LOOKUP RUNS
1Two searches, not one
Keyword (BM25) and meaning (embeddings) run over the same store.
2Fuse by rank, not by score
Reciprocal rank fusion. The two scores aren't on the same scale, so ranks are the only honest way to merge them.
3Re-score the shortlist
A cross-encoder reads question and candidate together and reorders them.
Why two memories instead of one
"What did a similar query look like?" and "what does this term mean?" need different neighbours. One store blurs both.
Why hybrid, and why a reranker
Keyword misses paraphrase. Embeddings miss exact table names. And the most similar examples aren't the most useful.
BENCHMARKING OFFLINE LLM AGENTS · PYCON 2026A2 / A2