I'm Khang Publication

How I Evaluated an Ecommerce Analytics Agent: Answers, SQL, Tool Use, and Safety

Khang Nguyen — Wed, 16 Sep 2026 15:46:19 GMT

Proof boundary. I built this as a capstone for the Google × Kaggle 5-Day AI Agents course. The public repository records 17/17 Google ADK evaluation cases and 12/12 agents-cli evaluation cases passing in a single recorded local run. This is a local evaluation result, not a production deployment, customer outcome, cost study, or latency benchmark.

Metric coverage caveat. The four custom metrics are repository-defined and uneven. intent_satisfaction carries markers for 12 of the 17 ADK cases, so 5 cases pass it without a marker; grounded_response passes on a minimum-length check; tool_use_quality passes on a table-name match in prose. Read the counts as coverage of the recorded suites, not as independent proof of analytical quality. Repeatability across model runs is future work, not a measured result.

The decision I wanted to support

Operational data questions often arrive in plain language: which state has the worst on-time delivery, where freight is highest, or which seller pattern needs review? The useful system behavior is not “sound confident.” It is to choose the right data path, run a bounded query, and return an answer that can be checked.

I built the Olist Ecommerce Analytics Agent around Brazilian ecommerce data in BigQuery. It is a Google ADK multi-agent system for read-only questions across fulfillment, seller operations, customer experience, payments, and geography.

System boundary

Stage	Component	Boundary
Input	Operations question	Plain-language question about fulfillment, seller operations, customer experience, payments, or geography
Routing	ADK routing	Selects the single best agent via `transfer_to_agent`
Agents	Fulfillment, seller operations, customer experience, data analyst	One agent per department, plus direct specialists
Read path	Custom `query_bigquery`	SELECT only; 10 GB processing cap; 30-second timeout
Read path	First-party BigQueryToolset	`WriteMode.PROTECTED` blocks permanent writes to user tables; forecast and anomaly tools may create temporary session artifacts such as models
Output	Cited analysis	Query-backed answer returned to the operator

The custom query_bigquery path accepts SELECT statements and enforces a 10 GB processing cap plus a 30-second timeout. The first-party BigQueryToolset has a different boundary: WriteMode.PROTECTED blocks permanent writes to user tables, but forecast and anomaly tools may create temporary session artifacts such as models. The cap and timeout are not documented across every first-party tool path. These controls narrow risk; they do not make the capstone production-safe.

What I evaluated

I used two local suites recorded in the public repository.

Suite	Recorded result	What it checks
Google ADK evaluation	17/17 cases passed	Fulfillment, seller operations, customer experience, payments, schema/BI, refusal, and cross-domain briefing cases
agents-cli evaluation	12/12 cases passed	Tool-use quality, grounded response, intent satisfaction, and SQL safety

The cases include questions such as late delivery by state, seller review gaps, payment mix, freight by seller state, schema discovery, out-of-scope refusal, and an executive briefing pipeline. A passing count matters only with a boundary: it means these recorded cases passed under this evaluation design. It does not prove performance on new schemas, live operational data, adversarial prompts, changing business definitions, or high-volume traffic.

Why answer quality alone is too weak

For operational analytics, an answer can sound plausible while using the wrong tool, applying the wrong filter, or computing a number that does not reconcile to the source table. I kept four dimensions visible:

Grounded response: answer follows available data rather than invented values.
Tool-use quality: agent selects a relevant data path and parameters.
Intent satisfaction: answer addresses the decision question.
SQL safety: no write query, excessive processing, or unsupported behavior.

This is why I would evaluate an analytics agent as a system, not as a chat response. Future work should add repeatability across model runs, accepted SQL/result traces, semantic tests for ambiguous terms such as “on time,” “late,” “seller,” or “state,” and a human approval record for any proposed action.

What this build does not prove

No production deployment or customer workflow outcome.
No measured business improvement, revenue, stockout reduction, or cost saving.
No current latency or inference-cost benchmark in this article.
No authorization model for a customer warehouse or ERP.
No evidence that an agent should autonomously write purchase orders, inventory transfers, or supplier actions.

The appropriate next boundary is read-only decision support: retrieve the relevant facts, show query-backed reasoning, and let an operator decide whether to act.

Reproduce or inspect it

The public repository includes the ADK app, tool boundary, evaluation datasets/configuration, and commands for re-running the local suites. Start with the repository README, then inspect its test and tool directories before adapting the pattern to another dataset.

If you are building a similar system, begin with a small decision set, define expected answers and allowed queries, add refusal cases, and keep the database access read-only until you can inspect failure traces and human-review requirements.

Sources

How Archify Turns Codebase Maps Into Reviewed Technical Artifacts

Khang Nguyen — Tue, 01 Sep 2026 06:30:26 GMT

Used for: assessing a tool for explaining AI-assisted codebases while preparing this publication workflow. Tested: source review of Archify v2.16.0, 2026-09-01.

Verdict: use when a team needs a reproducible technical map that an AI coding agent can refine and a human can inspect. Avoid it when the job is freehand sketching or a general-purpose diagram editor.

Quick verdict

Best for: architecture reviews, onboarding maps, workflow explanations, and before/after diagram comparisons.
Avoid if: the requirement is WYSIWYG drawing, automatic Mermaid parsing, or hosted collaborative diagram editing.
Cost: Archify is MIT licensed. Running it still requires a compatible coding-agent environment and Node.js where its renderer runs.
Main limitation: validation can check the supplied diagram structure and rendering rules; it cannot prove that an agent’s model of a real system is complete or operationally correct.

AI-assisted coding makes it cheap to add routes, workers, integrations, and data stores. It does not make the resulting system easy to explain. The usual answer is a diagram, but screenshots drift, manual edits are difficult to review, and a polished drawing can still overstate what a codebase does.

Archify is built around a narrower job: turn a codebase analysis or system description into an inspectable technical artifact. An agent produces typed JSON intermediate representation (IR); Archify validates it and deterministically compiles it to self-contained HTML and SVG. The result can be opened locally, shared as one file, and exported in common formats.

What I used it for

This was a research-only evaluation, not a claim that Archify has been run against every system in this publication. The practical question was narrower: what should a developer use when a growing project needs a system map that is more durable than a whiteboard screenshot?

That question appears early in small products. A project may begin with a page and an API. Then it accumulates authentication, queues, caches, background work, analytics, and third-party services. The code can remain understandable file by file while the overall path becomes hard to describe. The same issue shows up in applied builds: a public project such as the Olist Ecommerce Analytics Agent is easier to evaluate when its roles, data boundary, and query constraints can be presented as a concrete system map rather than a long paragraph.

Archify supports five diagram types, each aimed at a different question:

Architecture diagrams for components, services, storage, boundaries, and a primary path.
Workflow diagrams for ordered steps, approvals, branches, exceptions, and runbooks.
Sequence diagrams for calls and returns over time, such as an authentication check or cache fallback.
Data-flow diagrams for sources, transforms, stores, consumers, and sensitivity boundaries.
Lifecycle diagrams for states, retries, waits, cancellation, and terminal outcomes.

Choosing a type matters. An architecture map should not pretend to explain timing. A sequence should not attempt to carry every infrastructure detail. A lifecycle should make waits and terminal states legible instead of hiding them behind generic arrows.

Setup or workflow

The repository documents the skill installation command:

npx skills add tt-a1i/archify -g

Archify documents workflows for Cursor, Claude Code, Codex CLI, and OpenCode. A useful first prompt is deliberately scoped: ask for the core components, one primary path, external dependencies, and trust boundaries. Resist the urge to ask for every class and service at once. A map with too many edges is not more truthful; it is only harder to read.

For an unfamiliar repository, a bounded request can look like this:

Analyze this repository, then use Archify to create a high-level runtime architecture diagram.
Show 8–12 core components, one primary path, external dependencies, and trust boundaries.
Put supporting detail in cards instead of adding more edges.

The agent then creates typed JSON IR. Archify’s documented process validates schema, layout, HTML/SVG rendering, routes, and label-to-route clearance before a showcase-quality artifact replaces a prior verified output. This is the useful boundary: an agent can propose the interpretation, while the renderer applies repeatable checks to the artifact it delivers.

After that, refinement should stay specific. Add one dependency. Move one component. Highlight one rollback path. A controlled change is easier to review than regenerating the entire diagram from a vague request.

What worked in the documented design

Archify’s strongest idea is not a visual preset. It is the separation between meaning and rendering. Typed IR provides a source that can be inspected and revised; deterministic compilation produces a consistent artifact from that source; validation keeps layout and route constraints inside the delivery loop.

The resulting HTML supports more than a static view. The documented viewer can search nodes, trace authored routes, inspect upstream or downstream reach, compare semantic roles, and play guided stories. Those interactions only reuse authored nodes and relationships. That constraint is important: a technical viewer should not invent topology while a reader is trying to understand a real system.

Architecture Delta is another useful example of restraint. Given validated Before, Delta, and After snapshots, it reports authored additions, removals, changes, moves, and reroutes. It does not claim performance impact, security improvement, merge safety, or runtime risk unless evidence outside the diagram supports those conclusions. For code review, that distinction is healthier than treating a changed diagram as proof that a change is safe.

The export surface is practical: Archify documents self-contained HTML plus PNG, SVG, WebM, and 1200×630 share cards. That suits an architecture review, a README, a release note, or a team handoff without requiring a separate hosted diagram product.

What broke or costs more than expected

A validated diagram is not a verified system. The renderer can reject malformed source, confusing routes, and visual collisions. It cannot know whether the agent missed a hidden queue, misunderstood a permissions boundary, or inferred behavior from an incomplete repository scan. Treat the diagram as a communication artifact that deserves code review, not as runtime monitoring or a substitute for tests.

There is also an intentional scope boundary. Archify is not a general drawing editor, a Mermaid theme, automatic Mermaid parsing, hosted sharing, or a WYSIWYG canvas. That is not a defect when the need is repeatable technical communication. It is a poor fit when a workshop needs people to sketch freely and negotiate the model live.

The same rule applies to source evidence. Archify can link evidence-backed nodes to revision-verified public source when that mode is requested. Ordinary diagrams may remain source-free. Readers should ask which kind of map they are viewing before treating it as a source-level audit.

Alternatives and trade-offs

Mermaid is the lower-friction option when plain-text diagrams inside documentation are enough. A design canvas is better when manual composition and collaborative sketching are the main jobs. A slide can be better when the audience needs a short, curated narrative rather than an explorable map.

Archify fits between those choices. It is for teams that already work through coding agents and want a diagram source, renderer, validator, and viewer in the same workflow. It brings more structure than a hand-drawn diagram and more visual interaction than a text-only diagram definition. That structure costs prompt discipline and a review step.

Tips to reuse

Start with one question per map: system structure, one workflow, one request path, one data journey, or one lifecycle.
Keep the first architecture map to 8–12 core components. Put nuance in cards instead of crossing every possible edge.
Ask the agent to state unknowns. Missing information is better than invented topology.
Generate a Before snapshot before a structural change and an After snapshot afterward. Use Delta to explain the authored difference, not to claim operational impact.
Review the artifact alongside source and tests. A map helps people reason; it does not replace evidence.

For broader context on using AI tools in real project work, see this publication’s project notes and public build records.