[Discuss][CLI] Add an accuracy benchmark for the AI CLI to the main repo? (100 tasks, 3 verdict gates incl. real execution, repair-loop measurement) #11549

Search before asking

• I had searched in the issues and found no similar issues.

Description

While working on the SeaTunnel AI CLI (#10789, #11109), we built an accuracy benchmark to answer a question the project currently has no way to answer quantitatively: when the CLI's prompts, connector metadata, or repair logic change, does config-generation accuracy actually improve — and by how much, for which models?

We would like to discuss whether the community wants this benchmark to live in the main repository (e.g. under seatunnel-cli/benchmark/), before we open a PR.

What it is

A standalone, deterministic evaluation harness for the AI CLI (pure additions — no product code changes):

• 100 tasks in three complexity tiers (20 simple / 45 medium / 35 complex-DAG), covering 12 ETL scenario categories (batch sync, file ETL, streaming ingest, CDC, multi-pipeline, conditional routing, ...), with 10 Chinese-language prompts and 18 "rule probes" targeting known LLM failure modes (conditional option misuse, BATCH-vs-STREAMING inference, plugin_input/plugin_output wiring).
• Three verdict gates mirroring the CLI's own check → dry-run → run pipeline:
  • L1 static (HOCON parse + connector metadata + assertions, offline);
  • L2 engine dry-run (--dry-run static, per [Feature][Zeta] Implement proper dry-run mode with progressive validation layer0 #10763);
  • L3 real execution — the generated config runs on the official apache/seatunnel Docker image against a docker-compose data environment (MySQL/PostgreSQL/Kafka/ClickHouse/Elasticsearch/MinIO, optional Doris/StarRocks), batch verdict by exit code, streaming by 60s health.
• Repair-loop measurement: on failure, the failing gate's real error output is fed to the CLI's own repair agent (max 3 rounds), producing pass@1 / pass@≤3 and per-gate repair conversion rates.
• All verdicts are deterministic (exit codes, liveness, regex assertions) — no LLM-as-judge. Reports are stamped with the CLI version + git commit, so any CLI change can be A/B measured. Credentials are env-vars only (ASF compliant); results directories are git-ignored.

What it found (evidence it earns its place)

A full run across 7 Bedrock models (Claude Fable 5 / Opus 4.8 / Sonnet 5, GPT-5.6 Terra / Sol, Qwen3-Coder-Next, DeepSeek V3.2) produced actionable results:

• Static rankings invert under real execution: the static-gate leader dropped to last among the top-3 when configs actually ran (93% static → 74% real), while the static fixed developing workflow #3 won at 85% — "looks correct" and "actually runs" are different model capabilities, and only L3 distinguishes them.
• 13 tasks failed for every model — clustering into four concrete CLI improvement targets (Doris/StarRocks option knowledge, PostgreSQL-CDC prerequisites, conditional-routing golden examples, complex DAG wiring), i.e. a data-driven roadmap for prompt/metadata work.
• The benchmark run itself surfaced four real defects, three now fixed in [Fix][CLI] Fix Bedrock compatibility: temperature rejection and engine template variables #11509 and one each reported in [Bug] [Docker] opengauss-jdbc in image lib/ hijacks org.postgresql.Driver — all PostgreSQL connections fail with "Protocol error. Session setup failed" #11510 (opengauss-jdbc hijacking the PostgreSQL driver in the official image) and [Bug] [Core] Released 2.3.13 --check is a no-op stub: invalid configs exit 0 (dev fixed by #10763, needs backport or release-note warning) #11511 (released --check is a no-op stub).

Discussion points

1. In-repo or separate? Options: (a) seatunnel-cli/benchmark/ in the main repo (versioned with the CLI it measures — our preference, since the CLI-version stamp makes each PR's accuracy impact measurable); (b) the apache/seatunnel-tools repo; (c) keep it external.
2. Scope of CI integration (optional, later): a 10–15 task L1-only smoke subset could gate CLI PRs at zero API cost; the full suite needs LLM credentials and is only meaningful as a manually-triggered / release-time job.
3. Maintenance expectations: tasks are declarative JSON; adding a scenario is a JSON entry, no harness changes. Docker environment pins the same image versions the e2e suite uses.

If the direction is agreed, we will open a PR with the harness + tasks + docs (roughly +4,000 lines, all new files), followed by a docs PR adding a model-selection guide based on the results.

Usage Scenario

CLI contributors measuring accuracy impact of prompt/metadata/repair changes; users choosing an LLM for the CLI based on measured (not anecdotal) accuracy.

Related issues

#10789 (AI CLI), #10763 (dry-run), #11508/#11509 (fixes found by this benchmark), #11510, #11511, #11547

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct