[Umbrella][AI CLI] Roadmap Discussion: From Config Generator to Data Integration Copilot #11616

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I had searched in the issues and found no similar issues.

Description

Background

The SeaTunnel AI CLI has completed its first phase. Shipped so far:

• Core capability: natural-language → SeaTunnel config generation, with a Planner → Generator → Validator → Repair multi-agent pipeline ( [Feature][CLI] Add SeaTunnel CLI for natural language config generation #10789, [Improve][Core] Support transform and value constraint option rules in CLI #11109)
• Provider coverage: Anthropic / OpenAI-compatible / AWS Bedrock (compat fixes in [Fix][CLI] Fix Bedrock compatibility: temperature rejection and engine template variables #11509) / bedrock-mantle for Responses-API-only Bedrock models ([Feature][CLI] Add bedrock-mantle provider for OpenAI-family Bedrock models #11548)
• Website documentation suite, en + zh ([Docs][CLI] Add AI CLI documentation section (overview, quickstart, design, model benchmark) #11551, [Docs][CLI] Add "Running the Benchmark Yourself" section to the AI CLI benchmark pages #11605)
• Accuracy benchmark: 100 tasks, layered verdict gates up to real job execution, repair-loop measurement, 7 models evaluated ([Feature][CLI] Add accuracy benchmark for the AI CLI (100 tasks, layered verdict gates, repair-loop measurement) #11553; proposed and accepted in [Discuss][CLI] Add an accuracy benchmark for the AI CLI to the main repo? (100 tasks, 3 verdict gates incl. real execution, repair-loop measurement) #11549; measured results on the website under AI CLI → Model Benchmark)

With the benchmark in place, we now have quantified evidence of where generation fails and why — 13 tasks fail across nearly every model, and the failures cluster into four fixable knowledge gaps. This umbrella proposes a roadmap for what comes next, and invites the community to discuss scope and priorities — and to pick up sub-tasks.

Vision

Evolve the AI CLI along three stages:

Stage 1: Generate accurately   ── the current focus, backed by benchmark data
Stage 2: Operate reliably      ── from generation tool to operations assistant
Stage 3: Broaden scenarios     ── from tool to platform entry point

A guiding principle carried over from the benchmark work: every accuracy-related change must be justified by benchmark deltas. No prompt tweaks by intuition. The benchmark stamps the CLI version + commit into every report, so any change is A/B-comparable on identical tasks.

Stage 1 — Generation Accuracy (active now)

Benchmark analysis identified four systematic knowledge gaps (13 tasks failed by 6–7 of 7 models) and two behavioral defects. Each item below is a small, independent PR.

Combined expectation for 1.1–1.4: +8–13pp overall pass rate per model, to be verified by before/after benchmark runs included in each PR.

Proposed architecture direction: pluggable Skill Packs

Items 1.1–1.4 should not be hardcoded into prompts. Proposal: a declarative per-connector knowledge-pack format (loaded alongside the existing skills/golden-examples layers), so future connector knowledge can be contributed without touching CLI core code — analogous in effort to adding a connector doc page. This creates a low-barrier contribution channel; design sketch to follow in a sub-issue.

Stage 2 — Runtime Operations (design discussion now; implementation after the Stage-1 foundation)

The engine already provides everything needed: diagnostic REST endpoints (/job-info, /logs, /metrics, /thread-dump, checkpoint history) and event push (JobEventHttpReportHandler). Proposed order:

Diagnosis quality is measurable with the same benchmark methodology (inject known failures → score root-cause hit rate), keeping the "no intuition-driven AI features" principle.

Stage 3 — Scenario Expansion (collecting feedback; not scheduled)

How to contribute

• Items marked help wanted / good first issue will get dedicated sub-issues with context and code pointers — comment here to claim one
• Once the Skill Pack format lands, connector-knowledge contribution requires no Python/Java changes — an ideal first contribution
• Every Stage-1 PR should include before/after benchmark subset results (how-to: website → AI CLI → Model Benchmark → "Running the Benchmark Yourself")

Open questions for discussion

1. Stage-2 boundary: where should CLI operations capability stop relative to the Web UI?
2. Is the confidence-tiered auto-repair model (2.5) acceptable to operators, and what defaults are safe?
3. Skill Pack format: YAML-based declarative packs vs. structured markdown — preferences?
4. Which Stage-3 scenario has the most real-world demand from your usage?

I'll break Stage-1 items into sub-issues over the coming weeks and link them here. Feedback on prioritization is very welcome — especially from users running the CLI against real production configs.