[Feature][API] Add EXTENSION operator for pluggable validation in ConditionOperator - #11048
Conversation
|
@dybyte @davidzollo @yzeng1618 Can you help me review it? thank you very much. |
…ct signature for the EXTENSION method
DanielLeens
left a comment
There was a problem hiding this comment.
Thanks for working on this. I went through the full diff locally and traced both the runtime validation path and the existing metadata consumer paths. The core EXTENSION hook makes sense, but I still see two blockers before this is ready to merge.
What This PR Solves
- User pain: the built-in
OptionRuleoperators are not expressive enough for structural checks such asList<Map>child configs or nested cross-key validation. - Fix approach: add
ConditionOperator.EXTENSION, carry aConditionExtension<T>insideCondition, evaluate it fromConditionEvaluators, and expose the description through the RESToption-rulespath. - One-line summary: the runtime validation path is wired correctly, but the current head has not finished converging all existing metadata consumers on the new extension semantics.
Runtime Chain Rechecked
Job submission validation
-> FactoryUtil.createAndPrepareSource()/createAndPrepareSink()/createAndPrepareTransform()
-> ConfigValidator.validate(factory.optionRule())
-> ConfigValidator.validate(Condition) [ConfigValidator.java:458-479]
-> ConditionEvaluators.evaluate(cur, config)
-> EXTENSION delegates to ext.evaluate(cfg, value) [ConditionEvaluators.java:172-177]
REST metadata path
-> GET /option-rules
-> OptionRulesService.buildResponse()
-> toConditionNode()
-> EXTENSION writes description() into expectValue [OptionRulesService.java:285-297]
CLI metadata export path
-> SeaTunnelMetadataExporter.main() [SeaTunnelMetadataExporter.java:27-40]
-> MetadataExportCommand.exportCondition() [MetadataExportCommand.java:417-444]
-> only writes expectValue when condition.getExpectValue() != null
-> EXTENSION description is still dropped here
Findings
Issue 1: the existing CLI metadata exporter still drops the extension description from the structured condition tree
- Location:
seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/command/MetadataExportCommand.java:417-444 - Why this is a blocker:
OptionRulesServicenow exposes the extension description on the REST path, butSeaTunnelMetadataExporterstill serializes the old condition shape. That means one official metadata consumer gets the new semantics while another official metadata consumer still loses them. - Better fix:
mirror the same EXTENSION handling inMetadataExportCommand.exportCondition()that you already added inOptionRulesService.toConditionNode(), and add a focused metadata-export regression test for an extension-based rule. - Severity: High
- Already raised by others: No
Issue 2: the /option-rules REST docs still describe the old condition-tree contract
- Location:
docs/en/engines/zeta/rest-api-v2.md:153-155,docs/zh/engines/zeta/rest-api-v2.md:151-153 - Why this matters:
the current implementation now emitsconditionOperator=EXTENSION,conditionOperatorCategory=EXTENSION,compareOperator=extension, and usesexpectValuefor the human-readable extension description. The REST docs still say non-comparison conditions leave these fieldsnull. - Better fix:
update both the English and Chinese REST API docs to explain the EXTENSION shape and add a minimal JSON example. - Severity: Medium
- Already raised by others: No
Test Coverage
- The runtime validation tests in
ConfigValidatorTest/OptionRuleTestlook good for the evaluation path itself. - The missing coverage is on the metadata consumer side: there is still no regression test for REST/CLI export of an extension-based rule.
- Stability rating: Stable. The added tests are pure in-memory validation checks with no timing, network, or environment coupling.
Merge Conclusion
Conclusion: can merge after fixes
- Blocking items
- Issue 1: finish the existing CLI metadata-export path so extension-based rules are represented consistently across official metadata consumers.
- Suggested but non-blocking follow-up
- Issue 2: sync the REST API docs with the new EXTENSION condition-tree shape.
Overall, I think the runtime validation design is on the right track. The remaining gap is not the ConditionExtension evaluation itself, but the fact that the metadata consumer chain is still only partially updated on the current head.
DanielLeens
left a comment
There was a problem hiding this comment.
Thanks for the update. I re-reviewed the latest head end to end, from the runtime validation path down to both official metadata consumers. The two blockers from my last round are fixed on the current head, so I don't see anything that should block merge now.
What This PR Solves
- User pain: the built-in
OptionRuleoperators are still not expressive enough for structural validation such asList<Map>children or nested cross-key checks. - Fix approach: add
ConditionOperator.EXTENSION, carry aConditionExtension<T>insideCondition, evaluate it fromConditionEvaluators, and expose the rule description through both REST metadata and the CLI metadata exporter. - One-line summary: the extension hook is now wired through the real validation path and the two official metadata consumer paths consistently.
Runtime Chain Rechecked
Job submission validation
-> FactoryUtil.createAndPrepareSource()/Sink()/Transform()
-> factory.optionRule()
-> ConfigValidator.validate(rule)
-> ConfigValidator.validate(Condition) [ConfigValidator.java:458-479]
-> ConditionEvaluators.evaluate(cur, config)
-> EXTENSION delegates to ConditionExtension.evaluate(cfg, value)
REST metadata path
-> GET /option-rules
-> OptionRulesService.buildResponse()
-> toConditionNode()
-> for EXTENSION, expectValue = condition.getExtension().description()
CLI metadata export path
-> SeaTunnelMetadataExporter.main()
-> MetadataExportCommand.exportConnector()
-> exportCondition()
-> for EXTENSION, expectValue = condition.getExtension().description()
Findings
I did not find any merge-blocking issue on the current head.
Issue 1: the Extension docs still describe the evaluate() trigger point a bit too broadly
- Location:
docs/en/architecture/configuration-and-option-system.md:443-444,docs/zh/architecture/configuration-and-option-system.md:443-444 - Why this matters:
the docs currently sayConditionExtension.evaluate()runs during both job submission and REST metadata queries. Looking at the actual code path, REST metadata only consumesdescription()throughOptionRulesService.toConditionNode()and does not callConditionEvaluators.evaluate(...). - Suggestion:
tighten the wording to say thatevaluate()runs during validation/job submission, while REST metadata queries only serialize the human-readable rule description. - Severity: Medium
- Already raised by others: No
Issue 2: the new public extension entry points could use one more layer of API docs
- Location:
seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/ConditionExtension.java:22,seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/Conditions.java:144-145 - Why this matters:
this is now a public extension surface for connector authors, so a short type-level Javadoc onConditionExtensionplus a method-level Javadoc onConditions.extension(...)would make the contract easier to use correctly. - Suggestion:
add a concise API-level description for the interface and the factory method, especially around the pure-validation expectation and error-reporting guidance. - Severity: Low
- Already raised by others: No
Test Coverage
- The runtime coverage in
ConfigValidatorTest/OptionRuleTestlooks good for the validation path itself. MetadataExportCommandTestandOptionRulesServiceTestnow cover the two metadata consumer paths that were missing before.- Stability rating: Stable. The added tests are all in-memory unit tests with no timing, network, or environment coupling.
Merge Conclusion
Conclusion: can merge
- Blocking items
- None from my side.
- Suggested follow-up
- Issue 1: tighten the wording in the Extension docs so it matches the actual runtime contract.
- Issue 2: add a bit more API-level Javadoc around the new public extension entry points.
Overall, I think this is in good shape now. The main runtime path works, the REST and CLI metadata views are aligned, and the remaining items are documentation polish rather than correctness blockers.
DanielLeens
left a comment
There was a problem hiding this comment.
Thanks for the update. I re-reviewed the latest head from scratch against dev, including the runtime validation path and both official metadata-consumer paths. The newest head includes a CI-trigger commit, but after rechecking the full current diff I still do not see a code blocker.
What This PR Solves
- User pain: the built-in
OptionRuleoperators are not expressive enough for some real validation cases, such asList<Map>structural checks or nested cross-key validation. - Fix approach: add
ConditionOperator.EXTENSIONplusConditionExtension<T>, route it through the normal validation chain, and expose its human-readable description consistently through REST metadata and CLI metadata export. - One-line summary: SeaTunnel now has a first-class pluggable validation hook, and the current head keeps the runtime path and both official metadata views aligned.
Runtime Chain Rechecked
Job submission validation
-> FactoryUtil.createAndPrepareSource()/Sink()/Transform()
-> ConfigValidator.validate(factory.optionRule())
-> ConfigValidator.validate(Condition)
-> ConditionEvaluators.evaluate(...)
-> EXTENSION -> ConditionExtension.evaluate(config, value)
REST metadata path
-> OptionRulesService.toConditionNode() [OptionRulesService.java:272-297]
-> for EXTENSION, serializes description() into expectValue
CLI metadata export path
-> MetadataExportCommand.exportCondition() [MetadataExportCommand.java:417-445]
-> for EXTENSION, serializes description() into expectValue
Review Result
I did not find a merge-blocking issue on the current head.
Key evidence I rechecked locally:
ConditionExtension.java:22-62now documents the contract clearly, including thatevaluate()runs during validation while metadata paths only serializedescription().Conditions.java:142-145exposes the extension hook cleanly.OptionRulesService.java:272-297andMetadataExportCommand.java:417-445now handleEXTENSIONconsistently.- The metadata-consumer gap from my earlier round is closed on the latest head.
Test Coverage
- The runtime validation coverage is good.
- The previously missing metadata-consumer coverage is now present.
- Stability rating: Stable.
- The new tests are in-memory unit tests.
- I do not see timing, environment, network, or shared-state flaky-test patterns here.
Merge Conclusion
Conclusion: can merge
- Blocking items
- None from my side.
- Suggested follow-up
- None from my side.
Overall, this looks ready from a code-review perspective. The validation hook is wired correctly, the REST and CLI metadata outputs are aligned, and I do not see a protocol or compatibility blocker on the current head. The current Build is still queued, so merge should still wait for the check result, but I do not see a code-side reason to hold this revision.
yzeng1618
left a comment
There was a problem hiding this comment.
There are minor details that do not affect functionality and can be optimized later.
Purpose of this pull request
Add
ConditionOperator.EXTENSIONto allow connectors to plug custom validation logic into the existingConditionframework.Problem:
Built-in operators (
greaterOrEqual,notBlank,notEmpty, etc.) handle single-value checks well, but connectors sometimes need structural validation on complex types:List<Map<String, Object>>contains required keys (field,type)table_configschild configs have valid numeric ranges across nested mapsToday this kind of logic is scattered in imperative code like
buildWithConfig(), making it invisible to the declarativeOptionRulesystem, REST metadata API, and CLI tooling.Solution:
Introduce
ConditionExtension<T>interface +ConditionOperator.EXTENSIONenum constant. Connector developers implement the interface and wire it viaConditions.extension(option, ext).The EXTENSION operator is a first-class citizen in the existing pipeline:
valueConstraintsevaluation inConfigValidator— no changes toOptionRuleorConfigValidatorConditions.extension(Option<T>, ConditionExtension<T>)enforces type safety at compile time.and()/.or()and mixes freely with any built-in operatorconditionOperator: "EXTENSION"withexpectValuefromdescription()falsefor auto-composed messages, or throwOptionValidationExceptionfor context-rich detailsUsage:
Does this PR introduce any user-facing change?
No.
How was this patch tested?
./mvnw -pl seatunnel-api test