Skip to content

[Feature][API] Add EXTENSION operator for pluggable validation in ConditionOperator - #11048

Merged
davidzollo merged 10 commits into
apache:devfrom
nzw921rx:feature/optionrule-condition-extension-operator
Jun 12, 2026
Merged

[Feature][API] Add EXTENSION operator for pluggable validation in ConditionOperator#11048
davidzollo merged 10 commits into
apache:devfrom
nzw921rx:feature/optionrule-condition-extension-operator

Conversation

@nzw921rx

@nzw921rx nzw921rx commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Purpose of this pull request

Add ConditionOperator.EXTENSION to allow connectors to plug custom validation logic into the existing Condition framework.

Problem:

Built-in operators (greaterOrEqual, notBlank, notEmpty, etc.) handle single-value checks well, but connectors sometimes need structural validation on complex types:

  • Validate every entry in List<Map<String, Object>> contains required keys (field, type)
  • Validate table_configs child configs have valid numeric ranges across nested maps

Today this kind of logic is scattered in imperative code like buildWithConfig(), making it invisible to the declarative OptionRule system, REST metadata API, and CLI tooling.

Solution:

Introduce ConditionExtension<T> interface + ConditionOperator.EXTENSION enum constant. Connector developers implement the interface and wire it via Conditions.extension(option, ext).

The EXTENSION operator is a first-class citizen in the existing pipeline:

  • Reuses valueConstraints evaluation in ConfigValidator — no changes to OptionRule or ConfigValidator
  • Conditions.extension(Option<T>, ConditionExtension<T>) enforces type safety at compile time
  • Chains with .and() / .or() and mixes freely with any built-in operator
  • REST metadata exposes conditionOperator: "EXTENSION" with expectValue from description()
  • Supports two error reporting modes: return false for auto-composed messages, or throw OptionValidationException for context-rich details

Usage:

// Inline — validate encoded API key format (decode + check structure)
.optional(API_KEY_ENCODED, Conditions.extension(API_KEY_ENCODED,
        new ConditionExtension<String>() {
            @Override
            public String description() {
                return "must be Base64-encoded 'id:api_key'";
            }

            @Override
            public boolean evaluate(ReadonlyConfig cfg, String v)
                    throws OptionValidationException {
                try {
                    return new String(Base64.getDecoder().decode(v)).contains(":");
                } catch (IllegalArgumentException e) {
                    return false;
                }
            }
        }))

// Static inner class — validate List<Map> structure with cross-element uniqueness
static class TableConfigsValidator
        implements ConditionExtension<List<Map<String, Object>>> {
    @Override
    public String description() {
        return "each entry must contain a non-empty 'table_name', "
                + "and all table names must be unique";
    }

    @Override
    public boolean evaluate(ReadonlyConfig config, List<Map<String, Object>> value)
            throws OptionValidationException {
        if (value.isEmpty()) {
            return false;
        }
        Set<String> seen = new HashSet<>();
        for (Map<String, Object> entry : value) {
            Object name = entry.get("table_name");
            if (!(name instanceof String) || ((String) name).isEmpty()) {
                return false;
            }
            if (!seen.add((String) name)) {
                return false;
            }
        }
        return true;
    }
}

.exclusive(TABLE_CONFIGS, SCHEMA)
.optional(TABLE_CONFIGS,
        Conditions.extension(TABLE_CONFIGS, new TableConfigsValidator()))

// Chain with built-in operators — extension + numeric range on the same option
.required(BATCH_SIZE,
        Conditions.greaterOrEqual(BATCH_SIZE, 1)
                .and(Conditions.extension(BATCH_SIZE, new ConditionExtension<Integer>() {
                    @Override
                    public String description() {
                        return "must be a power of 2";
                    }

                    @Override
                    public boolean evaluate(ReadonlyConfig cfg, Integer v)
                            throws OptionValidationException {
                        return (v & (v - 1)) == 0;
                    }
                })))

Does this PR introduce any user-facing change?

No.

How was this patch tested?

./mvnw -pl seatunnel-api test

@nzw921rx

Copy link
Copy Markdown
Collaborator Author

@dybyte @davidzollo @yzeng1618 Can you help me review it? thank you very much.

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this. I went through the full diff locally and traced both the runtime validation path and the existing metadata consumer paths. The core EXTENSION hook makes sense, but I still see two blockers before this is ready to merge.

What This PR Solves

  • User pain: the built-in OptionRule operators are not expressive enough for structural checks such as List<Map> child configs or nested cross-key validation.
  • Fix approach: add ConditionOperator.EXTENSION, carry a ConditionExtension<T> inside Condition, evaluate it from ConditionEvaluators, and expose the description through the REST option-rules path.
  • One-line summary: the runtime validation path is wired correctly, but the current head has not finished converging all existing metadata consumers on the new extension semantics.

Runtime Chain Rechecked

Job submission validation
  -> FactoryUtil.createAndPrepareSource()/createAndPrepareSink()/createAndPrepareTransform()
      -> ConfigValidator.validate(factory.optionRule())
          -> ConfigValidator.validate(Condition) [ConfigValidator.java:458-479]
              -> ConditionEvaluators.evaluate(cur, config)
                  -> EXTENSION delegates to ext.evaluate(cfg, value) [ConditionEvaluators.java:172-177]

REST metadata path
  -> GET /option-rules
      -> OptionRulesService.buildResponse()
      -> toConditionNode()
          -> EXTENSION writes description() into expectValue [OptionRulesService.java:285-297]

CLI metadata export path
  -> SeaTunnelMetadataExporter.main() [SeaTunnelMetadataExporter.java:27-40]
      -> MetadataExportCommand.exportCondition() [MetadataExportCommand.java:417-444]
          -> only writes expectValue when condition.getExpectValue() != null
          -> EXTENSION description is still dropped here

Findings

Issue 1: the existing CLI metadata exporter still drops the extension description from the structured condition tree

  • Location: seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/command/MetadataExportCommand.java:417-444
  • Why this is a blocker:
    OptionRulesService now exposes the extension description on the REST path, but SeaTunnelMetadataExporter still serializes the old condition shape. That means one official metadata consumer gets the new semantics while another official metadata consumer still loses them.
  • Better fix:
    mirror the same EXTENSION handling in MetadataExportCommand.exportCondition() that you already added in OptionRulesService.toConditionNode(), and add a focused metadata-export regression test for an extension-based rule.
  • Severity: High
  • Already raised by others: No

Issue 2: the /option-rules REST docs still describe the old condition-tree contract

  • Location: docs/en/engines/zeta/rest-api-v2.md:153-155, docs/zh/engines/zeta/rest-api-v2.md:151-153
  • Why this matters:
    the current implementation now emits conditionOperator=EXTENSION, conditionOperatorCategory=EXTENSION, compareOperator=extension, and uses expectValue for the human-readable extension description. The REST docs still say non-comparison conditions leave these fields null.
  • Better fix:
    update both the English and Chinese REST API docs to explain the EXTENSION shape and add a minimal JSON example.
  • Severity: Medium
  • Already raised by others: No

Test Coverage

  • The runtime validation tests in ConfigValidatorTest / OptionRuleTest look good for the evaluation path itself.
  • The missing coverage is on the metadata consumer side: there is still no regression test for REST/CLI export of an extension-based rule.
  • Stability rating: Stable. The added tests are pure in-memory validation checks with no timing, network, or environment coupling.

Merge Conclusion

Conclusion: can merge after fixes

  1. Blocking items
  • Issue 1: finish the existing CLI metadata-export path so extension-based rules are represented consistently across official metadata consumers.
  1. Suggested but non-blocking follow-up
  • Issue 2: sync the REST API docs with the new EXTENSION condition-tree shape.

Overall, I think the runtime validation design is on the right track. The remaining gap is not the ConditionExtension evaluation itself, but the fact that the metadata consumer chain is still only partially updated on the current head.

@github-actions github-actions Bot added the core SeaTunnel core module label Jun 10, 2026
@nzw921rx
nzw921rx requested a review from DanielLeens June 10, 2026 13:04

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. I re-reviewed the latest head end to end, from the runtime validation path down to both official metadata consumers. The two blockers from my last round are fixed on the current head, so I don't see anything that should block merge now.

What This PR Solves

  • User pain: the built-in OptionRule operators are still not expressive enough for structural validation such as List<Map> children or nested cross-key checks.
  • Fix approach: add ConditionOperator.EXTENSION, carry a ConditionExtension<T> inside Condition, evaluate it from ConditionEvaluators, and expose the rule description through both REST metadata and the CLI metadata exporter.
  • One-line summary: the extension hook is now wired through the real validation path and the two official metadata consumer paths consistently.

Runtime Chain Rechecked

Job submission validation
  -> FactoryUtil.createAndPrepareSource()/Sink()/Transform()
      -> factory.optionRule()
      -> ConfigValidator.validate(rule)
          -> ConfigValidator.validate(Condition) [ConfigValidator.java:458-479]
              -> ConditionEvaluators.evaluate(cur, config)
                  -> EXTENSION delegates to ConditionExtension.evaluate(cfg, value)

REST metadata path
  -> GET /option-rules
      -> OptionRulesService.buildResponse()
          -> toConditionNode()
              -> for EXTENSION, expectValue = condition.getExtension().description()

CLI metadata export path
  -> SeaTunnelMetadataExporter.main()
      -> MetadataExportCommand.exportConnector()
          -> exportCondition()
              -> for EXTENSION, expectValue = condition.getExtension().description()

Findings

I did not find any merge-blocking issue on the current head.

Issue 1: the Extension docs still describe the evaluate() trigger point a bit too broadly

  • Location: docs/en/architecture/configuration-and-option-system.md:443-444, docs/zh/architecture/configuration-and-option-system.md:443-444
  • Why this matters:
    the docs currently say ConditionExtension.evaluate() runs during both job submission and REST metadata queries. Looking at the actual code path, REST metadata only consumes description() through OptionRulesService.toConditionNode() and does not call ConditionEvaluators.evaluate(...).
  • Suggestion:
    tighten the wording to say that evaluate() runs during validation/job submission, while REST metadata queries only serialize the human-readable rule description.
  • Severity: Medium
  • Already raised by others: No

Issue 2: the new public extension entry points could use one more layer of API docs

  • Location: seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/ConditionExtension.java:22, seatunnel-api/src/main/java/org/apache/seatunnel/api/configuration/util/Conditions.java:144-145
  • Why this matters:
    this is now a public extension surface for connector authors, so a short type-level Javadoc on ConditionExtension plus a method-level Javadoc on Conditions.extension(...) would make the contract easier to use correctly.
  • Suggestion:
    add a concise API-level description for the interface and the factory method, especially around the pure-validation expectation and error-reporting guidance.
  • Severity: Low
  • Already raised by others: No

Test Coverage

  • The runtime coverage in ConfigValidatorTest / OptionRuleTest looks good for the validation path itself.
  • MetadataExportCommandTest and OptionRulesServiceTest now cover the two metadata consumer paths that were missing before.
  • Stability rating: Stable. The added tests are all in-memory unit tests with no timing, network, or environment coupling.

Merge Conclusion

Conclusion: can merge

  1. Blocking items
  • None from my side.
  1. Suggested follow-up
  • Issue 1: tighten the wording in the Extension docs so it matches the actual runtime contract.
  • Issue 2: add a bit more API-level Javadoc around the new public extension entry points.

Overall, I think this is in good shape now. The main runtime path works, the REST and CLI metadata views are aligned, and the remaining items are documentation polish rather than correctness blockers.

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. I re-reviewed the latest head from scratch against dev, including the runtime validation path and both official metadata-consumer paths. The newest head includes a CI-trigger commit, but after rechecking the full current diff I still do not see a code blocker.

What This PR Solves

  • User pain: the built-in OptionRule operators are not expressive enough for some real validation cases, such as List<Map> structural checks or nested cross-key validation.
  • Fix approach: add ConditionOperator.EXTENSION plus ConditionExtension<T>, route it through the normal validation chain, and expose its human-readable description consistently through REST metadata and CLI metadata export.
  • One-line summary: SeaTunnel now has a first-class pluggable validation hook, and the current head keeps the runtime path and both official metadata views aligned.

Runtime Chain Rechecked

Job submission validation
  -> FactoryUtil.createAndPrepareSource()/Sink()/Transform()
      -> ConfigValidator.validate(factory.optionRule())
          -> ConfigValidator.validate(Condition)
              -> ConditionEvaluators.evaluate(...)
                  -> EXTENSION -> ConditionExtension.evaluate(config, value)

REST metadata path
  -> OptionRulesService.toConditionNode() [OptionRulesService.java:272-297]
      -> for EXTENSION, serializes description() into expectValue

CLI metadata export path
  -> MetadataExportCommand.exportCondition() [MetadataExportCommand.java:417-445]
      -> for EXTENSION, serializes description() into expectValue

Review Result

I did not find a merge-blocking issue on the current head.

Key evidence I rechecked locally:

  • ConditionExtension.java:22-62 now documents the contract clearly, including that evaluate() runs during validation while metadata paths only serialize description().
  • Conditions.java:142-145 exposes the extension hook cleanly.
  • OptionRulesService.java:272-297 and MetadataExportCommand.java:417-445 now handle EXTENSION consistently.
  • The metadata-consumer gap from my earlier round is closed on the latest head.

Test Coverage

  • The runtime validation coverage is good.
  • The previously missing metadata-consumer coverage is now present.
  • Stability rating: Stable.
    • The new tests are in-memory unit tests.
    • I do not see timing, environment, network, or shared-state flaky-test patterns here.

Merge Conclusion

Conclusion: can merge

  1. Blocking items
  • None from my side.
  1. Suggested follow-up
  • None from my side.

Overall, this looks ready from a code-review perspective. The validation hook is wired correctly, the REST and CLI metadata outputs are aligned, and I do not see a protocol or compatibility blocker on the current head. The current Build is still queued, so merge should still wait for the check result, but I do not see a code-side reason to hold this revision.

@yzeng1618 yzeng1618 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are minor details that do not affect functionality and can be optimized later.

@davidzollo davidzollo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@davidzollo
davidzollo merged commit e2951f2 into apache:dev Jun 12, 2026
14 checks passed
@nzw921rx
nzw921rx deleted the feature/optionrule-condition-extension-operator branch June 14, 2026 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants