[Test][SqlServer-CDC] Add multi-table e2e testcase - #11054
Conversation
|
+1 |
DanielLeens
left a comment
There was a problem hiding this comment.
Thanks for adding coverage for the multi-table SQL Server CDC path. I pulled the current head locally and traced the full path from the new E2E config through the CDC source and into the JDBC sink auto-create flow. The scenario is valuable, but the current test still has one merge-blocking issue on the latest head.
What This PR Solves
- User pain: the existing SQL Server CDC E2E coverage does not exercise one source capturing multiple tables and routing them into multiple sink tables in the same job.
- Fix approach: add a new multi-table E2E, extend the DDL with
full_types_2, and run a single CDC source against two SQL Server tables withtablePrefix = "sink_". - One-line summary: the target scenario is worth covering, but the current test couples multi-table CDC routing with SQL Server sink auto-create schema semantics, and that combination is failing deterministically in CI.
Runtime Chain Rechecked
Test setup
-> SqlServerCDCIT.initializeSqlServerTable(...) [SqlServerCDCIT.java:573-603]
-> executes ddl/column_type_test.sql
-> creates full_types_2 [column_type_test.sql:78-126]
Job startup
-> sqlservercdc_to_sqlserver_with_multi_table_mode_two_table.conf [28-56]
-> source.table-names = [full_types, full_types_2]
-> sink.generate_sink_sql = true
-> sink.tablePrefix = "sink_"
Sink auto-create path
-> JdbcSinkFactory.createSink() [JdbcSinkFactory.java:72-125, 195-205]
-> rebuilds sink TableIdentifier from the source CatalogTable
-> JdbcSaveModeHandler.createTable() [JdbcSaveModeHandler.java:45-49]
-> catalog.createTable(...)
SQL Server type mapping
-> SqlserverTypeMapper.mapping(...) [SqlserverTypeMapper.java:50-53]
-> nchar / nvarchar precision is multiplied by 2
-> SqlServerTypeConverter.reconvert(...) [SqlServerTypeConverter.java:189-195]
-> emits NCHAR(length) using that widened length
Assertion phase
-> SqlServerCDCIT.testSqlServerCdcMultiTableE2e() [275-310]
-> directly compares source rows with rows from the auto-created sink tables
-> CI failure: expected <cč > but was <cč >
Findings
Issue 1: the new E2E couples multi-table CDC routing verification with SQL Server sink auto-create character-length semantics, and the current head fails deterministically
- Location:
SqlServerCDCIT.java:275-310,sqlservercdc_to_sqlserver_with_multi_table_mode_two_table.conf:43-56,SqlserverTypeMapper.java:50-53,SqlServerTypeConverter.java:189-195 - Why this is a blocker:
the new test does not only verify “can one CDC source route two tables into two sinks”. Becausegenerate_sink_sql = trueis enabled, it also requires the auto-created SQL Server sink schema to be character-for-character equivalent to the source schema. On the current code path,nchar/nvarcharlengths are widened during schema derivation, so the sink table pads more spaces and the assertion fails withexpected: <cč > but was: <cč >. - Risk:
this is not a timing blip. It is a deterministic CI failure on the current head, so it will keep the PR red and also mixes two different validation goals into one test. - Better fix:
- Option A: if this PR is meant to validate the full “multi-table CDC + auto-create” path, first fix the SQL Server sink auto-create length semantics so the generated sink schema matches the intended assertion target.
- Option B (recommended for minimal scope): if this PR mainly wants multi-table CDC routing coverage, pre-create the sink tables exactly like the existing single-table case and keep the auto-create schema semantics out of this test.
- Severity: High
- Already raised by others: No
Test Coverage
- The new test does hit the intended multi-table CDC path.
- Stability rating: High risk.
- The failure is deterministic in CI, not a flaky wait issue.
- I confirmed the failing signal from the current CI log:
expected: <cč > but was: <cč >atSqlServerCDCIT.java:276.
Merge Conclusion
Conclusion: can merge after fixes
- Blocking items
- Issue 1: decouple or fix the sink schema semantics behind this E2E before merging.
- Suggested follow-up
- None beyond the blocker above.
Overall, I like the direction of adding multi-table coverage here. The part that still blocks merge is that the current assertion target is broader than the scenario the PR is trying to validate, and it is already failing deterministically on the latest head.
DanielLeens
left a comment
There was a problem hiding this comment.
Thanks for the update. I re-reviewed the latest head from scratch against the new SQL Server CDC multi-table E2E path, the DDL setup, and the sink side of the test.
What This PR Solves
- User pain: the existing SQL Server CDC E2E coverage does not exercise one source capturing multiple tables and routing them into multiple sink tables in the same job.
- Fix approach: add a new multi-table E2E, extend the DDL with a second table, and run a single CDC source against both SQL Server tables.
- One-line summary: the earlier sink-auto-create blocker is closed on the latest head because the test now pre-creates the sink tables and focuses on the multi-table routing path itself.
Runtime Chain Rechecked
DDL setup
-> column_type_test.sql
-> create source tables full_types and full_types_2
-> create sink tables sink_full_types and sink_full_types_2
job startup
-> sqlservercdc_to_sqlserver_with_multi_table_mode_two_table.conf
-> source.table-names = [full_types, full_types_2]
-> sink.generate_sink_sql = false
-> sink.tablePrefix = "sink_"
assertion phase
-> SqlServerCDCIT.testSqlServerCdcMultiTableE2e()
-> verify both source tables are captured and written to the matching pre-created sink tables
Findings
I do not see a new source-level blocker on the current head.
The earlier Daniel blocker is closed: this test no longer couples multi-table CDC routing with SQL Server sink auto-create type semantics. By pre-creating the sink tables and disabling generate_sink_sql, the latest revision now exercises the routing scenario it actually wants to validate.
Test Coverage / Stability
- The new E2E now directly covers one source capturing two tables and landing them in two corresponding sink tables.
- Stability rating: Stable.
- I do not see hard sleeps or weak timing assertions in the added path.
- The scenario is now narrower and more deterministic than the previous auto-create version.
CI Note
- The current Build is queued on the latest head.
- I do not see a code-side blocker from my re-review, but merge should still wait for the Build result.
Merge Conclusion
Conclusion: can merge
- Blocking items
- None from my side on the current code revision.
- Suggested follow-up
- Wait for the queued Build result before merging.
Overall, I like this updated test much better. The latest head validates the intended multi-table CDC routing path without dragging unrelated sink-schema semantics into the assertion target.
What does this PR do?
Closes #5678.
This adds a SQL Server CDC multi-table E2E case by:
Validation
Executed in this worktree:
./mvnw spotless:apply./mvnw -q -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-sqlserver-e2e -am -DskipTests verifyTEST_IN_PR=true RUN_ALL_CONTAINER=false RUN_ZETA_CONTAINER=true ./mvnw -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-sqlserver-e2e -Dtest=SqlServerCDCIT#testSqlServerCdcMultiTableE2e -DfailIfNoTests=false testValidation scope notes:
5801, which blocked the SEATUNNEL engine path8081, the local Docker Desktop environment still hit SQL Server memory pressure (Error 701/Error 17300) and Flink taskmanager startup timeouts