[Fix][Connector-V2] Fix parquet read failure when column name contains Avro-illegal characters - #10960
Conversation
…s Avro-illegal characters Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
|
Please enable CI check following by the instruction https://github.com/apache/seatunnel/pull/10960/checks?check_run_id=77884719128 |
|
Thanks for working on this. I pulled the latest head locally and traced the real parquet source path instead of only checking the minimal repro from the PR body. What this PR fixes
Runtime chain I checkedFindingsIssue 1: the new fallback still cannot read illegal-name parquet files that contain LIST / MAP logical types
Issue 2: the current Build is still red, and the failing jobs have not been proven unrelated yet
Test coverage
Merge conclusionConclusion: can merge after fixes
Overall, this PR fixes a real problem and the minimal repro is now covered, but the current fallback is still incomplete for common complex parquet schemas, so I would not merge this revision yet. |
…llback reader
The native Parquet Group reader fallback (used when column names contain
Avro-illegal characters) previously threw an exception when encountering
LIST or MAP logical types. This adds proper handling for:
- 3-level LIST encoding (standard: group → repeated wrapper → element)
- 2-level LIST encoding (legacy: group → repeated element directly)
- MAP fields (group → repeated key_value → {key, value})
- Element count and repetition index fixes for correct iteration
Also adds a test covering Avro-incompatible column names combined with
LIST fields to verify the fallback path works end-to-end.
DanielLeens
left a comment
There was a problem hiding this comment.
Thanks for the update. I re-reviewed the latest head from the real parquet fallback path.
What this PR solves
- User pain: parquet files with column names that are valid in parquet but illegal in Avro fail immediately on the existing Avro reader path.
- Fix approach: keep the Avro path first, then fall back to a native parquet-group reader when the failure is specifically the Avro-field-name parse problem.
- One-line summary: the current head closes the complex-type gap I called out before, and I did not find a new blocker on the latest revision.
Runtime path I checked
ParquetReadStrategy.read(...)
-> try AvroParquetReader path
-> illegal Avro field name
-> fallback to native Group API reader
-> resolveGroupType(...)
-> LIST -> readList(...)
-> MAP -> readMap(...)
-> nested ROW -> recursive resolveGroupObject(...)
Re-review result
- The old blocker is fixed: the native fallback now handles LIST / MAP / nested row shapes instead of rejecting them outright.
- The new tests cover the previously missing cases, including illegal-name parquet plus list fields and nested array/map decoding.
- I did not find a new blocking issue in the current fallback logic.
Tests / CI
- The new regression tests are materially better than the previous round.
- The current GitHub
Buildis green.
Conclusion: can merge
- Blocking items
- None from my side on the latest head.
- Suggested non-blocking follow-up
- None more important than preserving the current regression coverage.
Seatunnel log