Skip to content

[Fix][Connector-V2] Fix parquet read failure when column name contains Avro-illegal characters - #10960

Merged
davidzollo merged 5 commits into
apache:devfrom
yuluo-yx:0524-yuluo/fix
Jun 1, 2026
Merged

[Fix][Connector-V2] Fix parquet read failure when column name contains Avro-illegal characters#10960
davidzollo merged 5 commits into
apache:devfrom
yuluo-yx:0524-yuluo/fix

Conversation

@yuluo-yx

Copy link
Copy Markdown
Member

Seatunnel log

message:pod status failed, details: .run(Thread.java:750)
Caused by: org.apache.seatunnel.shade.connector.file.org.apache.avro.SchemaParseException: Illegal character in: job_blue-collar
	at org.apache.seatunnel.shade.connector.file.org.apache.avro.Schema.validateName(Schema.java:1566)
	at org.apache.seatunnel.shade.connector.file.org.apache.avro.Schema.access$400(Schema.java:91)
	at org.apache.seatunnel.shade.connector.file.org.apache.avro.Schema$Field.<init>(Schema.java:546)
	at org.apache.seatunnel.shade.connector.file.org.apache.avro.Schema$Field.<init>(Schema.java:585)
	at org.apache.seatunnel.shade.connector.file.org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:295)
	at org.apache.seatunnel.shade.connector.file.org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:279)
	at org.apache.seatunnel.shade.connector.file.org.apache.parquet.avro.AvroReadSupport.prepareForRead(AvroReadSupport.java:134)
	at org.apache.seatunnel.shade.connector.file.org.apache.parquet.hadoop.InternalParquetRecordReader.initialize(InternalParquetRecordReader.java:190)
	at org.apache.seatunnel.shade.connector.file.org.apache.parquet.hadoop.ParquetReader.initReader(ParquetReader.java:166)
	at org.apache.seatunnel.shade.connector.file.org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:135)
	at org.apache.seatunnel.connectors.seatunnel.file.source.reader.ParquetReadStrategy.read(ParquetReadStrategy.java:115)
	at org.apache.seatunnel.connectors.seatunnel.file.source.reader.MultipleTableFileSourceReader.pollNext(MultipleTableFileSourceReader.java:81)
	... 12 more

…s Avro-illegal characters

Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
@davidzollo

Copy link
Copy Markdown
Contributor

Please enable CI check following by the instruction https://github.com/apache/seatunnel/pull/10960/checks?check_run_id=77884719128

@davidzollo davidzollo added the First-time contributor First-time contributor label May 26, 2026
@DanielLeens

Copy link
Copy Markdown
Contributor

Thanks for working on this. I pulled the latest head locally and traced the real parquet source path instead of only checking the minimal repro from the PR body.

What this PR fixes

  • User pain: today the parquet file source fails immediately when a column name is valid in parquet but illegal in Avro, for example job_blue-collar.
  • Fix approach: try the existing AvroParquetReader path first, then fall back to a native parquet GroupReadSupport reader when the failure is specifically an illegal-Avro-field-name exception.
  • One-line summary: the fallback direction is reasonable, but the current implementation only fixes the flat-field case and still breaks on complex parquet types.

Runtime chain I checked

MultipleTableFileSourceReader.pollNext()
  -> readStrategy.read(split, output)
      -> ParquetReadStrategy.read(...)
          -> readWithAvro(...)
              -> AvroSchemaConverter.convertFields(...)
              -> illegal Avro field name -> SchemaParseException
          -> catch -> readWithNativeParquet(...)
              -> resolveGroupObject(...)
                  -> primitive field -> readPrimitiveGroupObject(...)
                  -> LIST/MAP logical group -> resolveGroupType(...)
                      -> current code throws convertToSeaTunnelTypeError

Findings

Issue 1: the new fallback still cannot read illegal-name parquet files that contain LIST / MAP logical types

  • Location: ParquetReadStrategy.java:255-272, ParquetReadStrategy.java:631-679
  • Problem: the schema inference path still maps parquet LIST / MAP into ArrayType / MapType, but the new native fallback read path rejects any logical-type group in resolveGroupType(...).
  • Risk: users move from the old Avro-name failure to a different runtime failure on the same main path as soon as the illegal-name file also contains arrays, maps, or similar nested logical groups.
  • Better fix:
    • Option A: make the native fallback support the same parquet type surface that the existing reader already advertises, at least for LIST, MAP, and nested row shapes.
    • Option B: if this PR only intends to support flat schemas for now, narrow the scope explicitly in code/tests/PR text and fail with a clear, deliberate limitation instead of implying the generic problem is solved.
  • Severity: High
  • Previously raised by others: No

Issue 2: the current Build is still red, and the failing jobs have not been proven unrelated yet

  • Location: GitHub Actions Build check, fork run 26455737240
  • Problem: the current metadata still shows failures in unit-test (8, ubuntu-latest), updated-modules-integration-test-part-5 (11, ubuntu-latest), and connector-file-sftp-it (8, ubuntu-latest).
  • Risk: even after the source fix is adjusted, the current head still does not have a mergeable green build signal.
  • Better fix: address issue 1 first, then rerun Build on the updated head and split the remaining failures by concrete job logs.
  • Severity: Medium
  • Previously raised by others: partially related to @davidzollo's CI reminder, but not yet technically analyzed there

Test coverage

  • The new regression test is stable, but it only covers a flat INT32 + STRING schema.
  • I do not see coverage yet for the real blind spot on the new fallback path: illegal-name parquet files that also contain LIST, MAP, or nested logical groups.

Merge conclusion

Conclusion: can merge after fixes

  1. Blocking items
  • Issue 1: please align the native fallback read capability with the parquet types the reader already claims to support, or explicitly narrow the supported scope.
  • Issue 2: please bring the current Build back to green on the updated head.
  1. Suggested non-blocking follow-up
  • A short comment on the fallback path would help future readers understand why there are now two parquet decoding paths and what the boundary is.

Overall, this PR fixes a real problem and the minimal repro is now covered, but the current fallback is still incomplete for common complex parquet schemas, so I would not merge this revision yet.

yuluo-yx added 3 commits May 29, 2026 23:08
…llback reader

The native Parquet Group reader fallback (used when column names contain
Avro-illegal characters) previously threw an exception when encountering
LIST or MAP logical types. This adds proper handling for:

- 3-level LIST encoding (standard: group → repeated wrapper → element)
- 2-level LIST encoding (legacy: group → repeated element directly)
- MAP fields (group → repeated key_value → {key, value})
- Element count and repetition index fixes for correct iteration

Also adds a test covering Avro-incompatible column names combined with
LIST fields to verify the fallback path works end-to-end.
Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>
Signed-off-by: yuluo-yx <yuluo08290126@gmail.com>

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. I re-reviewed the latest head from the real parquet fallback path.

What this PR solves

  • User pain: parquet files with column names that are valid in parquet but illegal in Avro fail immediately on the existing Avro reader path.
  • Fix approach: keep the Avro path first, then fall back to a native parquet-group reader when the failure is specifically the Avro-field-name parse problem.
  • One-line summary: the current head closes the complex-type gap I called out before, and I did not find a new blocker on the latest revision.

Runtime path I checked

ParquetReadStrategy.read(...)
  -> try AvroParquetReader path
  -> illegal Avro field name
  -> fallback to native Group API reader
      -> resolveGroupType(...)
          -> LIST -> readList(...)
          -> MAP -> readMap(...)
          -> nested ROW -> recursive resolveGroupObject(...)

Re-review result

  • The old blocker is fixed: the native fallback now handles LIST / MAP / nested row shapes instead of rejecting them outright.
  • The new tests cover the previously missing cases, including illegal-name parquet plus list fields and nested array/map decoding.
  • I did not find a new blocking issue in the current fallback logic.

Tests / CI

  • The new regression tests are materially better than the previous round.
  • The current GitHub Build is green.

Conclusion: can merge

  1. Blocking items
  • None from my side on the latest head.
  1. Suggested non-blocking follow-up
  • None more important than preserving the current regression coverage.

@davidzollo davidzollo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job
+1

@davidzollo
davidzollo merged commit 8ef362f into apache:dev Jun 1, 2026
6 checks passed
@yuluo-yx
yuluo-yx deleted the 0524-yuluo/fix branch June 1, 2026 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants