[Fix][Transform-V2] Align Scala compiler with Spark Scala version - #11117
Merged
davidzollo merged 2 commits intoJun 22, 2026
Conversation
DanielLeens
approved these changes
Jun 18, 2026
DanielLeens
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the contribution. I reviewed the full current diff from the real dynamic Scala compile path, not just the POM cleanup.
What this PR fixes
- User pain:
seatunnel-scala-compilerwas pinning its own Scala version to 2.13.11 while the repository root and Spark-facing baseline are still on 2.12.15, so the shaded compiler used byScalaClassParsercould drift away from the runtime Scala baseline. - Fix approach: remove the local Scala version override from
seatunnel-scala-compiler, let it inherit the rootscala.version, switchScalaClassParserto the constructor path that matches the current compiler version, and add a CI guard that prevents the module from silently diverging again. - One-line summary: the fix aligns the compiler module back to the repository-wide Scala baseline, and I did not find a source-level blocker in the latest head.
Runtime path I checked
dynamic compile transform
-> ScalaClassParse.parse(...)
-> ScalaClassParser.parseSourceCodeWithCache(...)
-> static IMain initialization
-> scalaInterpreter.compileString(sourceCode)
-> scalaInterpreter.classLoader().loadClass(className)
Key findings
- The normal path really does hit this change.
ScalaClassParseris the actual runtime bridge for dynamic Scala transform compilation. - Removing the local
scala.version=2.13.11override and inheriting the rootpom.xmlScala baseline is the right direction here. It eliminates a silent split between the shaded compiler module and the rest of the build. - The new
ScalaCompilerVersionCheckTestis useful because it protects the build contract from drifting again. - My only remaining gap is test shape, not source correctness: the new test guards the POM/dependency contract, but it does not directly execute a minimal
ScalaClassParsercompile path yet.
Other reviewer / maintainer input
- There were no prior non-Daniel reviews or comment threads on this PR when I reviewed it, so there was nothing to de-duplicate here.
Testing / stability
- The new test is structurally stable: pure file/XML inspection, no timing, ports, threads, or external services.
- I did not run local Maven in this batch; this is a source-level PR review only.
- GitHub
Buildwas still in progress when I reviewed.
Merge conclusion: can merge
- Blocking items
- None from my side at the source level.
- Suggested follow-up
- A small follow-up runtime smoke test that actually drives
ScalaClassParser.parseSourceCodeWithCache(...)on a trivial Scala snippet would make this even stronger, because the current new test only guards the version contract, not the end-to-end compile path.
Overall, this is a focused and defensible fix. The main runtime path is real, the version alignment is materially better than before, and I do not see a source-level reason to block merge on the current head.
davidzollo
approved these changes
Jun 22, 2026
davidzollo
left a comment
Contributor
There was a problem hiding this comment.
+1
LGTM
Thanks for your contribution!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes #11115.
seatunnel-scala-compilerwas pinned to Scala 2.13 while the root build and Spark translation modules use Scala 2.12. That lets Scala 2.13 compiler/reflect classes enter the transform package and can break Spark 2.12 runtime reflection with Scala signature version mismatch errors.Changes
seatunnel-scala-compilerso it inherits the root Scala 2.12 version.ScalaClassParserto use the Scala 2.12-compatibleIMain(Settings)constructor instead of the Scala 2.13-only reporter implementation package.seatunnel-scala-compilerfrom overriding the root Scala version again.Verification
./mvnw spotless:apply./mvnw -pl seatunnel-ci-tools -Dtest=ScalaCompilerVersionCheckTest -DforkCount=0 test./mvnw -pl seatunnel-transforms-v2 -am -DskipTests verify./mvnw -q -DskipTests verify