Skip to content

[Fix][Transform-V2] Align Scala compiler with Spark Scala version - #11117

Merged
davidzollo merged 2 commits into
apache:devfrom
zhangshenghang:fix/spark-scala-compiler-212
Jun 22, 2026
Merged

[Fix][Transform-V2] Align Scala compiler with Spark Scala version#11117
davidzollo merged 2 commits into
apache:devfrom
zhangshenghang:fix/spark-scala-compiler-212

Conversation

@zhangshenghang

Copy link
Copy Markdown
Member

Purpose

Fixes #11115.

seatunnel-scala-compiler was pinned to Scala 2.13 while the root build and Spark translation modules use Scala 2.12. That lets Scala 2.13 compiler/reflect classes enter the transform package and can break Spark 2.12 runtime reflection with Scala signature version mismatch errors.

Changes

  • Remove the local Scala 2.13 override from seatunnel-scala-compiler so it inherits the root Scala 2.12 version.
  • Update ScalaClassParser to use the Scala 2.12-compatible IMain(Settings) constructor instead of the Scala 2.13-only reporter implementation package.
  • Add a CI test to prevent seatunnel-scala-compiler from overriding the root Scala version again.

Verification

  • ./mvnw spotless:apply
  • ./mvnw -pl seatunnel-ci-tools -Dtest=ScalaCompilerVersionCheckTest -DforkCount=0 test
  • ./mvnw -pl seatunnel-transforms-v2 -am -DskipTests verify
  • ./mvnw -q -DskipTests verify

@github-actions github-actions Bot added core SeaTunnel core module Transform-v2 labels Jun 17, 2026
@github-actions github-actions Bot added the dependencies Pull requests that update a dependency file label Jun 18, 2026

@DanielLeens DanielLeens left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. I reviewed the full current diff from the real dynamic Scala compile path, not just the POM cleanup.

What this PR fixes

  • User pain: seatunnel-scala-compiler was pinning its own Scala version to 2.13.11 while the repository root and Spark-facing baseline are still on 2.12.15, so the shaded compiler used by ScalaClassParser could drift away from the runtime Scala baseline.
  • Fix approach: remove the local Scala version override from seatunnel-scala-compiler, let it inherit the root scala.version, switch ScalaClassParser to the constructor path that matches the current compiler version, and add a CI guard that prevents the module from silently diverging again.
  • One-line summary: the fix aligns the compiler module back to the repository-wide Scala baseline, and I did not find a source-level blocker in the latest head.

Runtime path I checked

dynamic compile transform
  -> ScalaClassParse.parse(...)
      -> ScalaClassParser.parseSourceCodeWithCache(...)
          -> static IMain initialization
          -> scalaInterpreter.compileString(sourceCode)
          -> scalaInterpreter.classLoader().loadClass(className)

Key findings

  1. The normal path really does hit this change. ScalaClassParser is the actual runtime bridge for dynamic Scala transform compilation.
  2. Removing the local scala.version=2.13.11 override and inheriting the root pom.xml Scala baseline is the right direction here. It eliminates a silent split between the shaded compiler module and the rest of the build.
  3. The new ScalaCompilerVersionCheckTest is useful because it protects the build contract from drifting again.
  4. My only remaining gap is test shape, not source correctness: the new test guards the POM/dependency contract, but it does not directly execute a minimal ScalaClassParser compile path yet.

Other reviewer / maintainer input

  • There were no prior non-Daniel reviews or comment threads on this PR when I reviewed it, so there was nothing to de-duplicate here.

Testing / stability

  • The new test is structurally stable: pure file/XML inspection, no timing, ports, threads, or external services.
  • I did not run local Maven in this batch; this is a source-level PR review only.
  • GitHub Build was still in progress when I reviewed.

Merge conclusion: can merge

  1. Blocking items
  • None from my side at the source level.
  1. Suggested follow-up
  • A small follow-up runtime smoke test that actually drives ScalaClassParser.parseSourceCodeWithCache(...) on a trivial Scala snippet would make this even stronger, because the current new test only guards the version contract, not the end-to-end compile path.

Overall, this is a focused and defensible fix. The main runtime path is real, the version alignment is materially better than before, and I do not see a source-level reason to block merge on the current head.

@davidzollo davidzollo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
LGTM
Thanks for your contribution!

@davidzollo
davidzollo merged commit ac89062 into apache:dev Jun 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core SeaTunnel core module dependencies Pull requests that update a dependency file reviewed Transform-v2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

seatunnel-transforms-v2 compiled with Scala 2.13 incompatible with Spark Scala 2.12

3 participants