Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 3489 - Upgrade to Java 17 #3499

Open
wants to merge 22 commits into
base: 2000-upgrade-emr
Choose a base branch
from

Conversation

patchwork01
Copy link
Collaborator

Make sure you have checked all steps below.

Issue

  • My PR addresses the following issues and references them in the PR title. For example, "Issue 1234 - My Sleeper
    PR"

Tests

  • My PR adds the following tests OR does not need testing for this extremely good reason:
    • Covered by existing tests
    • Will run quick system test suite in AWS

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it, or I have linked to a
    separate issue for that below.

@patchwork01 patchwork01 marked this pull request as ready for review October 16, 2024 14:20
@patchwork01 patchwork01 added the pr-stacked-middle A stacked pull request that must wait until the target PR is merged, and is also a base for stacking label Oct 16, 2024
Copy link
Collaborator

@rtjd6554 rtjd6554 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passed awaiting testing completion

@patchwork01 patchwork01 assigned patchwork01 and unassigned rtjd6554 Oct 17, 2024
@patchwork01
Copy link
Collaborator Author

Bulk import on EKS is failing on this branch currently. It passes on EMR, on both serverless and EC2.

@patchwork01
Copy link
Collaborator Author

patchwork01 commented Oct 21, 2024

It looks like the problem is the Spark Docker image has the wrong version of Hadoop in it. The Spark Docker image includes Hadoop 3.3.4, but EMR and Sleeper use Hadoop 3.3.6. The Spark Docker image doesn't include the hadoop-aws module. We're bringing that in in Sleeper and the versions conflict. It looks as though the class IOStatisticsBinding has been moved between modules and it can't find it.

If we want to fix it we'll probably need a new Docker image.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
one-review-required pr-stacked-middle A stacked pull request that must wait until the target PR is merged, and is also a base for stacking
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants