New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set up Dask cluster for hazard indicator calculation #5

Open

joemoorhouse opened this issue Feb 3, 2023 · 1 comment

Assignees

Collaborator

joemoorhouse commented Feb 3, 2023 •

edited

Loading

Tasks to create hazard indicators typically involve

sourcing multi-dimensional array data from S3
transforming
pushing back to S3
Most of the work will be in manipulating (chunked) xarrays. Having multiple pods in a cluster should help in parallelizing both calculation and data sourcing.
The issue is to set up a Dask cluster to be used by hazard jobs.

Collaborator Author

joemoorhouse commented Feb 7, 2023 •

edited

Loading

Talking to @erikerlandson, using Ray clusters might be the best approach.

joemoorhouse assigned joemoorhouse and erikerlandson

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment