Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration with DAG framework Prefect #226

Open
tchaton opened this issue Jul 12, 2024 · 1 comment
Open

Integration with DAG framework Prefect #226

tchaton opened this issue Jul 12, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request help wanted Extra attention is needed

Comments

@tchaton
Copy link
Collaborator

tchaton commented Jul 12, 2024

🚀 Feature

Motivation

The library Prefect https://github.com/PrefectHQ/prefect is pretty great at building DAGs and orchestrating then.

In the future, LitData will enables to build a DAG of map, optimize, etc.. operators to perform large distributed processing.

The ideal goal would be the reverse engineer Prefect Orion Engine and monitoring and have an adapted version for the Lightning.AI platform.

Pitch

Alternatives

Additional context

@tchaton tchaton added enhancement New feature or request help wanted Extra attention is needed labels Jul 12, 2024
@tchaton tchaton changed the title Integration with Prefect Integration with DAG framework Prefect Jul 12, 2024
@deependujha deependujha self-assigned this Jul 12, 2024
@deependujha
Copy link
Collaborator

Orion engine is deprecated in Prefect - Release 2.13.0. So, I couldn't find much information.

And, I guess, we can already use litdata and prefect.

So, are we interested in Prefect workpools specific to lightning studio, create a deployment and run the workflows on lightning nodes (studio/jobs)?

Or, is it something else that we want?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants