Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Colab integration #133

Open
lorenzoh opened this issue Jul 1, 2021 · 15 comments
Open

Colab integration #133

lorenzoh opened this issue Jul 1, 2021 · 15 comments
Labels
documentation Improvements or additions to documentation help wanted Contributions welcome!

Comments

@lorenzoh
Copy link
Member

lorenzoh commented Jul 1, 2021

Ideally, all the documentation pages based off of Jupyter notebooks should have a button to launch them in Colab.

For example, fastai's documentation has such a button:
image

First step would be a tutorial on how to set up Colab with Julia and install the dependencies.

See this Discourse post that shows how to get Julia running on Colab.

@lorenzoh lorenzoh added documentation Improvements or additions to documentation help wanted Contributions welcome! labels Jul 1, 2021
@INF800
Copy link

INF800 commented Aug 9, 2021

Hi, I want to take this up.

@lorenzoh
Copy link
Member Author

Cool! Here's what I think needs to be done.

  • Conversion of regular Julia .ipynb notebooks to Colab notebooks
    This consists of

    • Adding cells with instructions and shell script that installs Julia, packages and the Julia kernel (see the Colab template for an example)
    • Updating the metadata of notebook with kernel name and accelerator info so that GPU is used by default

    This functionality may be best created as a separate repository (Colab.jl?) that does this conversion; that way the functionality will be easier to reuse but also maintain when the specifics of running Julia on Colab change

  • Integration with the documentation system (Pollen.jl)

    During documentation build, the following needs to happen for every source notebook file

    • create a Colab version of the notebook using the above conversion functionality
    • generate a Colab link that will auto-open the Colab version that is built to the GitHub Pages branch during doc build
    • insert the link with a badge into the documentation html output

    Since Pollen.jl is still undocumented, I could take care of this step once the first is done

Let me know if you need any other pointers 👍

@Chandu-4444
Copy link
Contributor

Hey, I'm interested in doing this. Can I try to do this?

@lorenzoh
Copy link
Member Author

Sure! Let me know if you need any other pointers aside from the above

@Chandu-4444
Copy link
Contributor

I could come up with this. But is there any way to speed up the initialization process (The time taken to run the first cell). Running this for every time a notebook that has been opened is kind of inefficient? Or is there a workaround?

@lorenzoh
Copy link
Member Author

One though I've had is using PackageCompiler.jl to create a sysimage once that is then saved somewhere and loaded into the Colab session which would allow reducing installation time significantly and almost completely removing compilation overhead.

However, there are a lot of problems that need to be dealt with, and I'm not sure it's worth the effort:

  • the sysimage needs to be built on Colab since it needs to be compiled for that hardware and software
  • it needs to be manually rebuilt on every update since Colab doesn't allow automation
  • the sysimage file needs to be stored somewhere and served (these are easily > 100MB)
  • I'm not sure if the randomly different GPU hardware you get on Colab would mess with the sysimage.

@Chandu-4444
Copy link
Contributor

Ok then. I'll add the Launch Colab button to all document pages that are based off a notebook.

Cool! Here's what I think needs to be done.

  • Conversion of regular Julia .ipynb notebooks to Colab notebooks
    This consists of

    • Adding cells with instructions and shell script that installs Julia, packages and the Julia kernel (see the Colab template for an example)
    • Updating the metadata of notebook with kernel name and accelerator info so that GPU is used by default

    This functionality may be best created as a separate repository (Colab.jl?) that does this conversion; that way the functionality will be easier to reuse but also maintain when the specifics of running Julia on Colab change

  • Integration with the documentation system (Pollen.jl)
    During documentation build, the following needs to happen for every source notebook file

    • create a Colab version of the notebook using the above conversion functionality
    • generate a Colab link that will auto-open the Colab version that is built to the GitHub Pages branch during doc build
    • insert the link with a badge into the documentation html output

    Since Pollen.jl is still undocumented, I could take care of this step once the first is done

Let me know if you need any other pointers +1

Sure! I started working on the first one.
Is it possible to update the metadata of a notebook file and add the extra Julia installation to the start of the notebook as soon as the user clicks on Launch Colab?
Or is it better to manually add this information to all notebook files and store them in a separate repo (Colab.jl) and we can link the corresponding notebook from Colab.jl as soon as a user clicks Launch Colab in FastAI documentation page.

@lorenzoh
Copy link
Member Author

First part is writing a program that converts a regular Julia notebook to one that can be run on Colab, i.e. has the setup cells and metadata to select the GPU. Should be independent of the buttons.

This code should probably live in a separate repo, but the notebook files themselves will be built during the documentation building step in FastAI.jl's CI and stored on the gh-pages branch.

@FluxML FluxML deleted a comment from Chandu-4444 Nov 29, 2021
@Chandu-4444
Copy link
Contributor

First part is writing a program that converts a regular Julia notebook to one that can be run on Colab, i.e. has the setup cells and metadata to select the GPU. Should be independent of the buttons.

I'll first try writing that conversion script. Thanks 👍

@Chandu-4444
Copy link
Contributor

First part is writing a program that converts a regular Julia notebook to one that can be run on Colab, i.e. has the setup cells and metadata to select the GPU. Should be independent of the buttons.

I'll first try writing that conversion script. Thanks +1

Hey, I made a repo that contains the code for converting a normal ipynb notebook to the one that can run on Colab.
https://github.com/Chandu-4444/Colab.jl
This is a sample one, there are few improvements to be made (Specifying the output file location etc).

Please look at the repo and let me know the next step.

@Chandu-4444
Copy link
Contributor

This code should probably live in a separate repo, but the notebook files themselves will be built during the documentation building step in FastAI.jl's CI and stored on the gh-pages branch.

I have an idea to do this. Is it ok if the configuration cells were hidden from getting displayed on the docs website and reveal when people click view on colab on colab page?

@lorenzoh
Copy link
Member Author

lorenzoh commented Mar 1, 2022

Sorry for letting this fall through the cracks!

Hey, I made a repo that contains the code for converting a normal ipynb notebook to the one that can run on Colab.
https://github.com/Chandu-4444/Colab.jl
This is a sample one, there are few improvements to be made (Specifying the output file location etc).

Thanks for starting on this

I have an idea to do this. Is it ok if the configuration cells were hidden from getting displayed on the docs website and reveal when people click view on colab on colab page?

They should be added to the notebook, and the documentation system will take care of hiding the setup cells as well as the "Open in Colab" button

@Chandu-4444
Copy link
Contributor

Chandu-4444 commented Mar 8, 2022

They should be added to the notebook, and the documentation system will take care of hiding the setup cells as well as the "Open in Colab" button

So, shall I replace all the current notebooks with the notebooks containing the config cells that can run on Colab?

@Chandu-4444
Copy link
Contributor

Colab.jl is now a registered Julia package. I think this can now be integrated into docs?

@lorenzoh
Copy link
Member Author

Hey, sorry for letting this sit again, integrating it into the docs is bottlenecked on my time making an integration with Pollen.jl right now. In any case, thanks for getting this running! I'll open an issue on the repo to propose some changes when I get around to it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation help wanted Contributions welcome!
Projects
None yet
Development

No branches or pull requests

3 participants