Compatibility layer for `IPython` / `Jupyter` globals

[jtpio]

Problem

Many Python libraries used in the classic notebook rely on IPython and Jupyter being exposed on window.

This makes porting extensions to JupyterLab / RetroLab a bit difficult since this is not supported anymore.

Proposed Solution

Maybe RetroLab (and only RetroLab) should provide a compat layer and expose window.IPython and window.Jupyter. So things like Jupyter.notebook.save() executed in a JS cell still work in RetroLab by default.

An alternative for now is to use the jupyterapp main application on window with --expose-app-in-browser:

https://github.com/jupyterlab/retrolab/blob/6ce134fa17cc87d79fca822cbb761e4c7237b879/retrolab/app.py#L191-L195

Which can be used to execute arbitrary commands:

window-jupyterapp.mp4

This can be implemented as a new extension in the retrolab repo. The extension would expose the classic notebook API, but call JupyterLab APIs internally.

Additional context

This would help with use cases like ucbds-infra/otter-grader#448

Linking to the custom front-end extension documentation for reference: https://jupyter-notebook.readthedocs.io/en/stable/extending/frontend_extensions.html

---

cc @yuvipanda @davidwagner @fperez @tonyfast @psychemedia

[fperez]

In general, I'm very positive to the idea of using retro as a place to experiment with what level of compatibility shims and new APIs would work well. It's a smaller, safer playground, and we can always promote into new core JupyterLab APIs anything deemed generic and robust enough.

This seems like an excellent idea, thanks @jtpio.

[psychemedia]

I don't really understand how JupyterLab / RetroLab extensions work so I'm not sure what this does or does not make possible, what any consequences are / what alternative approaches would be required to achieve what functionality provided by such a shim using current Jupyterlab/Retrolab API?

If the proposal was to work towards a way of putting a harness around the majority of current classic extensions, or providing a clear set of rules for how you could start to migrate different patterns from classic nb extensions to JupyterLab, that would give me a path I could start trying to walk my extensions down.

I also wonder what this does to the JupyterLab extension ecosystem? Would folk have to add tags to show whether their extension works only in JupyterLab, only in RetroLab, or in both?

And I wonder if these API can be exposed, why were they not exposed from the start? Or if they were, why were they withdrawn? What architectural or security principles are JupyterLab puritans being asked to give up by exposing these APIs?! ;-)

[jtpio]

If the proposal was to work towards a way of putting a harness around the majority of current classic extensions, or providing a clear set of rules for how you could start to migrate different patterns from classic nb extensions to JupyterLab, that would give me a path I could start trying to walk my extensions down.

Yes this is mostly about buying us a bit more time, since we probably can't expect all extensions to migrate to JupyterLab for notebook 7. Also this issue is more about targeting simple use cases, such as otter-grader executing code like display(Javascript("Jupyter.notebook.save_checkpoint();")), rather than complex extensions like nbgrader or RISE (for those it would be better to do a proper port). We don't need to cover the whole Jupyter.notebook API if it's not necessary.

I also wonder what this does to the JupyterLab extension ecosystem? Would folk have to add tags to show whether their extension works only in JupyterLab, only in RetroLab, or in both?

We should still actively encourage developers to make JupyterLab extensions as this is the way forward. As discussed in other places, the tooling around authoring JupyterLab extensions can still be improved so it's more welcoming to non-developers. Folks should not start making new extensions relying on IPython / Jupyter. Actually if we add them to retro we could mark them as deprecated from the start. Because they will just be available to help with the transition.

And I wonder if these API can be exposed, why were they not exposed from the start? Or if they were, why were they withdrawn? What architectural or security principles are JupyterLab puritans being asked to give up by exposing these APIs?! ;-)

For several reasons I would say. Having Jupyter.notebook exposed on window is quite of a strong assumption specific to classic. In JupyterLab there might not even be a notebook displayed on the page.

[jtpio]

experiment with what level of compatibility shims and new APIs

We should probably scope this to provide some kind of compatibility with old extensions only. New API development should be done in the main JupyterLab repo, so we don't fragment the lab ecosystem with new extensions that only work in RetroLab.

[yuvipanda]

Right now, there's no 'simple' way for python code to perform actions on the currently open notebook - something that a number of projects want to do. I think we can and should expose a safe subset of actions to this, and a global object accessible from display seem a good way to go. I do think it should be thought of as separate from a compatibility shim though - I don't want to have to install 200 extensions that all just simply let a python (or R or...) piece of code ask to save the notebook.

So for the otter case, instead of compatibility shims, we could instead expose something else that can be called via display() - just something that can perform a list of actions we have decided are safe for python code to do - save and checkpoint comes to mind, as is getting the name of the notebook. We can make these function calls that can somehow perhaps work with 'current notebook' even in lab - since this code goes into python libraries, they should ideally work across frontends. This can totally be just an extension to be installed, although I think for max compatibiltiy these should ship by default.

If needed, we can also search GitHub for code that calls Jupyter / IPython stuff via display() and provide backwards compat shims for that, but that can be a separate activity.

[jtpio]

We can make these function calls that can somehow perhaps work with 'current notebook' even in lab - since this code goes into python libraries, they should ideally work across frontends. This can totally be just an extension to be installed, although I think for max compatibiltiy these should ship by default.

It can definitely be implemented as an extension. In fact some folks have already implemented some kind of proxy to do that in lab (jupyterlab/jupyterlab#5660).

This can totally be just an extension to be installed, although I think for max compatibiltiy these should ship by default.

By default in JupyterLab? Not sure. For example these wouldn't really apply to the code consoles.

[bollwyvl]

I'm still generally -1 on re-hoisting a compatibility layer by default. Requirejs, jquery, jquery ui is going to be a slippery slope, and would firmly want to not see it on lab core.

However, unifying commands as a display type, and increasing the robustness of command definition e.g. jupyterlab/lumino#58 sounds lovely.

A application/jupyter-command+json (sic) display type would be very handy, but i think would need some significant (user specified) guardrails. Basically, if i say, my kernel can save this notebook shouldn't mean it can delete all the files. So the first time it asks, you'd get an in-line display (no modals, thx):

Kernel Python 3 would like to:

> Save Notebook Foo.ipynb

[ALLOW] [DENY] [REMEMBER...]

Where the "remember" would let you update your settings. In a managed hub setting, this could be configured with overrides.json.

[jtpio]

I'm still generally -1 on re-hoisting a compatibility layer by default. Requirejs, jquery, jquery ui is going to be a slippery slope, and would firmly want to not see it on lab core.

Agree we should not support requirejs or jquery. I think the idea here was to limit the scope to what the IPython and Jupyter globals provide in the classic notebook.

A application/jupyter-command+json (sic) display type would be very handy,

And this could already be experimented with as an extension.

[blink1073]

Colab offers a limited API for interacting between JS and Python: https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb. It seems like an appropriate scope for what could be offered in "Notebook 7".

[gutow]

Colab offers a limited API for interacting between JS and Python: https://colab.research.google.com/notebooks/snippets/advanced_outputs.ipynb. It seems like an appropriate scope for what could be offered in "Notebook 7".

I am all in favor of at minimum providing the limited API that Collab exposes. However, Collab appears to be a version of Jupyter notebook not JLab (see Using Collab locally).

To add to the discussion of why I think we should have this:

1. There are use cases such as my Jupyter Physical Science Lab package (see especially JupyterPiDAQ), where menus and other user interface items need to be adjusted depending upon what packages are loaded or the state of the kernel. This requires two way communication between javascript and the kernel in a way that is not supported by the current widget framework. I find it hard to generalize what might need to be done. This suggests a general solution that handicaps the communications is likely to be ineffective for me and others.
2. Another thing I do is generate computed snippets on the fly dependent upon data available in the kernel. This also requires two way communications.

I have found the easiest way to solve the problems is to execute javascript using IPython.display.javascript(scriptstr) to start something in javascript ( I also usually wrap the script in an expiration time to avoid re-execution when opening a trusted notebook -- see JPSLUtils.OTJS). Then I run necessary python commands using promises to make sure the notebook state is appropriate for the next step. The simplest boiler plate for a generic python command in the old Jupyter notebook is:

function wait_for_python(cmdstr){
    return new Promise((resolve,reject) => {
        var callbacks = {
            iopub: {
                output: (data) => resolve(data.content.text.trim())
            },
            
            shell: {
                reply: (data) => resolve(data.content.status)
            }

        };
        Jupyter.notebook.kernel.execute(cmdstr, callbacks);
    });
}

It would be nice if something nearly this simple would work in JLab. I can use this to set python variables on the fly, extract data from the kernel, run a command after the notebook front end has reached an appropriate state, etc...