Create isolated Jupyter ipython kernels with pyenv and virtualenv

Posted on

Everyone loves isolation. Makes our life easier and our systems much more robust. Isolating Jupyter notebooks makes no exception. Maybe you want to try some cutting edge scientific library, or more simply your latest project dependencies are not compatible with your current system setup.

Whatever is your situation, follow me in this simple tutorial on how to create an isolated python notebook kernel.

Introduction

If you are a day-to-day Jupyter user you probably know what kernels are. If you are not, a kernel is simply a language virtual machine running behind the scenes and connected to the Jupyter browser interface. Each time you create a new notebook you have to select the kernel from the top right drop down menu of the interface as shown here:

Screen Shot 2015-12-10 at 19.42.29

The goal of this blog post is to add a new kernel on a python environment that is different from the one that I have installed already. This is especially useful if you are trying out some project and you want the dependencies of the new project to be installed in a separate python local setup. We can do that in 3 “easy” steps 🙂

Step 1: Install pyenv, virtualenv, and pyenv-virtualenv

First we need a way to create different python environments. Each environment will have its own version and isolated package set. On paper it is far from easy, but thanks to Pyenv 1 and virtualenv2 the job turns out to be pretty simple.

To install it on MacOSx all you need to do is:

and add to your .bashrc the following:

Installing virtualenv is also fairly easy:

Finally, install pyenv-virtualenv 3:

and add to your .bashrc the following:

Step 2: Create an isolated python environment

Let’s assume now that I want to test my latest project in a Jupyter notebook running a Python kernel with Python 2.7.3. Since I have several projects running on this Python version I also want to have a dedicated Python 2.7.3 environment for my latest project.

First let’s see all the python versions available to pyenv:

and let’s install the one we wants:

Now it is time to create a dedicated environment for our project. Suppose I am working on a bleeding age implementation of k_means clustering algorithm (…). This is how I create my working environment:

and I now see the following:

Let’s switch our python environment to the one we created for our new project

and let’s install the basic scientific packages we need:

these packages will be local to our k_means python installation and will not affect our system python (for example).
To access this environment from Jupyter you need the python kernel too, so let’s install it:

Finally, let’s deactivate our environment.

Step 3: Create your isolated Jupyter python kernel

Now we have to connect our Jupyter to the isolated python enviroment we created in the previous two steps. I am assuming you have already jupyter installed in your system Python. If not, go ahead and install it here.

If you are in the right environment with jupyter installed you should see the following:

In order to install a new Jupyter kernel you need to check where Jupyter is reading its configuration files. To do that simply run the following:

Jupyter search for the kernels in the data directories in the order they are displayed. First, we need to find out where pyenv is storing our k_means environment, and we do it by executing the following:

Now we are ready to create our kernel. First let’s create the folder:

and let’s add the following kernel.json file:

If you now run jupyter notebook you will have the new kernel available!

Screen Shot 2015-12-10 at 19.25.10

Conclusions

In this blog post I presented how to create an isolated python kernel in your jupyter installation. This is not the only way to do it, so please share in the comments if you have better ways to achieve the same result!

If you enjoyed this blog post you can also follow me on twitter.

References

  1. Pyenv homepage
  2. Virtualenv homepage
  3. Pyenv-virtualenv homepage

9 thoughts on “Create isolated Jupyter ipython kernels with pyenv and virtualenv”

  1. This is great, but one can go further! By installing a pyenv hook, one can get pyenv to automatically set the environment variables used by Jupyter to determine the locations of its configuration and data directories.

    In my shell startup scripts, I define the following two “template” environment variables:


    export _JUPYTER_CONFIG_DIR='${PYENV_ROOT}/versions/$(pyenv version-name)/etc/jupyter’

    export _JUPYTER_DATA_DIR='${PYENV_ROOT}/versions/$(pyenv version-name)/share/jupyter’

    Next, I create the hook script as ~/.pyenv/pyenv.d/exec/jupyter-paths.bash. This script contains only two lines:


    export JUPYTER_CONFIG_DIR=    eval "echo ${_JUPYTER_CONFIG_DIR}"
    export JUPYTER_DATA_DIR=eval "echo ${_JUPYTER_DATA_DIR}"

    Now you can use Jupyter’s standard toolset for installing kernels, etc., and everything ought just to work.

    Reply
  2. Pingback: Create isolated Jupyter ipython kernels with pyenv and virtualenv – Alfredo Motta

  3. If I try to use matplotlib I get the following error messages.

    import matplotlib.pyplot as plt

    Do you have the same problem?

    RuntimeError Traceback (most recent call last)
    in ()
    1 #Laden der Module
    —-> 2 import matplotlib.pyplot as plt
    3 import os,sys
    4
    5 #Background Color Figure

    /Users/chrischi/.pyenv/versions/geoEnv/lib/python2.7/site-packages/matplotlib/pyplot.py in ()
    112
    113 from matplotlib.backends import pylab_setup
    –> 114 _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
    115
    116 _IP_REGISTERED = None

    /Users/chrischi/.pyenv/versions/geoEnv/lib/python2.7/site-packages/matplotlib/backends/init.pyc in pylab_setup()
    30 # imports. 0 means only perform absolute imports.
    31 backend_mod = import(backend_name,
    —> 32 globals(),locals(),[backend_name],0)
    33
    34 # Things we pull in from all backends

    /Users/chrischi/.pyenv/versions/geoEnv/lib/python2.7/site-packages/matplotlib/backends/backend_macosx.py in ()
    22
    23 import matplotlib
    —> 24 from matplotlib.backends import _macosx
    25
    26

    RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are Working with Matplotlib in a virtual enviroment see ‘Working with Matplotlib in Virtual environments’ in the Matplotlib FAQ

    Reply

    2. I have had this problem before, in the end I gave up and didn’t try to use a virtual env…not ideal I know.

      Reply

  4. hi,

    i have installed hydrogen in atom. i wanted to use the python kernel installed by anaconda. the kernel and other files are installed under /opt/anaconda/bin.

    how to let hydrogen know of the kernel and the jupyter notebook run from the anacondoa env.

    Reply

  5. Hi,

    I’ve had the very same problem. The solution I found was to install a pyenv plugin called “pyenv-register”.

    You can get it here: https://github.com/doloopwhile/pyenv-register

    Then you can register the anaconda python with pyenv:
    pyenv register /opt/anaconda/bin/python3.6

    And set it as your default system python:
    pyenv global system-3.6.3

    Finally register the already installed kernel for hydrogen to find it:
    python -m ipykernel install –user

    Reply
  6. Pingback: Virtual environment – Howl's Moving Castle

Leave a Reply

Your email address will not be published. Required fields are marked *

Answer the question * Time limit is exhausted. Please reload CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.