jupyter

module (bold = default) build dependencies
jupyter/b1 python/2.7.10/b1

Starting a jupyter notebook on a single compute node inside of slurm

Securing your notebook

Before starting a jupyter notebook, it is a good idea to secure your jupyter notebook with a password as well as creating a certificate to encrypt your password. This is a one time procedure.

The easy way

There is a script to take care of this for you

  init_jupyter.sh

This will generate a config file for jupyter, generate a self-signed certificate for the https connection, and prompt you to enter a password to secure your notebook server.

The manual way

If you prefer to do things manually, you can follow these steps instead:

  • First, generate a config file for jupyter using
module load jupyter
jupyter notebook --generate-config
  • Then navigate to your jupyter config directory and generate a self-signed certificate
cd ~/.jupyter
openssl req -x509 -nodes -newkey rsa:1024 -keyout key.pem -out cert.pem 
  • You can then create a password by using
python -c 'from notebook.auth import passwd; print(passwd());'
  • This will print out a hashed version of your password which you will need to copy and paste in your jupyter config file. Using your favorite text editor, open the config file (where YourNetID should be replaced by your actual netid)
/home/YourNetID/.jupyter/jupyter_notebook_config.py
  • and uncomment and modify the following three lines where YourNetID is your netid, and the password within the quotes is the hashed version of your password generated above.
    c.NotebookApp.certfile = u'/home/YourNetID/.jupyter/cert.pem'
    c.NotebookApp.keyfile = u'/home/YourNetID/.jupyter/key.pem'
    c.NotebookApp.password = u'sha1:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
    

Running your notebook server

The easy way

srun -p debug -n 1 -t 60 --mem=2gb launch_jupyter

This will print out an ssh command which you can copy and then paste into a new terminal on your computer. You then just need to open a browser on your local computer and go to https://localhost:8888

The manual way

If you want to do things the manual way, you can start a jupyter notebook on a compute node by first getting an interactive session

interactive -p debug -n 1 -t 60 --mem=2gb

And then loading the jupyter module and start the notebook server

module load jupyter
jupyter notebook --ip=0.0.0.0 --port=8888

You can then open up firefox inside of X2go and navigate to the URL of your notebook server where COMPUTE_NODE should be replaced by your actual compute node

https://COMPUTE_NODE:8888

Or you can tunnel the connection back to your workstation following the OS-specific instructions for Mac, Windows, or Linux

Starting a parallel compute engine connected to a ipython notebook

To run a ipython notebook that can connect to a parallel engine, you can use the following sample script

#!/bin/sh
#SBATCH -p debug
#SBATCH -J jupyter
#SBATCH --ntasks-per-node=22
#SBATCH -N 2
#SBATCH --exclusive
#SBATCH --time=1:00:00
#SBATCH --mem-per-cpu=1g
module load jupyter
ipcontroller --ip='*' --port=8081 --location=`hostname` &
sleep 10
srun ipengine --ip=`hostname` --port=8081 &
sleep 25
jupyter notebook --ip=`hostname`.bluehive.circ.private --port=8080

Then you can connect to the python notebook server as before and execute parallel code.

from IPython.parallel import Client
c=Client()
c.ids

Running ipython notebooks in batch mode

If you want to execute an ipython notebook in batch mode, you can use runipy

runipy input.ipynb output.ipynb

instead of spawning the ipython notebook server.

And if you want to execute a python notebook and update the output in place, you can use

runipy -o MyNotebook.ipynb

also see Using Software.