jupyter

  • About: Python programming language
  • Category: Programming Tools
  • URL: http://python.org
module (bold = default) build dependencies
jupyter/b1 gcc 4.8.5 texlive/2016

Starting a jupyter notebook on a single compute node inside of slurm

Before starting a jupyter notebook, it is a good idea to secure your jupyter notebook with a password as well as creating a certificate to encrypt your password

First, generate a config file for jupyter using

module load jupyter
jupyter notebook --generate-config

Then navigate to your jupyter config directory and generate a self-signed certificate

cd ~/.jupyter
openssl req -x509 -nodes -newkey rsa:1024 -keyout key.pem -out cert.pem 

You can then create a password by using

python -c 'from notebook.auth import passwd; print(passwd());'

This will print out a hashed version of your password which you will need to copy and paste in your jupyter config file. Using your favorite text editor, open the config file (where YourNetID should be replaced by your actual netid)

/home/YourNetID/.jupyter/jupyter_notebook_config.py

and uncomment and modify the following three lines where YourNetID is your netid, and the password within the quotes is the hashed version of your password generated above.

c.NotebookApp.certfile = u'/home/YourNetID/.jupyter/cert.pem'
c.NotebookApp.keyfile = u'/home/YourNetID/.jupyter/key.pem'
c.NotebookApp.password = u'sha1:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

You can now start a jupyter notebook on a compute node using

module load jupyter
srun -p debug -n 1 -t 60 --mem=2gb sh -c 'jupyter notebook --ip=0.0.0.0 --port=8888'

This should start a job on a compute node (say bhc0001) and jupyter will print out the URL of your notebook.

  The Jupyter Notebook is running at: https://bhc0001.bluehive.circ.private:8888/

You can then open up firefox inside of X2go and navigate to that URL

https://bhc0001.bluehive.circ.private:8888

Or you can tunnel the connection back to your workstation following the OS-specific instructions for Mac, Windows, or Linux

Starting a parallel compute engine connected to a ipython notebook

To run a ipython notebook that can connect to a parallel engine, you can use the following sample script

#!/bin/sh
#SBATCH -p debug
#SBATCH -J jupyter
#SBATCH --ntasks-per-node=22
#SBATCH -N 2
#SBATCH --exclusive
#SBATCH --time=1:00:00
#SBATCH --mem-per-cpu=1g
module load jupyter
ipcontroller --ip='*' --port=8081 --location=`hostname` &
sleep 10
srun ipengine --ip=`hostname` --port=8081 &
sleep 25
jupyter notebook --ip=`hostname`.bluehive.circ.private --port=8080

Then you can connect to the python notebook server as before and execute parallel code.

from IPython.parallel import Client
c=Client()
c.ids

Running ipython notebooks in batch mode

If you want to execute an ipython notebook in batch mode, you can use runipy

runipy input.ipynb output.ipynb

instead of spawning the ipython notebook server.

And if you want to execute a python notebook and update the output in place, you can use

runipy -o MyNotebook.ipynb

also see Using Software.