Thursday, August 24, 2017 8:50 PM
- About: Python programming language
- Category: Programming Tools
- URL: http://python.org
|module (bold = default)||build||dependencies|
Starting a jupyter notebook on a single compute node inside of slurm
Before starting a jupyter notebook, it is a good idea to secure your jupyter notebook with a password as well as creating a certificate to encrypt your password
First, generate a config file for jupyter using
module load jupyter jupyter notebook --generate-config
Then navigate to your jupyter config directory and generate a self-signed certificate
cd ~/.jupyter openssl req -x509 -nodes -newkey rsa:1024 -keyout key.pem -out cert.pem
You can then create a password by using
python -c 'from notebook.auth import passwd; print(passwd());'
This will print out a hashed version of your password which you will need to copy and paste in your jupyter config file. Using your favorite text editor, open the config file (where YourNetID should be replaced by your actual netid)
and uncomment and modify the following three lines where YourNetID is your netid, and the password within the quotes is the hashed version of your password generated above.
c.NotebookApp.certfile = u'/home/YourNetID/.jupyter/cert.pem' c.NotebookApp.keyfile = u'/home/YourNetID/.jupyter/key.pem' c.NotebookApp.password = u'sha1:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
You can now start a jupyter notebook on a compute node using
module load jupyter srun -p debug -n 1 -t 60 --mem=2gb sh -c 'jupyter notebook --ip=0.0.0.0 --port=8888'
This should start a job on a compute node (say bhc0001) and jupyter will print out the URL of your notebook.
The Jupyter Notebook is running at: https://bhc0001.bluehive.circ.private:8888/
You can then open up firefox inside of X2go and navigate to that URL
Starting a parallel compute engine connected to a ipython notebook
To run a ipython notebook that can connect to a parallel engine, you can use the following sample script
#!/bin/sh #SBATCH -p debug #SBATCH -J jupyter #SBATCH --ntasks-per-node=22 #SBATCH -N 2 #SBATCH --exclusive #SBATCH --time=1:00:00 #SBATCH --mem-per-cpu=1g module load jupyter ipcontroller --ip='*' --port=8081 --location=`hostname` & sleep 10 srun ipengine --ip=`hostname` --port=8081 & sleep 25 jupyter notebook --ip=`hostname`.bluehive.circ.private --port=8080
Then you can connect to the python notebook server as before and execute parallel code.
from IPython.parallel import Client c=Client() c.ids
Running ipython notebooks in batch mode
If you want to execute an ipython notebook in batch mode, you can use runipy
runipy input.ipynb output.ipynb
instead of spawning the ipython notebook server.
And if you want to execute a python notebook and update the output in place, you can use
runipy -o MyNotebook.ipynb
also see Using Software.