Monday, February 6, 2017 11:37 PM
- About: Interactive Python
- Category: Programming Tools
- URL: http://www.python.org
|module (bold = default)||build||dependencies|
Starting a jupyter notebook on a single compute node inside of slurm
Securing your notebook
Before starting a jupyter notebook, it is a good idea to secure your jupyter notebook with a password as well as creating a certificate to encrypt your password. This is a one time procedure.
The easy way
There is a script to take care of this for you
This will generate a config file for jupyter, generate a self-signed certificate for the https connection, and prompt you to enter a password to secure your notebook server.
The manual way
If you prefer to do things manually, you can follow these steps instead:
- First, generate a config file for jupyter using
module load jupyter jupyter notebook --generate-config
- Then navigate to your jupyter config directory and generate a self-signed certificate
cd ~/.jupyter openssl req -x509 -nodes -newkey rsa:1024 -keyout key.pem -out cert.pem
- You can then create a password by using
python -c 'from notebook.auth import passwd; print(passwd());'
- This will print out a hashed version of your password which you will need to copy and paste in your jupyter config file. Using your favorite text editor, open the config file (where YourNetID should be replaced by your actual netid)
- and uncomment and modify the following three lines where YourNetID is your netid, and the password within the quotes is the hashed version of your password generated above.
c.NotebookApp.certfile = u'/home/YourNetID/.jupyter/cert.pem' c.NotebookApp.keyfile = u'/home/YourNetID/.jupyter/key.pem' c.NotebookApp.password = u'sha1:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
Running your notebook server
The easy way
srun -p debug -n 1 -t 60 --mem=2gb launch_jupyter
This will print out an ssh command which you can copy and then paste into a new terminal on your computer. You then just need to open a browser on your local computer and go to https://localhost:8888
The manual way
If you want to do things the manual way, you can start a jupyter notebook on a compute node by first getting an interactive session
interactive -p debug -n 1 -t 60 --mem=2gb
And then loading the jupyter module and start the notebook server
module load jupyter jupyter notebook --ip=0.0.0.0 --port=8888
You can then open up firefox inside of X2go and navigate to the URL of your notebook server where COMPUTE_NODE should be replaced by your actual compute node
Starting a parallel compute engine connected to a ipython notebook
To run a ipython notebook that can connect to a parallel engine, you can use the following sample script
#!/bin/sh #SBATCH -p debug #SBATCH -J jupyter #SBATCH --ntasks-per-node=22 #SBATCH -N 2 #SBATCH --exclusive #SBATCH --time=1:00:00 #SBATCH --mem-per-cpu=1g module load jupyter ipcontroller --ip='*' --port=8081 --location=`hostname` & sleep 10 srun ipengine --ip=`hostname` --port=8081 & sleep 25 jupyter notebook --ip=`hostname`.bluehive.circ.private --port=8080
Then you can connect to the python notebook server as before and execute parallel code.
from IPython.parallel import Client c=Client() c.ids
Running ipython notebooks in batch mode
If you want to execute an ipython notebook in batch mode, you can use runipy
runipy input.ipynb output.ipynb
instead of spawning the ipython notebook server.
And if you want to execute a python notebook and update the output in place, you can use
runipy -o MyNotebook.ipynb
also see Using Software.