Jupyter
From FarmShare
(move renewal section to later) |
(clarity - added ssh instructions to renewal stage) |
||
Line 120: | Line 120: | ||
The virtual terminal created by the above instructions will persist on the FarmShare server for one week, after which its Kerberos tickets will expire and the Jupyter notebook will no longer function. To restore the notebook after this time, SSH into the '''same''' FarmShare system used to create the virtual terminal. | The virtual terminal created by the above instructions will persist on the FarmShare server for one week, after which its Kerberos tickets will expire and the Jupyter notebook will no longer function. To restore the notebook after this time, SSH into the '''same''' FarmShare system used to create the virtual terminal. | ||
+ | |||
+ | $ '''ssh <font color=blue>jane</font>@corn<font color=blue>99</font>.stanford.edu''' | ||
+ | |||
+ | substituting sunetid for <font color=blue>'''jane'''</font> and the appropriate FarmShare server for '''corn<font color=blue>99</font>'''. | ||
corn99:~$ '''tmux attach -d''' | corn99:~$ '''tmux attach -d''' |
Revision as of 14:05, 5 January 2017
Contents |
Introduction
Project Jupyter evolved out of the IPython project (specifically the IPython notebook) with the goal to provide an interactive, web-browser driven, language-independent programming environment. Jupyter notebooks can be deployed on the FarmShare servers to enable an accessible, powerful, and persistent computational platform.
Features
At the end of this guide, the resulting Jupyter notebook will support:
- Python, MATLAB, R, SAS, and Julia programming languages
- an encrypted, token protected, and web-browser enabled programming environment
- indefinite persistence of the Jupyter notebook server environment with simple weekly renewals (the maximum duration of Stanford Kerberos tickets)
- file/data storage on the Stanford AFS servers (5GB user quota, Stanford-wide, automatic backups)
- shared file/data storage to Class Disk AFS Space
- easy deployment on any of the Stanford FarmShare systems
Overview
The guide consists of three sections: installation, Jupyter notebook server management, and client connection. In typical usage (i.e., once installation is complete and the Jupyter notebook server is running), only the client connection steps are necessary at every use.
Installation
Jupyter setup is best performed via the Linux console. This guide will be mostly step-by-step, but general familiarity with Linux is helpful.
Commands to be typed in are in bold and text to substitute is in blue.
Useful tutorials are available as a free book, online tutorial, or video series.
In a terminal, ssh into a FarmShare computer.
ssh jane@corn.stanford.edu
substituting sunetid for jane.
SSH provides an encrypted remote shell into another system and is the primary way that Jupyter will be installed and accessed.
A terminal program and SSH client is shipped with Mac and Linux systems. Windows does not come with an SSH client, but putty is a a free and lightweight SSH client for Windows.
Bind to Jupyter virtual environment
corn99:~> bash jane@corn99:~$ source /afs/ir.stanford.edu/group/bil/env/jupyter/bin/activate (jupyter)jane@corn99:~$
These two commands switch the shell to bash and then update the environment to use a pre-built Jupyter installation. The prompt should have the (jupyter)
prefix if the environment change is successful.
The pip packages installed for the Jupyter environment are stored in a requirements.txt
file at /afs/ir.stanford.edu/group/bil/env/jupyter/requirements.txt
. Additional packages can be installed upon request.
Setup encryption and default configuration
(jupyter)jane@corn99:~$ /afs/ir.stanford.edu/group/bil/env/scripts/jupyter_config_wrapper
This command calls a script that will create encryption keys for the Jupyter notebook and configure the jupyter config file to use these newly created keys. Note, this will modify existing Jupyter notebook config files (if they exist), but will not overwrite any parameters that have changed from their default values.
Installation is complete after this step is successful. Log out of FarmShare.
Jupyter notebook server
Jupyter notebook will be run within a persistent virtual terminal so it can stay running even after the user has logged out of FarmShare.
Create virtual terminal
This only needs to be performed once per FarmShare system (and after every server reboot).
SSH into a FarmShare system (e.g. corn). Make a note of exactly which corn server entered (e.g., corn14, corn22, etc).
corn99:~$ pagsh sh-4.3$ kinit; aklog Password for jane@stanford.edu: sh-4.3$ tmux (new blank terminal appears) corn99:~$ bash jane@corn99:~$ /afs/ir.stanford.edu/group/bil/env/scripts/keep_kerberos_afs Run: export KRB5CCNAME=FILE:/tmp/.krb5_jane.tgt jane@corn99:~$ export KRB5CCNAME=FILE:/tmp/.krb5_jane.tgt jane@corn99:~$ source /afs/ir.stanford.edu/group/bil/env/jupyter/bin/activate (jupyter)jane@corn99:~$ /afs/ir.stanford.edu/group/bil/env/scripts/jupyter_start
Substitute sunetid for jane in the export
line (as instructed by the keep_kerberos_afs
command). If done correctly, the output of this command will be
IMPORTANT: This Jupyter notebook is listening on TCP port 9876 [I 17:23:59.661 NotebookApp] Loading IPython parallel extension [I 17:23:59.668 NotebookApp] Serving notebooks from local directory: /afs/ir.stanford.edu/users/j/a/jane [I 17:23:59.668 NotebookApp] 0 active kernels [I 17:23:59.668 NotebookApp] The Jupyter Notebook is running at: https://localhost:9876/?token=ba682763f27d8e2d59862badef28b0eaecb552529933176e [I 17:23:59.668 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 17:23:59.671 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: https://localhost:9876/?token=ba682763f27d8e2d59862badef28b0eaecb552529933176e
The notebook will not work until a client connection is established. Copy the https://localhos...
URL and save it for subsequent use. Also note the TCP port that the Jupyter notebook is listening on. Use Ctrl+b
then d
to detach from tmux. Logout of FarmShare.
Ctrl+b then d (detaches tmux) sh-4.3$ exit corn99:~> logout
Note: the jupyter_start
script accepts an optional port argument, specifying which port to connect to (e.g., jupyter_start 9876
). This is useful in case a randomized port is not desired.
Client connection
Create SSH tunnel
Once the Jupyter notebook server is running, it is ready to accept client connections. For security, it only accepts connections from localhost (i.e., connections originating from the FarmShare system itself). Local connections can be created remotely through SSH tunnels. SSH into the same FarmShare system that the Jupyter notebook server virtual terminal was created on (e.g., corn14, corn22, etc) by using the server's FQDN (e.g., corn14.stanford.edu) with a local port tunnel to the TCP port that the server is listening on.
In Mac/Linux, setup the SSH tunnel in a terminal.
$ ssh jane@corn99.stanford.edu -L 9999:localhost:9999
substituting in the appropriate sunetid for jane
, the appropriate FarmShare server hostname for corn99
, and the appropriate TCP port for 9999
. Windows users can setup an SSH local tunnel using menu options in putty
Once the ssh tunnel has been established, paste the https://localhos...
URL provided from the jupyter_start
script into a web browser to connect to the notebook home page. If this URL is misplaced, use tmux attach -d
to bring up the virtual terminal and copy the URL again. Since the encryption keys were self-signed, the browser will warn about an insecure connection, but disregard these and continue/proceed past them.
Renewing virtual terminal
This only needs to be done a week after creating the virtual terminal when the kerberos credentials expire.
The virtual terminal created by the above instructions will persist on the FarmShare server for one week, after which its Kerberos tickets will expire and the Jupyter notebook will no longer function. To restore the notebook after this time, SSH into the same FarmShare system used to create the virtual terminal.
$ ssh jane@corn99.stanford.edu
substituting sunetid for jane and the appropriate FarmShare server for corn99.
corn99:~$ tmux attach -d Ctrl+b then c (creates a new window in tmux) jane@corn99:~$ kinit; aklog Password for jane@stanford.edu: Ctrl+b then 0 (returns tmux to the original window)
This will extend the terminal's credentials for another week. The same notebook URL can be used without interruption. Note, for subsequent renewals, use Ctrl+b then 1
to switch to window 1 instead of creating a new window. It is not necessary to create a new tmux window every time.
Examples
Python 3
import numpy as np import matplotlib.pyplot as plt x = np.r_[ -np.pi : np.pi : 0.1 ] plt.plot(x, np.sin(x), linewidth=3) plt.plot(x, np.cos(x), linewidth=3, color='r') plt.show()
MATLAB
x = -pi : 0.1 : pi; hold on; box on; plot(x, sin(x), 'lineWidth', 2); plot(x, cos(x), 'lineWidth', 2, 'color', 'r');
R
x = seq(-pi, pi, 0.1) plot(x, sin(x), type='l', lwd=4, col='blue', ann=FALSE) lines(x, cos(x), type='l', lwd=4, col='red')
SAS
data curves; do x = -constant("pi") to constant("pi") by 0.1; y = sin(x); z = cos(x); output; end; run; symbol1 interpol=join color=blue width=5; symbol2 interpol=join color=red width=5; axis1 label=none minor=none; proc gplot data=curves; plot y*x=1 z*x=2 / overlay vaxis=axis1 haxis=axis1; run;
Julia
using Gadfly plot([sin, cos], -pi, pi, Theme(line_width=4pt, guides=nothing))