From FarmShare

If you have any questions, file HelpSU or ask on farmshare-discuss@lists.stanford.edu The modification date of this page is in the footer below.

Contents 1 Connecting 1.1 Connecting from Windows 1.2 Connecting from OS X / Linux / other OS 2 Logging In 3 Moving files to/from the cluster 4 Directory paths 5 Data Limits 6 Mounting your files elsewhere 6.1 Windows 6.2 OS X 6.3 Linux 7 Running jobs on the cluster 7.1 Running interactive jobs 7.2 Running batch jobs 7.3 Running parallel jobs 7.4 Running OpenMPI jobs 7.5 job duration

Connecting

• The public-facing hostname is corn.stanford.edu
• Only SSH connections are allowed. This also includes SFTP.
  • Only SSH protocol v2 is supported.
  • SSH fingerprint for corn is

 0b:e7:b4:95:03:c1:1e:07:df:04:ca:a2:3d:8e:e3:37

• If you're behind a firewall, you may want to add "ServerKeepAliveInterval 60" or "ServerAliveInterval 60" to your SSH client's configuration.

Connecting from Windows

You will want to use an SSH client like one of these:

• PuTTY: http://www.chiark.greenend.org.uk/~sgtatham/putty/
• SecureCRT: https://itservices.stanford.edu/service/ess/pc

Connecting from OS X / Linux / other OS

You should probably just use the included SSH client. Stanford does provide an SSH GUI for OS X: http://itservices.stanford.edu/service/ess/mac/lelandssh

Logging In

After your account is created, you can connect using your PennKey credentials. You can test your PennKey via the main PennKey page: http://www.upenn.edu/computing/pennkey/

Moving files to/from the cluster

Since you can connect via SSH, you can use the sftp or scp commands to upload/download files to login. On Windows, you can use software like FileZilla or WS_FTP. On OS X users can use Fetch or Cyberduck. Linux and other unix like OS just use the included sftp or scp commands.

Directory paths

Your home directory is '/ifs/h/username' (or '/gpfs/fs0/u/username'). If you belong to group 'userlab', there may also be a shared lab directory '/gpfs/fs0/l/userlab'. You can also make your home directory readable by other users or writable by other users if you like, using standard Unix permissions.

There is also local scratch storage available on the compute nodes. The amount varies from ~150GB to ~500GB, depending on the node hardware. SGE will set the env vars $TMPDIR and $TMP to point to a directory like /tmp/$JOBID. Depending on your workload, it may be a good idea to copy input or reference data to the local scratch space on the node, and then copy the results back to your homedir.

Data Limits

Currently there are no artificial limits to how much data you can store. You pay $0.10 per GB per month (we bill for the average value for the month). You can see the total storage available in the storage system with the 'df -h' command. We tend to add more storage to the system when we reach 80% utilization. You can check your own usage by checking your monthly invoice or with a command like 'du -sm /gpfs/fs0/u/username' (this may take a few mins to run). You may want to do something like the following command, which will generate a timestamped file in your homedir with the sizes of your directories. This command may take a while to run, as it will stat every single file in your homedir.

 du -sm * |sort -n |tee ~/`pwd|tr '/' '_'`.du.`date +%Y-%m-%d`

Mounting your files elsewhere

You can make your cluster home directory accessible directly from your workstation. Again, access is only allowed over SSH, so you can use something like SSHFS (via FUSE). One caveat is that SSHFS doesn't work with concurrent/parallel access, so this solution is only appropriate if you're not accessing files from several places at once. E.g. don't have cluster jobs write some files while you access them via SSHFS.

Windows

You can try ExpanDrive (used to be SFTPDrive), $39, or WebDrive, $60, or Docan (Free Software).

OS X

Try MacFUSE or ExpanDrive (above)

Linux

Just install sshfs per http://c1.genomics.upenn.edu/wiki/index.php/HPC:NFS#SSHFS

Running jobs on the cluster

We use Sun Grid Engine. There are three types of jobs: interactive, batch and parallel. You can start by reading the man page for 'sge_intro'. Then the man page for 'qsub'. We currently have a limit of 500 jobs (running and/or queued) per user.

Running interactive jobs

Use 'qlogin' or 'qrsh'. This will allocate one slot on the cluster and request 2GB RAM by default. We have memory limits set on the login node (login.genomics.upenn.edu) that will prevent you from running large jobs there. In general for any memory or computationally intensive one-off jobs, open an interactive session on a cluster node and run the command there. See HPC:Using_qlogin

Running batch jobs

Use 'qsub'. This will allocate one slot on the cluster and request 2GB RAM by default. See the bottom of the qsub man page for an example. Google 'SGE qsub' for more help.

Check how much memory your job uses. You can try running it in an interactive session first, or run just one job and see its peak memory usage after it's done. We have different size nodes, so for a 4GB node with 4 slots, you don't want to use more than 1GB per slot. For an 8GB node with 4 slots, you don't want to use more than 2GB per slot, and so on. Make sure your submitted job doesn't use too much memory or it can crash the node. See HPC:Large_memory_jobs

Running parallel jobs

Use 'qsub' with the '-pe' parameter. Using the '-pe' parameter allows you to request more than one slot per job. We have several different "parallel environments" defined, they differ in how the slots are allocated. If you want your slots on the same node, use '-pe DJ'. If you want your slots spread across nodes, use '-pe make'.

Running OpenMPI jobs

See HPC:OpenMPI, contact manager@genomics.upenn.edu with any questions

job duration

We don't currently have any limit on job duration. But we recommend you try to keep your job duration to under a day or two. Jobs that run longer than that are more likely to be affected by intermittent problems.

When jobs fail, you typically have to re-run them. So try to split them into many small chunks (but not too many).