Computing

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
** [[:start|Back To Main Page]] **

====== Overview ======

The CfN cluster is a linux-based system (CentOS 6.9). You connect to it using //ssh//, a tool for remote secure connections between computers. The initial connection and your interaction with the cluster are done using a standard linux //shell// environment (aka 'command line', 'command line interface (CLI)' or 'terminal'). This is a text window where you type commands and get text back as a response. 

----
====== Linux Bash Shell ======

The default cluster linux shell is //bash//. Other shells you may see mentioned elsewhere are //sh//, //csh//, //tcsh//, and //ksh//. These are all variants of each other and can have confusing differences.

=== .bash_profile and Search Path and Prefernces ===
Modify your search path in your ~/.bash_profile file, and not ~/.bashrc.
If you don't know what this means, see the tutorials below.

=== Tutorials and Help ===

If you've never used a linux or unix shell before, you'll want to learn  something about it. There are a lot of resources online, or your colleague might have a favorite book. Here are some places to start:

Here's one for the very beginners that takes you through a tutorial:
[[http://ryanstutorials.net/linuxtutorial/|Linux Tutorial]]

Here's a good-looking introduction to the [[http://linuxcommand.org/lc3_learning_the_shell.php|basics of the shell]] and [[http://linuxcommand.org/lc3_writing_shell_scripts.php|writing scripts]]. It's also available as a more in-depth eBook online: [[https://docs.google.com/gview?url=http://it-ebooks.info/read.php?id%3D2012-1406926599-100d2f93bf736387a9d1f8008a48d765&chrome=true|The Linux Command Line]]. You can get it in print too.
 
A shorter introduction, more technical: [[http://www.tldp.org/LDP/Bash-Beginners-Guide/html/chap_01.html]]

----
====== Some Useful Linux Things =====

==== Screen & tmux ====

The ''screen'' and ''tmux'' tools are very useful. They let you create multiple terminal instances from one screen, and most importantly, let you 'detach' from a terminal instance/session and reconnect to it later, even if you've logged out, or lost your network connection or have closed your terminal window. This is **very** useful for running a long terminal program without having to keep your terminal and computer running.

Screen - [[https://uisapp2.iu.edu/confluence-prd/pages/viewpage.action?pageId=115540034|Very Simple Usage Instructions]]

tmux - [[https://hackernoon.com/a-gentle-introduction-to-tmux-8d784c404340|A Gentle Introduction to tmux]]

==== Simple Bash script loop ====

[[using_ogs_sge#simple_bash_loop|Here's a simple example]] of a Bash script to loop through patients and run a ''qsub'' job on the cluster.

----
====== Windows and the Linux/Bash Shell ======

There are a number of ways to use Linux/Bash commands on Windows machines. This can be useful for running ''rsync'' for file copying, for example. The options include:

== Windows Subsystem for Linux ==

This is a new subsystem in Windows 10 for running Bash shell. It's not a full Linux VM.

https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/

== Cygwin ==

Tried and true, a little awkward though. Known to work for ''ssh'' and ''rsync'', although I'm not sure about X-Windowing/X11/GUI support.

https://www.howtogeek.com/175008/the-non-beginners-guide-to-syncing-data-with-rsync/

== Git bash shell ==

Git on windows includes its own shell app. This has been touted as being good and simple for ''ssh''. ''rsync'' might work as well.

http://www.hurryupandwait.io/blog/need-an-ssh-client-on-windows-dont-use-putty-or-cygwinuse-git

----
====== Logging In ======

Run your //terminal// program to get a command line and login to the cluster.

==== Linux and Mac OSX terminal ====

These have a terminal application built-in. In the Desktop search tool, search for //terminal// and it should pop up.

==== Windows terminal ====

Sadly, it's not so easy for Windows. You need to install your own terminal application.

=== MobaXterm ===
[[http://mobaxterm.mobatek.net/ | MobaXterm ]] is a newer, good-looking ssh client terminal that includes an integrated X-Server. I haven't tried it, but some users like it.

=== Using PuTTy ===

A popular free terminal app is PuTTy. [[http://www.chiark.greenend.org.uk/~sgtatham/putty/|You can download it here]].

PuTTy is kind of awkward to work with. Some quick notes:

  * From the main window ('Session' on the left category menu)
      - In //Host Name (or IP address)//, enter ''chead.uphs.upenn.edu'' 
      - In //Port//, enter ''22''
      - For //Connection Type//, choose //SSH//
      - In //Saved Sessions//, enter ''chead'' 
      - Click the //Save// button to the right

  * To setup **X11 windowing** (X-forwarding) for GUI applications
      - In the //Category// menu on the left, click //SSH//, then //X11//
      - Click //Enable X11 forwarding// so the checkbox has a check in it
      - For //Remote X11 authentication protocol//, choose //MIT-Magic-Cookie-1//
      - In the //Category// menu on the left, click //Session// at the top
      - You're now back in the main window
      - Click //Save// button to the right again
      - You also need an X-Server app if you don't already. See below.

  * When you start a session in Putty (by clicking the "Open" button):
      - at the '' login as: ''  prompt, **enter just your username**. Do NOT enter "ssh my-username@chead"

Note that PuTTy makes a connection before it starts up a terminal window (i.e. where you type you commands), so the //ssh// command described below happens through the setup and when you choose //Load// or double-click a session name, and not from the terminal itself.


==== ssh - actually connecting ====

From your terminal, run the 'ssh' command to connect to the cluster, like so (unless you're using PuTTy on Windows, in which case [[#Windows terminal|see above]]):

  ssh -Y your-username@chead   (if this doesn't work, try the full address: chead.uphs.upenn.edu)

The '-Y' option is needed to run GUI applications on the cluster, see below.

=== Common errors ===

__If you see an error like one of these:__
  Error: Can't open display: 
  Warning: No display specified.
  Warning: No window system found
  cannot connect to X server
  
it means the X-forwarding isn't working. Make sure you used the ''-Y'' option to ssh, or otherwise setup your terminal software properly (e.g. [[#Windows terminal|PuTTy has particular settings]]).
  
__If you get an error like:__

  ssh: Could not resolve hostname chead: nodename nor servname provided, or not known

it most like means you are not connected to the UPHS Network, and need to [[accounts_and_vpn#UPHS Account and VPN Access|use VPN]].

----

====== Graphical/GUI Applications - X11 / XWindows ======

If you want to run GUI applications (aka graphical, or windowed applications) on the cluster (for example the Matlab graphical desktop environment), you need to run an X11 windowing session. That is, you need to have a //X Server// running on your computer. Note, the server/client terminology is reversed for X11 from traditional usage - your local computer on your desktop is the //X11 Server//, and the remote computer that's running the GUI application is the //X11 Client//.

===When X11 is slow===
If running a GUI application over X11 is slow, you'll probably be much better off using the //x2go// remote desktop tool. See [[x2go_Remote_Desktop|x2go Remote Desktop]]. Connecting over VPN from off-campus is often much better using x2go, but sometimes on-campus connections can be slow and x2go can help.

===The ssh -Y option===
When you ''ssh'' to chead, be sure to **use the -Y option** as mentioned above.

===A Quick Test===
Once you follow instructions below to get setup, a quick test of your X11 functionality is to login to chead and run the command '' xclock ''. You should get a graphical clock window open up on your computer.
==== -- Linux -- ====

There nothing to do, Linux should be setup out-of-the-box to accept X11 connections.

==== -- Mac OSX -- ====

You need to install [[http://xquartz.macosforge.org/landing/ | XQuartz]], the open-source X Server for Mac (developed orginally by Apple).

=== XQuartz 2.7.11 Issue ===
The issue mentioned below for 2.7.9 persists, however the fix is simpler. On your local machine (i.e. your laptop or desktop, not the cluster), run this command in Terminal:

    defaults write org.macosforge.xquartz.X11 enable_iglx -bool true

=== Xquartz 2.7.9 Issue ===
June 2016

Xquartz 2.7.9 may break some programs like ITK-SNAP when running over ssh. Here's a fix (thanks to Phil Cook!):

To fix, I had to edit the startup config **on my own machine (not on chead)**:

  sudo vi /usr/X11R6/bin/startx   <--- do this on your machine, not on chead

changing Line 59 to

  defaultserverargs="+iglx"

Then from the command line,

  defaults write org.macosforge.xquartz.X11 enable_iglx -bool true

After this and a restart of XQuartz, it works as normal.

=== Xquartz 2.8.8 fallback ===

**If this still doesn't work**, you might have luck downgrading to [[https://www.xquartz.org/releases/XQuartz-2.7.8.html| version 2.7.8]]
==== -- Windows -- ====

You'll need an X Server app like [[http://mobaxterm.mobatek.net/ | MobaXterm ]], or the [[http://sourceforge.net/projects/xming/ | Xming]].

Note that mobaXterm is multi-tool application, including a terminal and sftp client, but its X11 Server is available to other applications on your system (like the x2go client), in our experience.

=== Terminal application ===

You'll need a terminal application, that is, a command-line text application that can connect to remote servers.
We recommend the mobaXterm app (see above) that includes terminal and sftp (and other) tools along with the X11 Server functionality.

=== Putty on Windows ===

If you're using //PuTTy// as your terminal app, make sure to follow the directions under [[logging_in#Windows terminal]] to setup //X-forwarding//.

=== SecureCRT on Windows===

For SecureCRT, see this page: http://www.seas.upenn.edu/cets/answers/x11-forwarding.html

----
====== Your Password - Changing It ======

To change your password, run the ''passwd'' command on the command line. 

  * **You must use a strong password, that does not include dictionary words, repeating characters, and that does include some mix of capital letters, symbols and/or numbers.**

----
====== 'rm' command WARNING ======
**This is a general warning about using the command ''rm'' and in particular ''rm -fr''.**

**When you delete a file in Linux using ''rm'', you can NOT recover it.** We maintain tape backups of all cluster data on a 2-3 month cycle. These are intended for worst-case scenario disaster recovery, and thus generally backups are not available for mistaken deletions by individual users, although you can check with the admins.

This command removes whatever files **and directories** you list after it without asking for confirmation, //and// recursively, meaning it goes into every sub-directory and deletes that too. One user recently ran ''rm -fr /*'' instead of ''rm -fr ./*''. The missing '.' meant that instead of trying to delete everything in her current working directly, it tried to delete everything **on the whole front end**. Most files were protected so she couldn't delete them, but when the command hits /data, it will delete **everything** in there that is owned by  her, or for which she has permissions to delete because of group membership. This could be very bad, obviously. Luckily in the recent case the user stopped the command before it got to the /data directory.

This warning also applies to the ''rm'' and ''rm -r'' commands, which by default will remove files without asking confirmation.

You may also have an issue, for example, if you are in a different directory than you think, and run something like ''rm *.nii''. It will remove all files in the current directory without prompting for confirmation, and you'll have lost them for good.

===How to use it carefully===
Be careful. It's often good to replace your ''rm'' command with ''ls'' first to make sure you're getting what you intend. After you run ''ls <my list of files or glob command>'', use the up-arrow key in your terminal to recall the previous command and edit it to replace ''ls'' with your ''rm'' command, so that you're sure not to introduce a typo in the list of files or glob command.
----
====== Editing Text Files ======

You'll probably need to edit text files. If you're new to linux/bash, here are some common text editors. To use them, type their names on the command line:

  * **gedit** - a popular simple graphical text editor
  * **emacs** - __the__ classic unix/linux text editor. On the cluster it comes up as a GUI editor so is easier to use than the classic text-only version. It's all powerful but kinda dense to learn.
  * **vi** - simpler than emacs, but even more dense

----
====== Check Your Disk Quota ======

Your data and home directories have quotas setup that limit how much disk space you can use in them.
To check your quota run:

  df -h /data/<mydir>

where <mydir> is something like ''jet/mgstauff'' or ''jag/myusername''.

----
====== File Transfers & Accessing Data Drives ======

Generally you'll need to move files between your local/desktop computer and data directories on the cluster.

See [[fileaccess|this page]] for details.
Computing

User Tools

Site Tools

Page Tools
