Welcome to JHU Astro5 Compute Server

How to log in and who to contact

First you need to be added as an astro5 user. Request access by emailing jesse.warford [at] jhu.edu.

If you are outside Hopkins, start by launching VPN. If you are inside Hopkins, make sure you are on "hopkins" wireless network (as opposed to "eduroam", for example).

Log in by typing "ssh [email protected]" in your terminal / X11 window and using your usual JH password (replace jdoe1 by your JH ID). After you've logged in, make sure that you have window forwarding set up correctly. If you type "xterm &" it should open a new window with astro5 access; if no window appears, you need to fix issues with your terminal / X11 software. You might also try "ssh -X [email protected]" or "ssh -XY [email protected]" and see if this helps. For Macintosh users, the XQuartz software seems to provide adequate access.

Quotas are 50 GB per user. To check your current use: "quota -s" [still working on it, so you might see 'none' for the moment, so please watch your usage].

All files in /data1/ and /home/ are backed up via CrashPlan Pro to the cloud. Crash Plan monitors the file system and makes backups multiple times a day. It also keeps versions of files from each backup. If you need to have a file or files restored, please e-mail pha-help [at] jhu.edu with the specific files or location.


Page content


Some basic rules
What is on the server right now
Python instructions
IDL instructions
SM / SDSS spectro instructions
IRAF instructions
External mount instructions

Some basic rules

The idea is that large datasets that multiple users can access live in /data1/ and your own files, code, results of data processing etc live in your /home/ directory. Please observe this convention. Furthermore, regularly check how much disk space you occupy on /home/ and clean up.

There are five super-users who can edit and erase /data1/ directories created by some other users. This was done in response to some students storing large messy directories with their own calculations in the /data1/ directory. Right now anybody can write into the /data1/ directory, but please avoid doing that unless you are sure that you are uploading a useful dataset or you got an approval of one of the super-users.

The super-users are Nadia, Kate, Ai-Lei, Dominika and Jorge. It looks like as of now (June 2019) only Nadia is still a JHU-based super-user. Astro5 is a self-maintained computational resource. If you know what you are doing and you want to be a super-user, please let Nadia and Jesse know and we'll add you. We also need users to test software, report issues and propose solutions. We need (wo)manpower to maintain things in order!

What is on the server right now

Computing software:
SuperMongo -- seems to work, see setup instructions below
Python -- it looks like the most convenient way is to do this as a user, see below
IDL -- seems to work, see path intructions below
IRAF -- seems to work, including tv display forwarding, see instructions below

Public data:
/data1/sdss4/manga/mpl5 (IFU spectroscopic survey SDSS-IV, MaNGA, MPL5 release)
/data1/boss (spectroscopic survey SDSS-III/BOSS, v.5_7_0)
/data1/sdss_dr7 (spectroscopic survey SDSS-I/II)

Need another big dataset? If it's no larger than a few hundred GB, create a sub-directory in data1 with a self-explanatory name and put the data there. It will be accessible to all users, though only writeable / deletable by you. Unsure? Send an email to Jesse Warford.

Utilities:
Emacs
Latex [I need help testing the latex installation; so far I am having trouble compiling aastex6 files even when style files are supplied]
evince for eps, pdf viewing
okular for pdf viewing
xterm
ds9
fv
Working on: finger

Instructions for python

The preferred method of python use is a personalized user install. First download the anaconda for linux (64 bit) installer from here and put the file inside your user directory on astro5 /home/jdoe1. Then you follow the instructions on that webpage for installing either the python 2.7 or 3.5 version depending on which one you want and which file you download (this is just one bash command). If you don't know what you are doing, download the 2.7 version. It will also ask if you want to add this python path to your .bashrc file, to which you respond yes to make this version of python the default. The anaconda install includes and install of pip which you can now use to install [most] python packages. To make sure all the paths are working, log out and log back in.

Anaconda does not use the default centralized python installation. When you perform a user install of anaconda, it builds your version of python inside a new path, which by default is /home/jdoe1/anaconda2/bin/python. It also installs over 100 of the most common python packages including pip, so that you can install whatever else you need. For example, you should be able to do something like "pip install rebound" to install the Rebound package.

Instructions for IDL

A bunch of IDL libraries that are widely used, including coyote, astron, mpfit etc, have been uploaded to /data1/lib/idl. If there is another library you would like to add which you think might be of public interest, please email Jesse Warford.

idlutils and kcorrect have been installed (and require compilation), but not yet tested.

To take advantage of these libraries, you need to point your IDL to them. Add the following line to your /home/jdoe1/.bashrc file:
export IDL_PATH=+/data1/lib/idl:$IDL_PATH

Instructions for viewing SDSS spectra

Step 1. Setting up SuperMongo

SuperMongo v.2.4.38 is installed. (If you want to install SuperMongo on a Mac OSX 10.8 and later for your own use, ask Nadia how to modify the source code.)

Put this file in your home directory /home/jdoe1 and edit to reflect your user ID.

Create a directory named sm in your home directory. Into that directory place this file. The setup macro on top of that file points to the files which will be used to access SDSS-I/II and SDSS-III spectra (stored in /data1/lib/sm and accessible by all users). You can edit the greeting which is on the top line of the file (starts with a comment character) and you can add any other "include" statements if you have your own library of sm routines you want to include by default.

Now type into your command line prompt "sm setup". You should have received a greeting and you now have the sm command line prompt. If you want to try it out, copy the contents of this file into your buffer and paste into the sm prompt. This is a test file that generates some data, fits a line to the data and makes both an sm xterm plot and a postscript plot which should open automatically. Alternatively, if you are running sm from a directory which also contains the test.sm file, you can say "include test.sm play" with the same result as copy-pasting. If no windows pop up as a result of "sm setup" or "include test.sm play", your window forwarding on the terminal client is not set up properly.

SuperMongo is a lovely plotting tool, with detailed instructions and tutorial on Robert Lupton's page.

Step 2. Viewing SDSS spectra

Put together a list of your objects in ascii format, with columns plate fiber mjd. For example, here is a list of type 2 quasars from SDSS I / II and here is a list of type 2 quasars from BOSS in the required format.

In the directory where this file is located, type "sm setup". BOSS is loaded by default. So you can start viewing your BOSS spectra immediately by typing "looklist boss_type2.list". You can toggle showing of special lines; you can change the redshift if you don't think that the default one is correct; you can zoom in on a part of the spectrum to see it better. At the prompt, press 'h' for help with all available commands. For example, a very useful one is 'c' -- to place a comment on this object in the comments file (e.g., 'c this is not a type 2 quasar'). Press 'q' to exit. In your current directory, there will be a file boss_type2.list.comments with your comments.

To examine an SDSS I / II file, first type "dr7" in the sm prompt to load proper paths (you can return to BOSS by typing "boss"). Now you can do "looklist dr7_type2.list".

Another useful feature is to be able to look at all the spectra on a plate. For example, "boss" then "look 5000", or "dr7" then "look 400". You can examine spectra and make comments about them; they will be in the 400.comments file in your current directory. "Look" uses the MJD that gives the highest S/N for the plate. If you need the functionality to look at the plate with a particular MJD, let Nadia know and we'll fix the code to provide this functionality.

To interactively view one spectrum specified by plate-fiber-MJD, use "plotdate 4964 116 55749" (plotdate plate fiber mjd).

Instructions for using IRAF

IRAF is launched by typing "cl" or "ecl" in the command line. Apparently we have a couple of different installations, and we have other things we can try to install that we haven't tried yet. We would greatly appreciate those with IRAF experience to test these installations and if external packages are needed, please let Jesse know and we'll work together on this.

To make IRAF's "display" commands work, launch "ds9 &" before starting "cl", then "images", then "tv", then "display test" (if you have a file test.fits in the launch directory), and it should forward to the open ds9 window.

Instructions for mounting Astro5 as an external disk

Dominika: I wanted to share this really useful piece of software with you that just made my life so much easier. There is a piece of Mac software that allows you to mount a remote server (such as the astro5 server) as an external disk to your Mac Finder.

You need to download FUSE for MacOS and SSHFS (both are available from here) and then run this line in your terminal:

sshfs username@hostname:/remote/directory/path /local/mount/point -ovolname=NAME

The local mount point (in my case /Users/dominika/mount -- this directory first needs to be created using mkdir mount) then looks like a external disk in your finder and gets the name NAME (specified in ovolname) and you can just eject it when you are done as you would do with an external hard drive.

Example:
Install FUSE and SSHFS 'out of the box';
In your home directory run "mkdir mount";
Type something like "sshfs [email protected]:mywork /Users/jane/mount -ovolname=astro5";
When prompted, enter your astro5 password;
In your Finder, you should now see a directory "astro5" in your home directory;
In your terminal, you should be able to "cd /Users/jane/mount" and have access to the content of mywork directory on astro5.

Now I can use my local Python or IDL scripts and just refer to the data in the data1 folder on astro5 like I would as if they were on an external hard drive. In my case I use a local Python script with this line for example: list_filenames = np.loadtxt('/Users/dominika/mount/sdss4/manga/mpl5/map_list.txt', dtype = str)

This discovery just significantly improved my life as I could for example now edit map_list.txt (which is on the astro5 server) with my favorite text editor on my Mac. I upgraded to MacOS Sierra this weekend, so this works also for the latest version of the MacOS.

You can unmount the mounted directory (astro5 in the example above) from the Finder. If your connection gets interrupted without the mount directory properly unmounted, then it can be brute-force disconnected following instructions from here:
Find the culprit sshfs process: "pgrep -lf sshfs"
kill it: "kill -9 <pid_of_sshfs_process >"
sudo force unmount the unavailable directory: "sudo umount -f <full_path_to_mounted_dir>"

You can now remount the same directory using sshfs. If there is still a problem, another possibility is "diskutil unmount <full_path_to_mounted_dir>".