High Performance Computing

File transfer using the terminal

How do I open a terminal window?

MacOS/Linux

Terminal is already included in macOS.

Open the Terminal by opening the Applications folder, then the Utilities folder, and then click Terminal.app.
Alternatively, use the Mac 'Spotlight Search' tool (hit [Command+Space] or click the magnifying glass in the tool bar) and search for Terminal.

Windows

MobaXterm
Download and install the MobaXterm Home Edition (Installer edition).
Open MobaXterm and click Start local terminal.

Click here for more detailed instructions on installing and configuring MobaXterm.

Transferring files using SCP:

Use SCP to transfer single files.

In this example, the file filename will be copied to/from the user's scratch directory on Hazel using SCP.

From the local machine, open a terminal window and use SCP, where user_name is the Unity ID.

Local to Hazel
scp filename user_name@login.hpc.ncsu.edu:/share/group_name/user_name/
Hazel to local
scp user_name@login.hpc.ncsu.edu:/share/group_name/user_name/filename .

There will be a prompt for a password and Duo authentication.

Transferring files using SFTP:

NC State users with Unity IDs

To transfer multiple files without needing to re-authenticate, or to be able to navigate through the remote file system, use SFTP.

In this example, the file filename will be copied to/from the user's scratch directory on Hazel using SFTP.

From the local machine, open a terminal window and initiate an SFTP session.

To initiate an SFTP session, type:
sftp user_name@login.hpc.ncsu.edu
where user_name is the Unity ID. There will be a prompt for a password and Duo authentication.
An SFTP prompt will be opened:
sftp>
Navigate to the share directory:
sftp> cd /share/group_name/user_name
Local to Hazel
sftp> put filename
Hazel to local
sftp> get filename
To exit the SFTP session:
sftp> quit

Use man sftp for a complete list of SFTP commands.

ECU, UNC-Greensboro, UNC-Wilmington Users and Automated File Transfers

A file transfer node exists in the cluster to support sftp sessions from users who do not have Unity IDs and to support automated file transfers. Some setup is required to access the file transfer node.

Send request to OIT_HPC@help.ncsu.edu to be added to sftponly group
On your local computer that you will be transfering files to/from create an ssh key pair (if you do not already have a key pair for your account on that computer).
On Hazel, add the PUBLIC key to your $HOME/.ssh/authorized_keys file. It is critical to not alter the key already present in the file which enables access to cluster compute nodes.

From your local computer you can now run sftp as described above to transfer files to/from servxfer.hpc.ncsu.edu. Authentication will happen using the ssh key pair, there should be no password prompt.

File transfer using Globus

Use Globus to transfer files with a GUI interface or to transfer very large files.

Globus is a web based file transfer application that allows resilient, unattended file transfers between two Globus endpoints. Start the transfer and Globus ensures it completes successfully and sends email when the transfer is done. Globus may be preferable to SCP or SFTP when transferring very large files because it does so unattended, in the background, with status checking and fault tolerance.

To initiate a data transfer, go to the Globus web site and select the Log In button in top right corner of the page. Select North Carolina State University (or your institution) from the dropdown list of authentication sources and select Continue. On the NC State Shibboleth or your institution's authentication page, enter the Unity ID and password followed by Duo authentication or your institution account name and password followed by your institution's multi-factor authentication.

Select the two endpoints in between which the data transfer will occur. Data may be moved to or from either endpoint; the order does not matter. The Globus endpoint for the Hazel cluster is NC State Hazel HPC Cluster. To set a personal location such as a laptop or desktop as an endpoint, the Globus Connect Personal App must be installed on the local machine. Once endpoints are selected, navigate the directory trees from the web interface. Select the file or files to be transferred. Once files are selected, the arrow button for initiating the transfer will be highlighted. Press the button to initiate the transfer. Once initiated the browser window may be closed. Globus will send email when the transfer is complete.

Additional links on Globus for HPC data

Transfer files from Research Storage with Globus

Accessing Google Drive with Globus

Hazel Video tutorial - Transferring files to a personal endpoint (e.g. local machine)

ECU, UNC-Greensboro, and UNC-Wilmington users
To use the HPC Globus endpoint, ECU,UNCG, and UNCW users should do the following:

Follow the instructions to generate a Distinguished Name, connecting with UNCG credentials and using the UNCG provider.

Send the DN line to HPC Support requesting Globus access to Hazel.
The DN line should look something like

Certificate Subject:    /DC=org/DC=cilogon/C=US/O=University of North Carolina at Greensboro/CN=John Doe A10000000

Troubleshooting

The space /usr/local is protected and can only be accessed from a login node, thus attempts to transfer files from Globus to the /usr/local/usrapps directories will return permission denied. The mass storage is also only accessible via the login nodes.

The scratch space in /share is meant for intermediate results from running jobs, therefore the Globus Share function is disallowed. To clarify, Globus can be used to transfer data from /share to a space owned by the user (local machine or Google Drive), but it cannot use the Share function to share data that resides in /share between collaborators by default as that data is subject to automatic deletion if not accessed for 30 days.

Please contact oit_globus@help.ncsu.edu for questions and assistance.

File transfer using Open OnDemand

The Open OnDemand web portal includes a file browser that can be used to transfer files up to 10GB. The browser points to your home directory by default, however, you can navigate to /share directories by entering the path you want to move files to or from.

Synchronize local and remote files with rsync

Use rsync to synchronize the files in a local directory with a directory on Hazel.

Rsync transfers only the differences between the source files and the existing files in the destination. It is an effective way of backing up files on Hazel to a local machine. In this example, a directory on Hazel is /share/group_name/user_name/myfiles and the directory on the local machine is /Users/RealName/HPC/myfiles.

For Windows users, first see the instructions and video tutorial for customizing the home directory in MobaXterm to inform how to navigate to the desired local directory.

To update the files in the local directory with those that have been modified on Hazel, open a terminal window, navigate to the location of /Users/RealName/HPC/myfiles, and do
rsync -av user_name@login.hpc.ncsu.edu:/share/group_name/user_name/myfiles/ .

To see the many additional options and use cases, type man rsync or see the online man pages.

Before using rsync, it is highly recommended to use the -n option to test which changes are to be made:
rsync -anv user_name@login.hpc.ncsu.edu:/share/group_name/user_name/myfiles/ .