Supported mechanisms for file transfers include using a desktop terminal or using Globus.
Please review HPC Storage before moving data to the cluster.
Note: Part 2 of this HPC tutorial shows a live demo of scp and sftp.
Open the Terminal by opening the Applications folder, then the Utilities folder, and then click Terminal.app.
Alternatively, use the Mac 'Spotlight Search' tool (hit [Command+Space] or click the magnifying glass in the tool bar) and search for Terminal.
Download and install the MobaXterm Home Edition (Installer edition).
Open MobaXterm and click Start local terminal.
Use SCP to transfer single files.
In this example, the file filename will be copied to/from the user's scratch directory on Hazel using SCP.
From the local machine, open a terminal window and use SCP, where user_name is the Unity ID.
scp filename user_name@login.hpc.ncsu.edu:/share/group_name/user_name/
scp user_name@login.hpc.ncsu.edu:/share/group_name/user_name/filename .
To transfer multiple files without needing to re-authenticate, or to be able to navigate through the remote file system, use SFTP.
In this example, the file filename will be copied to/from the user's scratch directory on Hazel using SFTP.sftp user_name@login.hpc.ncsu.edu
cd /share/group_name/user_name
put filename
get filename
quit
Use man sftp
for a complete list of SFTP commands.
Use Globus to transfer files with a GUI interface or to transfer very large files.
Globus is a web based file transfer application that allows resilient, unattended file transfers between two Globus endpoints. Start the transfer and Globus ensures it completes successfully and sends email when the transfer is done. Globus may be preferable to SCP or SFTP when transferring very large files because it does so unattended, in the background, with status checking and fault tolerance.
To initiate a data transfer, go to the Globus web site and select the Log In button in top right corner of the page. Select North Carolina State University (or your institution) from the dropdown list of authentication sources and select Continue. On the NC State Shibboleth or your institution's authentication page, enter the Unity ID and password followed by Duo authentication or your institution account name and password followed by your institution's multi-factor authentication.
Select the two endpoints in between which the data transfer will occur. Data may be moved to or from either endpoint; the order does not matter. The Globus endpoint for the Hazel cluster is NC State Hazel HPC Cluster. To set a personal location such as a laptop or desktop as an endpoint, the Globus Connect Personal App must be installed on the local machine. Once endpoints are selected, navigate the directory trees from the web interface. Select the file or files to be transferred. Once files are selected, the arrow button for initiating the transfer will be highlighted. Press the button to initiate the transfer. Once initiated the browser window may be closed. Globus will send email when the transfer is complete.
Additional links on Globus for HPC data
ECU, UNC-Greensboro, and UNC-Wilmington users
To use the HPC Globus endpoint, ECU,UNCG, and UNCW users should do the following:
Certificate Subject: /DC=org/DC=cilogon/C=US/O=University of North Carolina at Greensboro/CN=John Doe A10000000
Troubleshooting
The Open OnDemand web portal includes a file browser that can be used to transfer files up to 10GB. The browser points to your home directory by default, however, you can navigate to /share directories by entering the path you want to move files to or from.
Use rsync to synchronize the files in a local directory with a directory on Hazel.
Rsync transfers only the differences between the source files and the existing files in the destination. It is an effective way of backing up files on Hazel to a local machine. In this example, a directory on Hazel is /share/group_name/user_name/myfiles and the directory on the local machine is /Users/RealName/HPC/myfiles.
To update the files in the local directory with those that have been modified on Hazel, open a terminal window, navigate to the location of /Users/RealName/HPC/myfiles, and do
rsync -av user_name@login.hpc.ncsu.edu:/share/group_name/user_name/myfiles/ .
To see the many additional options and use cases, type man rsync
or see the online man pages.
Before using rsync, it is highly recommended to use the -n
option to test which changes are to be made:
rsync -anv user_name@login.hpc.ncsu.edu:/share/group_name/user_name/myfiles/ .