File Transfer

Choosing a method

Part 2 of the HPC tutorial series shows a live demo of scp and sftp (opens in new window).

Open OnDemand

Best for moving a handful of files without leaving your browser.

The Open OnDemand web portal includes a file browser that uploads and downloads files with no terminal and no software to install. Log in to the portal, then:

1. Open the Files menu and choose a location. It opens in your home directory.
2. To reach scratch, use the Change Directory button and enter the full path, for example /share/group_name/user_name. The file browser will not show /share in its location menu, so you must navigate to it by path.
3. Use Upload to send files from your computer — drag and drop, or browse for them. Whole directories may be uploaded.
4. To download, tick the checkboxes next to the files or directories you want, then press Download.

See the Open OnDemand File Explorer documentation for annotated screenshots of each step.

Globus

Best for large files, many files, and transfers you would rather not babysit.

Globus is a web-based file transfer application that moves data between two Globus endpoints. You start the transfer and Globus sees it through: it runs in the background, retries after network failures, verifies the data, and emails you when it is done. You can close the browser. This is why Globus is preferable to scp or sftp for very large files — an interrupted scp starts over, while an interrupted Globus transfer resumes.

Starting a transfer

1. Go to the Globus web site (opens in new window) and select Log In in the top right corner.
2. Select North Carolina State University (or your own institution) from the dropdown list of authentication sources and select Continue. Authenticate with your Unity ID and password followed by Duo, or with your institution's credentials and multi-factor authentication.
3. Select the two endpoints between which data will move. The endpoint for the cluster is NC State Hazel HPC Cluster. Data may move in either direction; the order in which you choose them does not matter.
4. To use your own laptop or desktop as an endpoint, install the Globus Connect Personal app on that machine first.
5. Navigate the directory trees in the web interface and select the files to transfer. Once files are selected the transfer arrow is enabled. Press it, then close the browser if you like. Globus emails you when the transfer completes.

What Globus cannot reach

• /usr/local is protected and accessible only from a login node, so transfers to /usr/local/usrapps return permission denied. To stage data or software into usrapps, use an xfer partition job, which runs on a node that mounts it.
• The scratch space in /share holds intermediate results and is subject to automatic deletion after 30 days without access, so the Globus Share function is disallowed there. You may still transfer data out of /share to a space you own, such as your local machine or Google Drive — you simply cannot use Share to expose /share to collaborators.

Users at ECU, UNC-Greensboro, and UNC-Wilmington

To use the Hazel Globus endpoint, first establish a Distinguished Name:

1. Follow the instructions to generate a Distinguished Name (opens in new window), connecting with your institution's credentials and provider.
2. Send the DN line to HPC Support requesting Globus access to Hazel. It should look something like:

   Certificate Subject:    /DC=org/DC=cilogon/C=US/O=University of North Carolina at Greensboro/CN=John Doe A10000000
   

More on Globus

• Transfer files from Research Storage with Globus (opens in new window)
• Accessing Google Drive with Globus (opens in new window)
• Video tutorial: transferring files to a personal endpoint (opens in new window)

scp, sftp, and rsync from your own terminal

Best for quick, hands-on copies when you are sitting at the keyboard.

Copying single files with scp

In these examples the file filename is copied to and from the user's scratch directory, where user_name is the Unity ID. Open a terminal on your local machine and run:

• Local machine to Hazel:
  scp filename user_name@login.hpc.ncsu.edu:/share/group_name/user_name/
• Hazel to local machine:
  scp user_name@login.hpc.ncsu.edu:/share/group_name/user_name/filename .

Copying many files with sftp

An sftp session authenticates once and then lets you move several files and browse the remote filesystem, rather than re-authenticating per file as scp does.

• Start the session, answering the password and Duo prompts:
  sftp user_name@login.hpc.ncsu.edu
  An sftp> prompt opens.
• Navigate to your scratch directory:
  sftp> cd /share/group_name/user_name
• Local machine to Hazel:
  sftp> put filename
• Hazel to local machine:
  sftp> get filename
• End the session:
  sftp> quit

Run man sftp for the full command list.

Synchronizing directories with rsync

rsync transfers only the differences between source and destination, which makes it an efficient way to keep a local copy of work on Hazel up to date. In this example the directory on Hazel is /share/group_name/user_name/myfiles and the local directory is /Users/RealName/HPC/myfiles.

Windows users should first read the instructions and video tutorial for customizing the home directory in MobaXterm to learn how to navigate to a local directory.

Open a terminal, change into /Users/RealName/HPC/myfiles, and pull down whatever has changed on Hazel:

rsync -av user_name@login.hpc.ncsu.edu:/share/group_name/user_name/myfiles/ .

It is highly recommended to first run the same command with -n, which reports what would change without changing anything:

rsync -anv user_name@login.hpc.ncsu.edu:/share/group_name/user_name/myfiles/ .

See man rsync or the online man page (opens in new window) for the many other options.

Scripted and automated transfers

Best for cron jobs, CI pipelines, instruments that drop data on a schedule, and users without an NC State Unity ID.

The login nodes require Duo authentication on every connection. Nothing running without a person in front of it can satisfy that prompt, so a scheduled rsync or an automated sftp against login.hpc.ncsu.edu will hang and then fail.

Instead, connect to the data transfer node, servxfer.hpc.ncsu.edu. It authenticates with an SSH key pair and does not prompt for a password or Duo. The same path serves users at ECU, UNC-Greensboro, and UNC-Wilmington, who have no Unity ID to authenticate with.

One-time setup

1. Ask to be added to the sftponly group by contacting HPC Support.
2. On the remote computer that will initiate the transfers, create an SSH key pair if you do not already have one:
   ssh-keygen -t ed25519
3. On Hazel, append the remote computer's public key (id_ed25519.pub, not the private key) to your $HOME/.ssh/authorized_keys file:
   cat id_ed25519.pub >> $HOME/.ssh/authorized_keys

Running the transfer

Use the same sftp, scp, and rsync commands described above, with the host changed to servxfer.hpc.ncsu.edu. Authentication happens with the key pair, so there is no prompt of any kind:

• An unattended directory sync:
  rsync -av /local/data/ user_name@servxfer.hpc.ncsu.edu:/share/group_name/user_name/
• A batch sftp session driven by a file of commands, suitable for cron:
  sftp -b commands.txt user_name@servxfer.hpc.ncsu.edu

Transfers initiated from Hazel: the xfer partition

Best for pulling data onto the cluster, long unattended syncs, and anything that would otherwise run for hours on a login node.

The two preceding sections cover transfers you start from your own machine. This one covers the opposite: a transfer you start on Hazel, which reaches out to a remote server. Both use the data transfer node, and it is easy to confuse them:

Why not just run it on a login node?

Login nodes are shared by everyone and are not meant for sustained I/O; a multi-hour rsync there degrades the cluster for other users and dies when your session drops. The data transfer node exists precisely for this work: it sits on the fast external network path and mounts the cluster filesystems. Anything that pulls a large dataset from a collaborator's server, downloads from a public archive with wget or curl, or syncs to cloud storage with rclone or aws s3 belongs in an xfer job.

Submitting an xfer job

Write a batch script and submit it with sbatch. The only directive that matters is --partition=xfer:

#!/bin/bash
#SBATCH --partition=xfer            # Run on the data transfer node
#SBATCH --job-name=stage_data
#SBATCH --time=4:00:00              # Default is 1 hour; 1 day is the maximum
#SBATCH --ntasks=1                  # One transfer process, one core (max 4)
#SBATCH --mem=8G                    # Max 24G
#SBATCH --output=stage_data.%j.out

rsync -av --partial collab@remote.example.edu:/data/project/ \
      /share/group_name/user_name/project/

Submit it as you would any other job:

sbatch stage_data.sh

You do not need --qos or --account. The xfer partition accepts only the xfer QOS, so Slurm sets it for you and prints a note in your job output confirming it. Your account is selected automatically as well. If you do pass a --qos naming something else, it is corrected to xfer rather than rejected.

Limits

The data transfer node reserves half of its cores and memory for the operating system and the transfer daemons that serve Globus and sftp. That leaves 16 cores and about 94 GB for jobs, shared by everyone, so the per-job, per-user, and per-project limits are deliberately modest:

A transfer is limited by network and disk, not by CPU, so a single core is almost always the right request — asking for more slows down your neighbors without speeding up your job. If the node is busy your job waits in the queue until cores and memory free up.

Requesting more than the per-job or per-user limits above causes the job to be rejected at submit time rather than queued, so you will find out immediately.

What the transfer node can see

An xfer job can read and write /home, /share, and /usr/local.

Watching the transfer

Check on a running transfer with squeue --me, and read its progress in the file named by --output. See Monitoring Jobs for more.

Troubleshooting

• A script or cron job hangs, then times out. It is almost certainly waiting on a Duo prompt from a login node. Point it at servxfer.hpc.ncsu.edu with an SSH key instead — see Scripted and automated transfers.
• Globus reports permission denied writing to /usr/local/usrapps. Globus cannot reach /usr/local. Use an xfer partition job.
• A transfer fails partway with a disk or quota error. Check your usage against Quotas — both space and inode counts are limited, and many small files can exhaust the inode quota well before the space quota.
• Files disappeared from /share. Scratch is not backed up, and files that have not been accessed in 30 days are deleted automatically. Keep anything you need to retain elsewhere; see HPC Storage.
• Anything else. Please contact HPC Support for questions and assistance.