1. OIT HPC
  2. Storage
  3. Home Directory Quota

Tips to Remedy and Prevent Over Quota Storage Spaces

Webpage Table of Contents:

  • Diagnosing the problem
  • Special notes about home storage space
  • Special note about shared storage spaces such as /share/$GROUP and /usr/local/usrapps/$GROUP
  • Asking for larger space or inode quotas

    • Diagnosing the problem

    First, check which storage space is over quota:

    quota_display

    Note, quotas can also be viewed for your non-default groups (if any exist) using the following arguments

    quota_display $USER group_name    # fill in appropriate value for group_name

    The quota_display command shows quotas for home ($HOME), scratch (/share/$GROUP), and usrapps (/usr/local/usrapps/$GROUP) storage spaces. There are two types of quotas: size (i.e., space) and files (i.e., inode count). Commands below are designed to check both types of quotas.

    First, change directory to the storage space that is over quota, e.g., if your home space is near or over quota then issue the command cd $HOME. Then run these diagnostic commands:

    du -hd 1             # show size of each subdirectory, typical units are (K)ilobytes, (M)egabytes, and (G)igabytes
du -hd 1 --inodes    # show file count of each subdirectory, typical unites are (K)ilo, (M)ega, and (G)iga

    The above commands will help determine which subdirectories are large. Note, the "." directory is the current directory.

    To drill deeper into a large subdirectory:

    cd <large subdirectory>
du -hd 1
du -hd 1 --inodes

    Repeat as needed to find exactly what is consuming space or inodes.

    After deleting unnecessary files, run quota_display again to confirm the storage spaces are well within quota.

    Special notes about the the home storage space

    The $HOME/.cache directory

    A common directory that is large in size, and especially large in inodes, is the .cache directory. The .cache directory is where pip and conda store temporary files during installation of packages. The $HOME/.cache directory can be safely deleted (verify you have not manually added your own files to it first).

    Preventing home directory from filling up again

    1. Configure conda and pip package managers so that they do not use the home directory:
      1. conda: Follow directions under "Loading and Initializing Conda".
      2. pip: Follow directions under "Prevent pip and Python from filling up your home directory".
    2. Do not run jobs from your home directory. Use scratch space (/share/$GROUP) instead. Note, files not accessed for 30 days in the scratch space will be auto-deleted, so move important data elsewhere
    3. Do not install software in your home directory. Use usrapps space instead.

    Special note about shared storage spaces such as /share/$GROUP and /usr/local/usrapps/$GROUP

    When running the du commands above, there may be some directories that are inaccessible. If any directories show they are owned by a number (rather than a user name), then that is likely someone who no longer has HPC access (run "ls -l" to see owners of directories/files). With the HPC Project Account owner’s written permission, HPC Support can remove specific directories for you, or change ownership to the project owner so that they can delete what they need to. If file/directory deletion or change ownership is needed, then please email HPC Support, and cc the the project owner on the email. We will need direct written confirmation (via email) from the project owner in order to delete files/directories.

    Asking for larger space or inode quotas

    Please first try working through the methodologies on this page. If larger quotas are needed, then please email HPC Support