Conda

Conda is an open source package management system.

External Links
Conda website
Conda: User Guide





Contents


• Loading and initializing Conda
• Installing and activating a Conda environment
• Running a Conda installed application
• Warning: multithreading and MPI applications
• Tips and Troubleshooting
• What is a Conda environment anyway???

Loading and Initializing Conda

Before using Conda, the following two steps are required. Conda must be initialized, and a .condarc must be created. The default shell for the new Hazel cluster is bash (however the tcsh shell is also available, type "tcsh").

1) Initialize Conda environment:

This step is necessary only once for an HPC user, unless the initialization settings are removed.

To load the system installed Conda, load the module and use init to add it to the path. Normally using a login file to automatically set the environment is strongly discouraged, but in the case of Conda, many features cannot be used without setting this initialization file. Log out and then back in again after using conda init.

module load conda
conda init bash
[ignore the warnings, log out then back in again]

Optional - remove old environments:

For users who have already been using different Conda environments and would like to begin installing with the new recommended procedures, clean out the remnants of old Conda environments by doing the following.

Check for these 'dot' files:

cd ~
more .bashrc
more .tcshrc

If these files contain information for old Conda environments, edit the files and delete this section:

# >>> conda initialize >>>
(stuff)
# >>> conda initialize >>>

2) Create a .condarc file:
This step is mandatory: Conda will fill the quota of the home directory if pkgs_dirs is not set in this file.

By default, Conda stores packages in the /home directory. The /home directory is too small for that, and the packages are only needed temporarily. They should not be saved, taking up space in permanent directories. To change the default location to /share, use a text editor to create a file called .condarc. The path to that file should be /home/$USER/.condarc and it should contain the path to the alternative location, e.g.:

pkgs_dirs: 
 - /share/$GROUP/$USER/conda/pkgs

In addition, many packages require adding a 'channel'. Common channels may be added before creating environments by editing the .condarc.

• To add channels, add them to ~/.condarc. For example, the bioconda and conda-forge channels may be added by adding the following lines to ~/.condarc:

  channels:
    - bioconda
    - conda-forge
  

The following displays a sample .condarc file:

[unityid@login01 ~]$ cd
[unityid@login01 ~]$ more .condarc
pkgs_dirs:
  - /share/group_name/unityid/conda/pkgs
channels:
  - bioconda
  - conda-forge
  - defaults

Installing and activating a Conda environment

Before installing any software, including a Conda environment, request a space for user maintained software to be used by all members of a Project. The path for that space is generally /usr/local/usrapps/groupname.

The following is the general idea of how to use Conda. Please use a YAML file to avoid package conflicts. See below for details.

To install Conda environments, specify a prefix, which will be the path to where the environment will be installed. Choose a descriptive name for the environment - Conda will create the directory, the directory should not already exist. For example, to create a Conda environment called env_ABC containing the packages AAA, BBB, and CCC, and install it in the directory /usr/local/usrapps/[groupname][username], do :

conda create --prefix /usr/local/usrapps/$GROUP/$USER/env_ABC AAA BBB CCC 

To activate the environment, do:

conda activate /usr/local/usrapps/$GROUP/$USER/env_ABC

Once in a Conda environment, a user can install additional packages using either conda install or pip install (after doing conda install pip); however, this is not recommended, as it is harder when doing so to maintain an environment where all software is compatible.

Best practice is to create a YAML file with all of the desired Conda packages. Conda will 'solve' the environment, that is, it will find a configuration where all desired packages are the correct version numbers to work together, assuming such a configuration exists. If a user needs a version of one software that is not compatible with another, then they would create two different Conda environments. When updating, create a new environment and do not delete the old version without thoroughly testing the new one.

To create a Conda environment from a YAML file called ABC.yml, do

conda env create --prefix /usr/local/usrapps/$GROUP/$USER/env_ABC -f ABC.yml

The YAML file will contain a name, a list of Conda channels to look for the packages, and a list of all the desired packages.
Here are some sample YAML files:
• datascience.yml - Contains many common data science programs
• sklearn.yml - Machine learning with scikit-learn
• biotools.yml - Contains applications for a bioinformatics workflow
• ncdfutil.yml - Used in sponsored software group ncdfutil, contains many NetCDF Utilities
• rlibs.yml - Used to create an environment with custom R libraries

When Conda creates an environment, it finds a configuration such that all of the packages/dependencies are compatible. If a great many packages are added to a YAML file, it might be impossible for Conda to resolve the necessary environment. In that case, multiple Conda environments will need to be created.

To deactivate the environment, do:

conda deactivate

Running a Conda installed application

Activating a Conda environment sets the compute environment, and is similar to loading a module. Here is a sample batch script that uses an application called mycode that was installed via a Conda environment:

#!/bin/bash
#BSUB -n 1
#BSUB -W 120
#BSUB -J mycode
#BSUB -o stdout.%J
#BSUB -e stderr.%J
source ~/.bashrc
conda activate /usr/local/usrapps/groupname/username/env_mycode
mycode
conda deactivate

NOTE: The "source ~/.bashrc" in the bash job script is necessary. You may run into issues if you don't have it before the "conda activate ......" command.

Warning: multithreading and MPI applications

• Multi-threading: Many of the applications available in Conda environments are automatically run in parallel, that is, they may auto-detect the number of cores on the nodes and spawn the same number of tasks. See the following for more information on testing, and contact HPC Staff for further assistance.
• Using MPI: Conda is package management system that installs applications and all of their dependencies. That means that if an installed package requires MPI, Conda will install its own MPI. User installed MPI will not work properly with LSF, therefore a user must use the system MPI. See the following documentation for of using system MPI to install R or Python packages requiring MPI.

  Tips and Troubleshooting

• To list the available packages contained in an activated Conda environment, do conda list.
• Alternate versions of Python and various packages may be specified by following the package name with =version.number, e.g. matplotlib=3.1.
• All installations must be done from a login node. Packages cannot be downloaded from a compute node, and neither a compute node nor an HPC-VCL node can write to /usr/local/usrapps.
• Use a YAML file! If you create a Conda environment and then attempt to add something to it, Conda may not be able to reconcile the existing applications and libraries with the new application. This is common in R or Python, when newer versions or libraries only work with newer releases. It also happens when attempting to install an older package into an existing environment. For example, older packages may need Python 2, and therefore cannot be used with a Python 3 environment. The message will say "solving environment" and it will continue for quite some time before eventually giving up. If this happens, create a new environment that includes the new application. (Keep the old environment!)
• Create multiple environments rather than adding to them. Using conda install on an existing environment may break that environment, resulting in your scripts suddenly not working anymore.
• Check your syntax: don't forget either the 'env' or the '-f' in the conda env create --prefix /path/to/env_ABC -f ABC.yml.
• PackagesNotFoundError: If you get this error, first check your syntax. Next, make sure the package exits, and if so, which channel or channels it is available from. Some Python packages are not available through Conda, and some Conda packages are only available through specific channels. You can find this information by searching for the package name on the Anaconda.org search page.
• The instructions use 'pip install'. Some packages are only available through GitHub or some other website or collaborator. In this case, go to the website for the package and carefully read the instructions. Create a YAML file with all of the dependencies listed in the instructions, including version numbers when indicated, and also include any additional applications or Python packages that you will use while doing your workflow. After creating and activating the Conda environment from the YAML file, follow the remaining instructions for the "pip install". The packages installed by pip will then be available when you activate that Conda environment.
• Python notebooks on the HPC-VCL: If the application includes examples using IPython or Jupyter notebook, they can be tested using the HPC-VCL. See these instructions for requesting access to and using the HPC-VCL.
  To add Jupyter notebook to a Conda environment, add 'notebook' to the dependencies in the YAML file:

   - notebook
  

• Python notebooks on a compute node: Jupyter notebook is a web-based coding environment that gives you the ability to run code in “chunks” rather than executing a whole script at once. Although Jupyter notebook runs in your browser, the underlying computation is executed on a remote node, which in our case is the Hazel cluster. This means that there are a few steps that need to be done to set it up.
  (1) Log in to Hazel.
  (2) Request an interactive node: As you may recall from earlier, a login node CANNOT BE USED TO RUN CODE. This means that we need to request (and log in to) a compute node. You can do this by typing:.

   bsub -Is -n 4 -R "span[hosts=1]" -W 8:00 bash
  

  This code requests an interactive job (bsub -Is) with four cores (-n 4) on the same host (hosts=1) and a time limit of 8 hours (-W 8:00), with a bash terminal. You can increase the number of cores, or reduce the time limit, or even specify a queue.
  In response to the request, the terminal will log you in to a compute node:

   
  Job <387980> is submitted to default queue single_chassis.
  Waiting for dispatch ...
  Starting on nxxx
  (base) [username@nxxx your-directory]$
  

  NOTE THE NAME OF THE NODE. In the above example it is nxxx. This is the remote host where the Jupyter Notebook instance will run.
  (3) Activate your conda environment that contains Jupyter Notebook installed. For example

  conda activate /share/$GROUP/$USER/group2_env
  

  (4) Launch your notebook by typing the following:

  jupyter notebook --no-browser --port=1113 --ip=0.0.0.0
  

  Choose a random value for the port (in this case, we chose 1113, but it can be any four digits). However, remember what port you chose, as this will be our “host port” later on.
  The terminal will pause for a minute, then show a series of lines of text, ending with the link and “token” to open the app
  (5) Copy the URL provided at the end of the page (for example,
  http://nxxx:1113/?token=ae7f7ea974d4d80f94c247e0a4bda5c3acb5408583e33a38).
  Don’t paste it anywhere yet, we first need to set up tunneling to our local machine for this to work.
  (6) Open a new local terminal on WSL, PowerShell or MacOS Terminal. In this new terminal, type the following:

  	
  ssh -N -L LOCALPORT:HOSTNAME:HOSTPORT UNITYID@login.hpc.ncsu.edu
  

  LOCALPORT can be any four digit number, just remember what you picked. For this example, I will choose 1112.
  HOSTNAME is the name of the node that Jupyter is running on. In my case, this is nxxx.
  HOSTPORT is the port that Jupyter is running on the host node, which I specified when I ran the jupyter notebook code (in my case, 1113).
  UNITYID is the same ID I use to log in to Hazel.
  So for the above example, I would type:

  ssh -N -L 1112:nxxx:1113 username@login.hpc.ncsu.edu
  

  (7) You will get prompted for your password again. Once you verify, this second terminal window will become unresponsive. This is fine, and it means that tunnelling is now set up, and you’re ready to open jupyter.
  (8) Take the link from the first window (for example, http://nxxx:1113/?token=ae7f7ea974d4d80f94c247e0a4bda5c3acb5408583e33a38), and replace the part between http:// and /? with “localhost” and the port number that you chose when you set up tunneling (LOCALPORT at point 6 above). In our case it would be: http://localhost:1112/?token=ae7f7ea974d4d80f94c247e0a4bda5c3acb5408583e33a38. Paste the link on google chrome or safari and it will open the jupyter notebook. Here you can create a new jupyter notebook file and run python code.
  (9) When you’re done and want to exit, you can shut down jupyter notebook by clicking on “Quit” on the homepage, or going back to the terminal window where jupyter notebook is running and hitting Ctrl+C (and then “yes”).
  You can also shut down the compute node by typing exit in the terminal.
  The second terminal window (the one that we set up tunneling through using ssh -N -L …) can be closed hitting Ctrl+C as well.
  NOTE: if you want to verify that Jupyter Notebook is running on a compute node, you can type in a chunk:

  import platform
  platform.node()
  

  This should return the name of the compute node that we logged into with bsub.

What is a Conda environment anyway???

• Most of the time, Conda installs applications by downloading precompiled binaries from package repositories. Sometimes it downloads source code, and then compiles it on Henry2, in which case other modules or packages may need to be installed or linked - for example, you may need to load the CUDA modules for ML/AI packages.
• The transient files (tar balls, source code, etc.) are downloaded to a temporary directory, which was defined in the .condarc in 'pkgs_dirs'. If that is not set, the default is to download to the home directory. The home directory is too small for this, and as these packages are intermediate products, they should not be saved to a directory with limited space or that is backed up. Don't waste permanent storage on it - put it in /share, as shown by the example .condarc file.
• After downloading, the applications will be installed to a location defined by '--prefix', which generally should be set to the project's space in /usrapps.
• Conda creates the equivalent of a module, and 'conda activate' does the equivalent of 'module load', i.e., it sets variables such as PATH and LD_LIBRARY_PATH to point to the proper locations in '--prefix'. (See more about environment variables here.)

Last modified: April 02 2024 13:40:32.