Python
Python is an interpreted, object-oriented, high-level programming language. Python is often used for rapid prototyping; it is optimized for developer productivity rather than execution speed.
External Links:
Python website
Python: Getting Started
Contents:
Prevent pip and Python from filling up your home directory
Python versions and updates
Running Python
Batch Python
Python from a Conda environment
Interactive test and debug
Running distributed memory (MPI) Python
Prevent pip and Python from filling up your home directory
The pip application is a standard package installer for Python. By default, it deposits files in your home directory, causing serious quota issues for users. Two steps must be taken to prevent this issue:
- Change the location of your pip cache directory: Navigate to this page and follow the instructions to prevent pip from filling up your home directory with temporary downloaded cached package installation files.
- Ensure that pip does not install packages into your home directory: Create a conda environment with pip in it so that pip installs packages into your conda environment (typically under
/usr/local/usrapps/$GROUP/$USER/)
Also, see this page if you are having problems with your home directory reaching quota (command quota_display shows the quota status of your home directory, among other spaces).
Python versions and updates
There is a default Python 2 that is included in the Linux distribution installed on Hazel, and a staff maintained Python 3 that is available as a module. The system Python is updated whenever there are system Linux upgrades, and Conda (which has a default of Python 3) is updated as soon as upgrades become available. Updates to Conda will be posted on the Cluster Status page.
Many Python packages have dependencies, and updating a dependency for one package may break another package. For this reason, the staff maintained Python does not contain any additional Python packages other than the basic install. Virtual environments created based on a Python version may break if the base Conda install is updated.
To avoid 'breaking' a Python environment, users should create Conda environments. A Conda environment should not change if the base Conda is updated.
To use the system Python 2 distribution, no environment changes are needed. To load the current Python 3 environment, usemodule load conda
Running Python
Do not use Python on a login node for anything other than compiling new Python packages or other non-computational tasks. To use Python interactively for test and debug, use an interactive session on a compute node.
Batch Python
Here is an example batch script. For more information on submitting batch scripts, see the documentation on running jobs.
Example:
For a Python script called hello.py, create a batch script submit.sh containing:
#!/bin/bash #BSUB -W 10 #BSUB -n 1 #BSUB -o out.%J #BSUB -e err.%J module load conda #delete if using system Python 2 python hello.py
Then use
bsub < submit.sh
The above script will request one core for 10 minutes. Check the documentation on running jobs to customize the batch script.
Python from a Conda environment
Creating the environment
To create an environment with a custom version of Python or a specific set of Python libraries (such as pip), see the Conda documentation.
NOTE: If pip is failing to install your Python packages, then delete the pip cache folder at /share/$GROUP/$USER/pip/cache. This is the standard pip path for temporary downloaded cached package installation files (more details here). You may want to double check you have not manually put any important files in this directory before deleting it.
Running a script
To run a script with a custom version of Python or Python libraries, create a batch script that activates the Conda environment.
#!/bin/bash #BSUB -W 10 #BSUB -n 1 #BSUB -o out.%J #BSUB -e err.%J conda activate /path/to/my_env python hello.py conda deactivate
Interactive test and debug
Interactive Compute Node
An interactive session may be used for short debugging or in order to prepare a production batch script. Interactive sessions must be kept to a minimum and only used when necessary. Nodes left idle or underutilized by long running interactive sessions may be terminated.
The following requests an interactive session with 1 core for 10 minutes and opens a prompt in Python 3.
bsub -Is -n 1 -W 10 bash hostname #This should not be a login node! module load conda python
Running distributed memory (MPI) Python
Most packages installed with Conda function with shared memory only, and these jobs must be submitted with #BSUB -R "span[hosts=1]". See the video on Parallel Jobs for more information about shared vs distributed memory.
The mpi4py package allows users to parallelize their scripts with MPI. Packages installed with Conda will not function properly unless they are installed with the system MPI. See the following for instructions on installing mpi4py.
Hello World with mpi4py
A simple Conda environment can be created to do this test. See the documentation for initializing Conda if it was not yet initialized.
conda create --prefix /share/$GROUP/$USER/env_mpi4py pip psutilThen, as per the instructions on installing mpi4py, do
conda activate /share/$GROUP/$USER/env_mpi4py module load PrgEnv-intel pip install mpi4py
This example uses mpi4py to say hello from each of 40 cores; the purpose is to write the host names and core numbers to check if Python is spawning workers correctly. If this is not done correctly, all of the workers may end up on one node. Submit using bsub < submit.sh. There were no errors.
Last modified: February 03 2026 00:04:18.