Quick Start

Use the Anaconda module to setup a new environment:

module load anaconda3/2024.06 
source $ANACONDA3_SH
conda activate

If you want the environment to be available from the Open OnDemand Jupyter app, you must install at least python and ipykernel, but you can install any other packages you like.

conda create -n TEST python ipykernel

List installed environments to confirm:

conda info --envs

Activate the envirnoment and confirm that python is installed there:

conda activate TEST
which python

Set up the environment for use as a kernel in Jupyter:

python -m ipykernel install --user --name TEST --display-name "Python (TEST)"

Open a new instance of the Jupyter app in Open OnDemand and you select your kernel (e.g., “Python (TEST)”) from the drop down menu.

Introduction

Anaconda3 is a popular data science platform used in various fields. Conda is Anaconda3’s package, dependency, and environment management solution. Initially designed for Python, Conda can easily handle nearly any programming language. In addition, it takes care of the tedious tasks related to packages and their dependencies so that you can focus on your work. In this guide, you will learn how to use Conda for your projects on the cluster.

Loading Anaconda3

Anaconda3 is a loadable module on the cluster. To verify which versions are installed on the cluster, run the command module avail anaconda3. For example, to load Anaconda3 version 2021.05, run the command module load anaconda3/2021.05.

module load anaconda3/<anaconda3-version>

To configure your shell to use Conda, execute Conda’s setup script as defined in the environment variable ANACONDA3_SH:

source $ANACONDA3_SH

Test the changes to your environment by logging out of the cluster and logging back in. You should be able to activate the base environment with the conda activate command.

Creating Environments

In Anaconda3, an environment is an isolated space where packages and dependencies can be installed without affecting other environments. To create a basic environment with no packages, execute the command shown:

conda create -p <env-dir-name>

Replace <env-dir-name> with a pathname. For instance, to create an environment in a directory named basic_env, you would execute conda create -p ~/basic_env. Always verify that the directory you specify is the correct one before you create the environment. Only create Conda environments in directories to which you have access such as your home directory or Lustre scratch space.

Conda permits you to install packages when you create an environment. To do this, specify the packages after the environment’s path. Consider a new environment in the py3_env directory that should have Python 3.6, NumPy, and SciPy installed. In this case, you would execute the command below. All three packages would be installed as part of the environment’s creation.

conda create -p ~/py3_env numpy scipy python=3.6

There is no limit on the number of packages you may install when an environment is created; however, remember that your home directory is limited to 10GB of storage space. If you require more than 10GB of storage for your environment, create the environment in Lustre space with the command presented below.

conda create -p $SCRATCHDIR/<env-dir-name>

Do be mindful of the 30-day purge policy on Lustre. For further considerations relevant to filesystems, please review the File Systems document.

With an environment successfully created, you may activate it with the conda activate command. Execute conda activate <path-to-env> to make that environment your active one. Replace <path-to-env> with the path to the environment. Environments in your home directory can be activated by using the tilde (~) character followed by the name of the directory in which the environment is installed. If you created an environment in Lustre space, execute conda activate $SCRATCHDIR/<env-dir-name>.

Environment Management

It is likely that after you create an environment, you will modify it as new and existing packages are no longer necessary. It may also become necessary to remove an existing environment to make room for a new one. Conda makes all these tasks simple and easy to manage.

To install new packages, you should search for the package you wish to install. The conda search command allows you to search for packages in the Anaconda3 repository.

conda search <package-name>

Take the ipython package, an interactive Python shell that provides several helpful features. You would first execute conda search ipython to ensure that the package exists in the Anaconda3 repositories.

Depending on the package, you may see extensive output because it has multiple versions. In ipython’s case, the Anaconda3 repository lists 148 different versions of the package. To narrow down the results to only version seven, execute conda search ipython=7. This output lists 84 ipython versions, which is easier to review.

Once you identify the correct package and version, use the command below. It will download and configure the specified package.

conda install -p <path-to-env> <package-name>

Replace the <path-to-env> with the absolute path to the environment. Conda will also determine any dependencies and install them with the package. Note that Conda will prompt you to confirm the installation. If you are absolutely certain that you are installing the correct package and its dependencies, execute conda install -y -p <env-dir-name> <package-name>. This will override the confirmation prompt during the installation. Be aware that you may specify the version of the package you wish to install. Using ipython as an example, you could install ipython version 7.0.1 with the conda install -p <path-to-env> ipython=7.0.1 command.

To remove old packages, you should first review the list of existing packages in your environment. It outputs the names, versions, builds, and channels of every package in your specified environment.

conda list -p <path-to-env>

If you are in the environment from which you plan to remove packages, execute conda list without any additional options or arguments.

After determining which packages to remove, execute the command displayed below. Conda will prompt you to confirm the removal unless you provide the -y flag to the conda remove command. Only use the -y flag if you are absolutely sure you wish to remove the specified package from your environment.

conda remove -p <path-to-env> <package-name>

Updates may need to be applied to the packages within an environment. Generally, this will not be necessary and, in certain circumstances, could be destructive. Carefully consider if an update is essential to your work before attempting one. If you wish to perform an update on the environment, use the command shown below. Every package in the environment will be updated to the latest version. To update a specific package, use conda update -p <path-to-env> <package-name>. In either case, you will be prompted to permit the update unless you provide the -y option.

conda update -p <path-to-env> --all

If you wish to switch from one environment to another, you first deactivate your existing environment, then activate the one you wish to use. The conda deactivate command will gracefully shut down your active environment and return you to a standard terminal prompt. Alternatively, you may use conda activate with no arguments to return to the base environment, then switch to the environment you wish to use.

To completely remove an environment from your storage space, you should first verify you are removing the correct one. Execute the conda info --envs command to determine which environments belong to you. Because all Conda environments on the cluster should be created with the -p option, you will only see the pathnames of the environments.

[<netid>@login2 ~]$ module unload PE-intel
[<netid>@login2 ~]$ module load anaconda3/2021.05 
Anaconda3 version 2021.05
[<netid>@login2 ~]$ module list
Currently Loaded Modulefiles:
1) anaconda3/2021.05(default)  
[<netid>@login2 ~]$ which conda 
/sw/isaac/applications/anaconda3/2021.05/rhel8_gcc10.2.0/anaconda3-2021.05/bin/conda
[<netid>@login2 ~]$ conda info --envs
# conda environments:
#
                         /lustre/isaac/scratch/<netid>/testsw/conda_env/Julia_env
                         /lustre/isaac/scratch/<netid>/testsw/conda_env/ml4s_test
                         /lustre/isaac/scratch/<netid>/testsw/env_anaconda3_2021.05/IJulia
                         /lustre/isaac/scratch/<netid>/testsw/env_anaconda3_2021.05/mpi4py
base                  *  /sw/isaac/applications/anaconda3/2021.05/rhel8_gcc10.2.0/anaconda3-2021.05
                         /sw/isaac/applications/anaconda3/2021.11/rhel8_cascadelake_intel2020.4.304/anaconda3-2021.11
                         /sw/isaac/applications/psi4/1.6.1/rhel8_cascadelake_conda_binary/psi4conda
                         /sw/isaac/compilers/intel/oneAPI_2021.2.0/intelpython/latest
                         /sw/isaac/compilers/intel/oneAPI_2021.2.0/intelpython/latest/envs/2021.2.0
                         /sw/isaac/compilers/intel/parallel_studio_xe_2020_update4/intelpython3

Identify the environment you wish to remove, then execute the command displayed below. Every package in the environment will be removed, then the environment itself will be deleted. The directory in which the environment resides will also be deleted.

conda remove -p <path-to-env> --all

Anaconda3 Troubleshooting

Issues with Conda generally involve environment- and package-related issues These can usually be remedied without assistance. Always double-check the spelling, capitalization, and punctuation of any names and paths. It is also important to review the documentation for the package(s) you use to understand its requirements. Other issues may require additional investigation; in these situations, use the relevant conda command with the –help option. For example, to review the options available to the conda install command, execute conda install –help. If you cannot remedy the issue using the steps outlined in this guide, please contact the OIT Help Desk (see https://help.utk.edu).

Loading the base Environment

In some cases, Conda may load the global base environment when you first log in to the cluster. Unless you switch to another environment with the conda activate command, you will be unable to install, remove, or modify packages. Please note that if you choose not to initialize Conda when you log in automatically, it is unlikely you’ll encounter this situation.

If you encounter this issue, open your ~/.bashrc file. Search for the line that reads “conda activate”. Using your preferred text editor, insert a hashtag (#) before this line. Once you insert this hashtag, save the file. Next time you log in to the cluster, Conda should be available to you without the base environment being automatically loaded.

Moving Environments

If you need to move an environment from one location to another, Conda provides the –clone option for the conda create command. Do not attempt to use the mv or cp commands on a Conda environment. If you do, it will no longer be recognized by Conda. To use the –clone option, specify the environment you wish to copy. For instance, to clone an existing environment named matplot_env in your home directory and place it in Lustre scratch space, use the command below. Executing this would copy the environment from your home directory and place it in your Lustre scratch space. It would then appear in the output of conda info --envs.

conda create -p $SCRATCHDIR/matplot_env --clone ~/matplot_env

Using pip and Conda

Conda and pip both serve the same purpose: package and dependency management. Therefore, using these two tools together can create conflicts. It is best to use these tools separately and not combine them. Please consult our guide on pip and virtualenv for more information. If you do use Conda and pip together, install as many packages as possible with Conda before using pip.