Introduction

Anaconda3 is a popular data science platform used in various fields. Conda is Anaconda3’s package, dependency, and environment management solution. Initially designed for Python, Conda can easily handle nearly any programming language. In addition, it takes care of the tedious tasks related to packages and their dependencies so that you can focus on your work. In this guide, you will learn how to use Conda for your projects on the cluster.

Loading Anaconda3

Anaconda3 is a loadable module on the cluster. As of this writing, five versions of Anaconda3 are installed on the cluster. Table 2.1 lists these versions. To verify which versions are installed on the cluster, execute module avail anaconda3. If you need help determining which distribution and version to use, review the documentation for the software package(s) you require. They should indicate which distribution and version you should use.

To load the appropriate Anaconda3 module, execute module load <anaconda3-version>. Replace <anaconda3-version> with the necessary distribution and version number. For example, to load Anaconda3 version 2021.05, execute module load anaconda3/2021.05 (Figure 2.1). If you do not require a specific version of Anaconda3, you should execute the same command but exclude the version number. The default version for that Anaconda3 distribution will be loaded. To determine which version is the default, review the output of module avail anaconda3. The term “default” will appear in parentheses next to the module.

module load anaconda3/<anaconda3-version>

Figure 2.1 – Load Anaconda3

To configure your shell to use Conda, execute Conda’s setup script. An environment variable defines the path to this script. The name of the environment variable ANACONDA3_SH for the Anaconda 3 distribution. Figure 2.2 shows how to execute this script for the Anaconda 3 distribution.

source $ANACONDA3_SH

Figure 2.2 – Executing the Anaconda3 Setup Script

Test the changes to your environment by logging out of the cluster and logging back in. You should be able to activate the base environment with the conda activate command.

Creating Environments

In Anaconda3, an environment is an isolated space where packages and dependencies can be installed without affecting other environments. To create a basic environment with no packages, execute the command shown in Figure 3.1.

conda create -p <env-dir-name>

Figure 3.1 – Creating an Environment in Anaconda3

Replace <env-dir-name> with a pathname. For instance, to create an environment in a directory named basic_env, you would execute conda create -p ~/basic_env. Always verify that the directory you specify is the correct one before you create the environment. Only create Conda environments in directories to which you have access such as your home directory or Lustre scratch space.

Conda permits you to install packages when you create an environment. To do this, specify the packages after the environment’s path. Consider a new environment in the py3_env directory that should have Python 3.6, NumPy, and SciPy installed. In this case, you would execute the command displayed in Figure 3.2. All three packages would be installed as part of the environment’s creation.

conda create -p ~/py3_env numpy scipy python=3.6

Figure 3.2 – Creating an Anaconda3 Environment with Specific Packages

There is no limit on the number of packages you may install when an environment is created; however, remember that your home directory is limited to 10GB of storage space. If you require more than 10GB of storage for your environment, create the environment in Lustre space with the command presented in Figure 3.3.

conda create -p $SCRATCHDIR/<env-dir-name>

Figure 3.3 – Creating an Anaconda3 Environment in Lustre Space

Do be mindful of the 30-day purge policy on Lustre. For further considerations relevant to filesystems, please review the File Systems document.

With an environment successfully created, you may activate it with the conda activate command. Execute conda activate <path-to-env> to make that environment your active one. Replace <path-to-env> with the path to the environment. Environments in your home directory can be activated by using the tilde (~) character followed by the name of the directory in which the environment is installed. If you created an environment in Lustre space, execute conda activate $SCRATCHDIR/<env-dir-name>.

Environment Management

It is likely that after you create an environment, you will modify it as new and existing packages are no longer necessary. It may also become necessary to remove an existing environment to make room for a new one. Conda makes all these tasks simple and easy to manage.

To install new packages, you should search for the package you wish to install. The conda search command allows you to search for packages in the Anaconda3 repository. Figure 4.1 shows the syntax for this command.

conda search <package-name>

Figure 4.1 – Searching for Packages in Anaconda3

Take the ipython package, an interactive Python shell that provides several helpful features. You would first execute conda search ipython to ensure that the package exists in the Anaconda3 repositories.

Depending on the package, you may see extensive output because it has multiple versions. In ipython’s case, the Anaconda3 repository lists 148 different versions of the package. To narrow down the results to only version seven, execute conda search ipython=7. This output lists 84 ipython versions, which is easier to review.

Once you identify the correct package and version, use the command presented in Figure 4.2. It will download and configure the specified package.

conda install -p <path-to-env> <package-name>

Figure 4.2 – Installing Packages in an Anaconda3 Environment

Replace the <path-to-env> with the absolute path to the environment. Conda will also determine any dependencies and install them with the package. Note that Conda will prompt you to confirm the installation. If you are absolutely certain that you are installing the correct package and its dependencies, execute conda install -y -p <env-dir-name> <package-name>. This will override the confirmation prompt during the installation. Be aware that you may specify the version of the package you wish to install. Using ipython as an example, you could install ipython version 7.0.1 with the conda install -p <path-to-env> ipython=7.0.1 command.

To remove old packages, you should first review the list of existing packages in your environment. Figure 4.3 shows the command to use for this task. It outputs the names, versions, builds, and channels of every package in your specified environment.

conda list -p <path-to-env>

Figure 4.3 – Reviewing the Packages in an Anaconda3 Environment

If you are in the environment from which you plan to remove packages, execute conda list without any additional options or arguments.

After determining which packages to remove, execute the command displayed in Figure 4.4. Conda will prompt you to confirm the removal unless you provide the -y flag to the conda remove command. Only use the -y flag if you are absolutely sure you wish to remove the specified package from your environment.

conda remove -p <path-to-env> <package-name>

Figure 4.4 – Removing a Package from an Anaconda3 Environment

Updates may need to be applied to the packages within an environment. Generally, this will not be necessary and, in certain circumstances, could be destructive. Carefully consider if an update is essential to your work before attempting one. If you wish to perform an update on the environment, use the command shown in Figure 4.5. Every package in the environment will be updated to the latest version. To update a specific package, use conda update -p <path-to-env> <package-name>. In either case, you will be prompted to permit the update unless you provide the -y option.

conda update -p <path-to-env> --all

Figure 4.5 – Updating a Package in an Anaconda3 Environment

If you wish to switch from one environment to another, you first deactivate your existing environment, then activate the one you wish to use. The conda deactivate command will gracefully shut down your active environment and return you to a standard terminal prompt. Alternatively, you may use conda activate with no arguments to return to the base environment, then switch to the environment you wish to use.

To completely remove an environment from your storage space, you should first verify you are removing the correct one. Execute the conda info --envs command to determine which environments belong to you. Because all Conda environments on the cluster should be created with the -p option, you will only see the pathnames of the environments. Figure 4.6 shows an example of two environments that only display pathnames.

[<netid>@login2 ~]$ module unload PE-intel
[<netid>@login2 ~]$ module load anaconda3/2021.05 
Anaconda3 version 2021.05
[<netid>@login2 ~]$ module list
Currently Loaded Modulefiles:
1) anaconda3/2021.05(default)  
[<netid>@login2 ~]$ which conda 
/sw/isaac/applications/anaconda3/2021.05/rhel8_gcc10.2.0/anaconda3-2021.05/bin/conda
[<netid>@login2 ~]$ conda info --envs
# conda environments:
#
                         /lustre/isaac/scratch/<netid>/testsw/conda_env/Julia_env
                         /lustre/isaac/scratch/<netid>/testsw/conda_env/ml4s_test
                         /lustre/isaac/scratch/<netid>/testsw/env_anaconda3_2021.05/IJulia
                         /lustre/isaac/scratch/<netid>/testsw/env_anaconda3_2021.05/mpi4py
base                  *  /sw/isaac/applications/anaconda3/2021.05/rhel8_gcc10.2.0/anaconda3-2021.05
                         /sw/isaac/applications/anaconda3/2021.11/rhel8_cascadelake_intel2020.4.304/anaconda3-2021.11
                         /sw/isaac/applications/psi4/1.6.1/rhel8_cascadelake_conda_binary/psi4conda
                         /sw/isaac/compilers/intel/oneAPI_2021.2.0/intelpython/latest
                         /sw/isaac/compilers/intel/oneAPI_2021.2.0/intelpython/latest/envs/2021.2.0
                         /sw/isaac/compilers/intel/parallel_studio_xe_2020_update4/intelpython3

Figure 4.6 – Output of conda info –envs

Identify the environment you wish to remove, then execute the command displayed in Figure 4.7. Every package in the environment will be removed, then the environment itself will be deleted. The directory in which the environment resides will also be deleted.

conda remove -p <path-to-env> --all

Figure 4.7 – Deleting an Anaconda3 Environment

Anaconda3 Troubleshooting

Issues with Conda generally involve environment- and package-related issues These can usually be remedied without assistance. Always double-check the spelling, capitalization, and punctuation of any names and paths. It is also important to review the documentation for the package(s) you use to understand its requirements. Other issues may require additional investigation; in these situations, use the relevant conda command with the –help option. For example, to review the options available to the conda install command, execute conda install –help. If you cannot remedy the issue using the steps outlined in this guide, please contact the OIT Help Desk (see https://help.utk.edu).

Loading the base Environment

In some cases, Conda may load the global base environment when you first log in to the cluster. Unless you switch to another environment with the conda activate command, you will be unable to install, remove, or modify packages. Please note that if you choose not to initialize Conda when you log in automatically, it is unlikely you’ll encounter this situation.

If you encounter this issue, open your ~/.bashrc file. Search for the line that reads “conda activate”. Using your preferred text editor, insert a hashtag (#) before this line. Once you insert this hashtag, save the file. Next time you log in to the cluster, Conda should be available to you without the base environment being automatically loaded.

Moving Environments

If you need to move an environment from one location to another, Conda provides the –clone option for the conda create command. Do not attempt to use the mv or cp commands on a Conda environment. If you do, it will no longer be recognized by Conda. To use the –clone option, specify the environment you wish to copy. For instance, to clone an existing environment named matplot_env in your home directory and place it in Lustre scratch space, use the command displayed in Figure 5.1. Executing this would copy the environment from your home directory and place it in your Lustre scratch space. It would then appear in the output of conda info --envs.

conda create -p $SCRATCHDIR/matplot_env --clone ~/matplot_env

Figure 5.1 – Cloning an Anaconda3 Environment

Using pip and Conda

Conda and pip both serve the same purpose: package and dependency management. Therefore, using these two tools together can create conflicts. It is best to use these tools separately and not combine them. Please consult our guide on pip and virtualenv for more information. If you do use Conda and pip together, install as many packages as possible with Conda before using pip.