Managing Python environments on LMS
- Emma Barnes
Virtual environments are a convenient way for you to have complete control over potentially many versions of Python. The LMS provides the conda utility to allow you to create and manage virtual Python environments. This page describes the basics of using conda, as well as some LMS-specific configuration that you are likely to find useful. The full documentation of conda, can be found in the conda online documentation.
Initial Setup
By default, conda will create environments and install packages into subdirectories of your HOME directory, which will end up containing a vast number of files. To avoid hitting quota limits, you can tell conda to store files in your scratch space instead, which is much larger. To do this, you need to work through these steps.
1. Create directories in your scratch space to contain your Python environments and packages. This can be achieved by running the following commands in a shell (N.B double check this is the correct path to where you wish to store your packages)
cd ~ mkdir -p /scratch/USERNAME/python_environments/envs mkdir -p /scratch/USERNAME/python_environments/pkgs
2. Create a configuration file for conda in your HOME directory called ".condarc" and add the following to tell conda to use the directories you created in step 1. Ensure you replace "USERNAME" in the directory with your IT services username.
envs_dirs: - /scratch/USERNAME/python_environments/envs pkgs_dirs: - /scratch/USERNAME/python_environments/pkgs
3. To use conda commands in the following steps, you will need to load the Miniconda module, which contains the conda utility.
module load lang/Miniconda3
Creating an environment
The most reliable way to create an environment using conda is by using an environment file. This is a .yaml file that describes the Python environment you would like to create. This method allows you to recreate the same environment in multiple places, and easily pass on the specification for a Python environment to other users. A simple example of an environment file is given below.
name: my_first_environment channels: - conda-forge dependencies: - python=3.7 - numpy - ipython - pip: - mido
This file, my_first_environment.yaml describes the following things about a Python environment:
- name: the name of the Python environment. This is the name to use to refer to this environment when using conda tools
- channels: the Anaconda Cloud channels that should be used to find packages for this environment. There are many channels available, but the two most common are defaults, which contains packages curated by the Anaconda team, and conda-forge, a channel lead by the community which contains a range of high-quality packages that are often more up to date than those in defaults.
- dependencies: the dependencies of the Python environment you want to create. In the example, we have specified which version of Python we want to use, some packages to be installed from the conda-forge channel, and a package to be installed through pip, as the package is not available through the conda-forge channel.
More can be found on environment files in the conda user guide.
To create a python environment, you will need to create a yaml file, in either your scratch or HOME directory.
Now you have an environment file, you can create that environment using conda. Using my_first_environment.yaml as an example:
conda env create -f my_first_environment.yaml
Here you are telling conda to create an environment using your environment file as a template. Once the environment has been installed, you can test to see if the environment exists by using the following command:
[abc123@vargpu01 Python]$ conda info --envs # conda environments: # base * /opt/apps/easybuild/software/lang/Miniconda3/4.4.10 my_first_environment /users/klcm500/scratch/python_environments/envs/my_first_environment
This will list all of the environments installed with conda, and in this example we can see that my_first_environment has been successfully created. Note the asterisk highlights the active environment, which is currently the base environment (over which you have no control).
Using an environment
Once an environment has been created, you can activate it with the following command:
[abc123@vargpu0 ~]$ conda activate my_first_environment
You have now activated my_first_environment, which has made all the necessary environment changes. Now you can use Python and all the dependencies included as normal, but with a guarantee of no conflict with other Python installations on the system. Your shell prompt will include the name of the current Python environment in parentheses to remind you that you are in a specific Python environment, as shown below.
If you wish to add more packages to your environment, you can use conda or pip to install them into the environment, although you must activate the environment first. In this example we will install pytest into my_first_environment using conda:
(my_first_environment) [abc123@vargpu0 ~]$ conda install pytest
Solving environment: done
## Package Plan ##
environment location: /users/klcm500/scratch/Python/envs/my_first_environment
added / updated specs:
- pytest
The following packages will be downloaded:
package | build
---------------------------|-----------------
certifi-2019.3.9 | py37_0 155 KB
pluggy-0.9.0 | py37_0 30 KB
attrs-19.1.0 | py_0 35 KB
more-itertools-6.0.0 | py37_0 89 KB
pytest-4.3.0 | py37_0 349 KB
py-1.8.0 | py37_0 140 KB
atomicwrites-1.3.0 | py_0 9 KB
------------------------------------------------------------
Total: 808 KB
The following NEW packages will be INSTALLED:
atomicwrites: 1.3.0-py_0
attrs: 19.1.0-py_0
more-itertools: 6.0.0-py37_0
pluggy: 0.9.0-py37_0
py: 1.8.0-py37_0
pytest: 4.3.0-py37_0
The following packages will be UPDATED:
certifi: 2019.3.9-py37_0 conda-forge --> 2019.3.9-py37_0
openssl: 1.1.1b-h14c3975_1 conda-forge --> 1.1.1b-h7b6447c_1
The following packages will be DOWNGRADED:
ca-certificates: 2019.3.9-hecc5488_0 conda-forge --> 2019.1.23-0
Proceed ([y]/n)? y
Downloading and Extracting Packages
certifi 2019.3.9: ################################################################## | 100%
pluggy 0.9.0: ###################################################################### | 100%
attrs 19.1.0: ###################################################################### | 100%
more-itertools 6.0.0: ############################################################## | 100%
pytest 4.3.0: ###################################################################### | 100%
py 1.8.0: ########################################################################## | 100%
atomicwrites 1.3.0: ################################################################ | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Here conda has downloaded and installed some dependencies for the new package, as well as solved some dependency issues, which caused some of the packages already installed to be downgraded. Once this is complete, you can immediately use the new package in your environment.
If the package you wanted to install is not available through conda, you can use pip install in the same way instead.
Once you are finished using your Python environment, it can easily be exited using the following command:
(my_first_environment) [abc123@vargpu0 ~]$ conda deactivate [abc123@vargpu0 ~]$
Note that the environment name is no longer appearing in brackets after running the conda deactivate command, meaning the environment is no longer active.
To summarise, we can specify and create Python virtual environments with conda, we can switch between them, use them and update them with any necessary new packages.