Module Environment
One compute cluster, many users… That leads necessarily to varying software demands. While my code runs with Python 3.8, yours needs a Python 3.10. While this can lead to a small headache on a local machine, a cluster faces the same problem, just amplified a thousandfold.
Luckily, there is an elegant way to deal with this: the module environment.
The module environment manages many different software version at the same time and makes them only available after you load them.
You can pick an choose the software that is visible to you (and your program).
A different version of Python, of gcc or openMPI, you got it!
TL;DR
Use module avail to check the software that is available.
If you see something that you like, load it with module load <name>.
The module command
To create a certain software environment, you have to load a module that contains the necessary module.
The module environment changes the environment variables such that the software gets accessible.
Quick reminder: environment variables are global variables that influence the behavior of programs.
A famous example is the PATH variable.
All folders that are listed when you execute echo $PATH will be searched for a program that you are trying to execute.
The folders are traversed in order of appearence. Thus, if you prepend a folder to PATH with
export PATH=/some/funny/dir:$PATH
the folder /some/funny/dir will be searched first for the program.
A word of warning: if you prepend folders that have executable files named ls or echo (or any other commonly used bash program), you are in for a ride!
Back to the module command.
Like git, module works with one main command (module) and many sub-commands.
The most important module commands are:
module availablemodule listmodule load <name>module unload <name>module purgemodule show <name>
The available subcommands prints all available modules to the terminal.
It can be shortened to module avail (or any other shorthand) as long as the command can be identified among the other subcommands.
A typical output of that command could look like
----------------------------------- /easybuild/modules/all ---------------------------------------------------------------
ATK/2.38.0-GCCcore-11.3.0 OpenBLAS/0.3.23-GCC-12.3.0
ATK/2.38.0-GCCcore-12.2.0 OpenBLAS/0.3.24-GCC-13.2.0 (D)
ATK/2.38.0-GCCcore-12.3.0 (D) OpenEXR/3.1.5-GCCcore-12.2.0
Abseil/20230125.2-GCCcore-12.2.0 OpenEXR/3.1.7-GCCcore-12.3.0 (D)
Abseil/20230125.3-GCCcore-12.3.0 (D) OpenJPEG/2.5.0-GCCcore-11.3.0
Armadillo/11.4.3-foss-2022b OpenJPEG/2.5.0-GCCcore-12.2.0
Armadillo/12.6.2-foss-2023a (D) OpenJPEG/2.5.0-GCCcore-12.3.0 (D)
Arrow/8.0.0-foss-2022a OpenMM/8.0.0-foss-2022a-CUDA-11.7.0
Autoconf/2.69-GCCcore-9.3.0 OpenMPI/4.1.4-GCC-11.3.0
Autoconf/2.71-GCCcore-11.3.0 OpenMPI/4.1.4-GCC-12.2.0
Autoconf/2.71-GCCcore-12.2.0 OpenMPI/4.1.5-GCC-12.3.0
Autoconf/2.71-GCCcore-12.3.0 OpenMPI/4.1.6-GCC-13.2.0 (D)
Autoconf/2.71-GCCcore-13.2.0 OpenPGM/5.2.122-GCCcore-11.3.0
Autoconf/2.71 (D) OpenPGM/5.2.122-GCCcore-12.2.0
Automake/1.16.1-GCCcore-9.3.0 OpenPGM/5.2.122-GCCcore-12.3.0
Automake/1.16.5-GCCcore-11.3.0 OpenPGM/5.2.122-GCCcore-13.2.0 (D)
Automake/1.16.5-GCCcore-12.2.0 OpenSSL/1.1 (L)
Automake/1.16.5-GCCcore-12.3.0 OrthoFinder/2.5.5-foss-2023a
Automake/1.16.5-GCCcore-13.2.0 Osi/0.108.9-GCC-12.3.0
Automake/1.16.5 (D) PCRE/8.45-GCCcore-11.3.0
Autotools/20180311-GCCcore-9.3.0 PCRE/8.45-GCCcore-12.2.0
Autotools/20220317-GCCcore-11.3.0 PCRE/8.45-GCCcore-12.3.0 (D)
Autotools/20220317-GCCcore-12.2.0 PCRE2/10.40-GCCcore-11.3.0
Autotools/20220317-GCCcore-12.3.0 PCRE2/10.40-GCCcore-12.2.0
Autotools/20220317-GCCcore-13.2.0 PCRE2/10.42-GCCcore-12.3.0 (D)
Autotools/20220317 (D) PETSc/3.20.3-foss-2023a
BCFtools/1.15.1-GCC-11.3.0 PLUMED/2.9.0-foss-2023a
BCFtools/1.18-GCC-12.3.0 (D) PMIx/4.1.2-GCCcore-11.3.0
BEDTools/2.31.0-GCC-12.3.0 PMIx/4.2.2-GCCcore-12.2.0
BLAST+/2.13.0-gompi-2022a PMIx/4.2.4-GCCcore-12.3.0
BLAST+/2.14.1-gompi-2023a (D) PMIx/4.2.6-GCCcore-13.2.0 (D)
The (D) identifies the default package in case you load the module without an explicit version (the number after the slash).
If you find a module that looks interesting to you, say foo, you can load it via module load foo.
If the foo module has different versions, module load foo will give you the default version, whereas module load foo/4.2.0 will load version 4.2.0.
To check fo available versions, you can use module spider foo.
module list prints all currently loaded modules to the terminal.
If you don’t need a module anymore, you can unload it via module unload <name>.
In case you want to unload all modules, use module purge.
Be careful with this command, since it also purges all the modules which might have been pre-loaded automatically by the cluster admins on starting up the session.
Thus, if you really want to clear all modules, be sure about what you are doing.
It might be that your job-submission system, e.g. SLURM, does not work as expected anymore afterwards.
Finally, module show <name> prints more details about a certain module.
Usually, it contains information about the paths which will be pre-prended to the PATH variable, etc.
As an example, this could be the result of module show gcc:
------
/foo/local/modulefiles/gcc/11.2.0:
------
whatis("adds GNU Cross Compilers to your environment variables ")
prepend_path("PATH","/cm/local/apps/gcc/11.2.0/bin")
prepend_path("LD_LIBRARY_PATH","/cm/local/apps/gcc/11.2.0/lib:/cm/local/apps/gcc/11.2.0/lib64:/cm/local/apps/gcc/11.2.0/lib32")
help([[ Adds GNU Cross Compilers to your environment variables,
]])
Here, the PATH and the LD_LIBRARY_PATH are adapted, such that gcc is used for complication and linking.
Patrick Emonts
Postdoctoral Researcher
My research interests include tensor networks, lattice gauge theories and quantum information.