Introduction#
Welcome to our tutorial series on utilizing the ETX4VELO repository and its associated packages.
As we begin, please keep in mind that my expertise is not in professional software development! Consequently, you might encounter areas within the repository where object definitions and overall structure could be improved.
Setting up#
To follow this series of tutorial, it is more convenient to have access to an NVIDIA GPU.
Additionally, a UNIX-like operating system is recommended to follow along with the setup and examples provided.
Check your CUDA drivers#
Start by ensuring that your system has the CUDA drivers installed.
You can verify this by running the nvidia-smi command in your terminal:
$ nvidia-smi
Mon Feb 5 21:37:10 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05 Driver Version: 535.154.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A100 80GB PCIe On | 00000000:E3:00.0 Off | On |
| N/A 38C P0 66W / 300W | 381MiB / 81920MiB | N/A Default |
| | | Enabled |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| MIG devices: |
+------------------+--------------------------------+-----------+-----------------------+
| GPU GI CI MIG | Memory-Usage | Vol| Shared |
| ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG |
| | | ECC| |
|==================+================================+===========+=======================|
| 0 1 0 0 | 344MiB / 40192MiB | 42 0 | 3 0 2 0 0 |
| | 2MiB / 65535MiB | | |
+------------------+--------------------------------+-----------+-----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 1 0 46496 C ...aforge/envs/etx4velo_new/bin/python 298MiB |
+---------------------------------------------------------------------------------------
The output should indicate a CUDA version of 11.8 or higher, which is necessary for compatibility with the latest PyTorch releases.
Note that it’s not required to manually install the CUDA toolkit as it will be included in the packages installed later on.
Install mamba#
Note
If you already have mamba, you can skip this section.
We recommend using mamba for package management due to its significantly faster performance compared to conda.
Note
If conda is already installed, you have the option to
install mamba in your base environment.
However, this approach is not generally recommended.
If you prefer, you can continue using conda, though mamba offers a more efficient experience!
As of September 2023, mamba is automatically included with miniforge.
Follow the steps below
to download and install the latest version of Miniforge, which bundles mamba:
curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh
During installation, you’ll be prompted to update your shell profile
to automatically initialize conda.
Accepting this option will add the necessary initialization commands to your ~/.bashrc file:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/{yourusername}/miniforge3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/home/{yourusername}/miniforge3/etc/profile.d/conda.sh" ]; then
. "/home/{yourusername}/miniforge3/etc/profile.d/conda.sh"
else
export PATH="/home/{yourusername}/miniforge3/bin:$PATH"
fi
fi
unset __conda_setup
if [ -f "/home/{yourusername}/miniforge3/etc/profile.d/mamba.sh" ]; then
. "/home/{yourusername}/miniforge3/etc/profile.d/mamba.sh"
fi
# <<< conda initialize <<<
(where /home/{yourusername}/miniforge3 is the path to your miniforge installation).
You may reopen your terminal for the changes to take effect.
You can check that conda is installed by executing
$ conda
usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
conda is a tool for managing and deploying applications, environments and packages.
options:
-h, --help Show this help message and exit.
-v, --verbose Can be used multiple times. Once for detailed output, twice for INFO logging,
thrice for DEBUG logging, four times for TRACE logging.
--no-plugins Disable all plugins that are not built into conda.
-V, --version Show the conda version number and exit.
commands:
The following built-in and plugins subcommands are available.
COMMAND
activate Activate a conda environment.
clean Remove unused packages and caches.
compare Compare packages between conda environments.
config Modify configuration values in .condarc.
create Create a new conda environment from a list of specified packages.
deactivate Deactivate the current active conda environment.
doctor Display a health report for your environment.
env See `conda env --help`.
info Display information about current conda install.
init Initialize conda for shell interaction.
install Install a list of packages into a specified conda environment.
list List installed packages in a conda environment.
notices Retrieve latest channel notifications.
package Create low-level conda packages. (EXPERIMENTAL)
remove (uninstall)
Remove a list of packages from a specified conda environment.
rename Rename an existing environment.
repoquery Advanced search for repodata.
run Run an executable in a conda environment.
search Search for packages and display associated information using the MatchSpec
format.
update (upgrade) Update conda packages to the latest compatible version.
Clone the ETX4VELO repository#
Clone the ETX4VELO repository
and switch to the dev branch.
git clone ssh://git@gitlab.cern.ch:7999/gdl4hep/etx4velo.git
cd etx4velo
git checkout dev
git submodule update --init # Fetch the `montetracko` git submodule
Set up the conda environment#
The configuration for our project environment is stored within the setup/ directory
of the ETX4VELO repository.
To create and install the environment, navigate to ETX4VELO’s root directory and execute the following command:
mamba create -f setup/gpu_environment_cuda118.yaml
If you get an error such as The following package could not be installed,
try to replace mamba create by mamba env create.
We encourage you to review the setup/gpu_environment_cuda118.yaml environment file
to understand the purpose behind each package included in this environment.
To check the environment is working as expected, launch a Python shell and execute the following commands to ensure that the critical libraries, especially those requiring CUDA, are correctly loaded and functioning.
$ python3
Python 3.10.13 | packaged by conda-forge | (main, Oct 26 2023, 18:07:37) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import faiss # check that CUDA-enabled libraries are loading correctly
>>> import cudf # check that CUDA-enabled libraries are loading correctly
>>> torch.cuda.is_available() # Confirm PyTorch recognizes your CUDA device
True
To start using the environment, activate it with the following command:
conda activate etx4velo_env
Optionally, activate the jupyter widgets
jupyter nbextension enable --py widgetsnbextension
Note
For those without access to an NVIDIA GPU, a CPU-only version of the environment is available. You can create this alternative environment using:
mamba create -f setup/gpu_environment_cpu.yaml
Download Data#
The necessary data for this tutorial series is stored on the LHCb EOS space, at
/eos/lhcb/user/a/anthonyc/tracking/data
We will focus on downloading the following subsets:
Training data:
/eos/lhcb/user/a/anthonyc/tracking/data/csv/v2.4/minbias-sim10b-xdigi_subsetTest data:
/eos/lhcb/user/a/anthonyc/tracking/data/csv/v2.4/minbias-sim10b-xdigi_subset
For those with access to LXPLUS, where the EOS system is mounted,
you can download the data directly to your desired directory using rsync (or scp):
# Navigate to your data directory
cd <datadir>
# Download the training data subset
rsync -arv <username>@lxplus.cern.ch:/eos/lhcb/user/a/anthonyc/tracking/data/csv/v2.4/minbias-sim10b-xdigi_subset .
# Download and extract the test samples
rsync -arv --progress <username>@lxplus.cern.ch:/eos/lhcb/user/a/anthonyc/tracking/data/data_validation/v2.4/reference_samples_tutorial.tar.lz4 .
lz4 -d reference_samples_tutorial.tar.lz4 -c | tar xvf - # untar archive
mv reference_samples_tutorial reference_samples # rename
rm reference_samples_tutorial.tar.lz4 # delete archive
Replace <datadir> with your target directory path and <username> with your LXPLUS username.
Note
Using the XRootD client provides an efficient alternative for downloading files from EOS. This method requires the installation of both the Kerberos client and the XRootD client on your machine.
For detailed instructions on setting up and using the XRootD client, refer to this guide.