Installation¶

We describe here how the analysis pipeline can be installed. The pipeline must be available from a git repository and follows the expected organization (see Test geniac on the geniac-demo pipeline). An example of installation is provided with the geniac demo pipeline.

Installation requires Cmake (version 3.0 or above) and consists of the following sequence.

# Create the geniac conda environment (use conda >= 4.12.0)
export GENIAC_CONDA="https://raw.githubusercontent.com/bioinfo-pf-curie/geniac/release/environment.yml"
wget ${GENIAC_CONDA}
conda env create -f environment.yml
conda activate geniac

# Prepare the working directory for the use case
export WORK_DIR="${HOME}/tmp/myPipeline"
export SRC_DIR="${WORK_DIR}/src"
export INSTALL_DIR="${WORK_DIR}/install"
export BUILD_DIR="${WORK_DIR}/build"
export GIT_URL="https://github.com/bioinfo-pf-curie/geniac-demo.git"

mkdir -p ${INSTALL_DIR} ${BUILD_DIR}

# clone the repository
# the option --recursive is needed if you use geniac as a submodule
# the option --remote-submodules will pull the last geniac version
# using the release branch from https://github.com/bioinfo-pf-curie/geniac
git clone --remote-submodules --recursive ${GIT_URL} ${SRC_DIR}

cd ${BUILD_DIR}

# configure the pipeline
cmake ${SRC_DIR}/geniac -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR}

# build the files needed by the pipeline
make

# install the pipeline
make install

Note

If you use a Linux distribution which is binary-compatible with the Red Hat Enterprise Linux (RHEL) operating system source code (such as AlmaLinux, CentOS or Rocky Linux), Cmake version 3 is available as cmake3 executable. You can alias cmake3 as cmake in your .bashrc if needed.

Different options can be passed to Cmake for the configuration step. They are described in the following section.

Configure¶

List of options¶

The configure options for the analysis pipeline start with the prefix ap and are in lower case. Options in upper case are Cmake language variables.

CMAKE_INSTALL_PREFIX¶

This is the Cmake variable to set the install directory.

ap_annotation_path¶

STRING
Path to the annotations. A symlink annotations with the given target will be created in the install directory. This is useful if the annotations are already available.
Default is empty.

ap_check_config_file_from_source¶

BOOL
Check if the nextflow config files generated by geniac are present in the source directory. If this is the case, they are compared with those automatically generated by geniac during the build process. If one is different an ERROR is thrown.
Default is ON.

ap_conda_release¶

STRING
When building the docker/singularity images, geniac bootstraps from docker containers available on the docker hub registry 4geniac. When a tool is installed with Conda, the container obviously needs Conda. Therefore, this variable defines which Conda release to use from 4geniac. For details, about the docker containers see 4geniac on GitHub.
Default is py311_23.10.0-1. See also options ap_linux_distro and ap_docker_registry.

ap_docker_registry¶

STRING
Docker registry used to build the containers from recipes automatically generated by geniac.
Default is 4geniac available on the official docker hub registry (see also 4geniac on GitHub). See also options ap_linux_distro and ap_conda_release.

ap_install_docker_images¶

BOOL
Generate and install Dockerfiles and images for docker if set to ON.
Default is OFF.

ap_install_docker_recipes¶

BOOL
Generate and install Dockerfiles for docker if set to ON.
Default is OFF.

ap_install_podman_images¶

BOOL
Generate and install Dockerfiles and images for podman if set to ON.
Default is OFF.

ap_install_podman_recipes¶

BOOL
Generate and install Dockerfiles for podman if set to ON.
Default is OFF.

ap_install_singularity_images¶

BOOL
Generate and install Singularity def files and images if set to ON.
Default is OFF.

ap_install_singularity_recipes¶

BOOL
Generate and install singularity def files if set to ON.
Default is OFF.

ap_keep_envyml_from_source¶

BOOL
Keep environment.yml file from the source code if set to ON. The environment.yml generated by geniac will be discarded. Indeed, it can be impossible to automatically generate the conda recipe. We may have no choice but to write the environment.yml manually.
Default is OFF.

ap_linux_distro¶

STRING
When building the docker/singularity images, geniac bootstraps from docker containers available on the docker hub registry 4geniac. This variable defines which Linux distro (i.e. which repository) and which version (i.e. which tag) to use from 4geniac. For details, about the docker containers see 4geniac on GitHub.
Default is almalinux:9.3. See also options ap_conda_release and ap_docker_registry.

ap_mount_dir¶

Option is deprecated.

ap_nf_executor¶

STRING
executor used by nextflow (e.g. pbs, slurm, etc.).
Default is pbs.

ap_singularity_image_path¶

STRING
Path to the singularity images. A symlink containers/singularity with the given target will be created in the install directory. This is useful if the singularity containers are already available.
Default is empty.

Warning

Options ap_install_singularity_images and ap_singularity_image_path are exclusive.

ap_singularity_build_options¶

STRING
Allow to pass specific options when building singularity images. (e.g. –fakeroot).
Default is empty.

Warning

Options ap_install_singularity_images and ap_singularity_image_path are exclusive.

Set options in CLI¶

All the options can be set on the command line interface. If your want to install the pipeline in $HOME/myPipeline directory and build and install the singularity images, run:

cd ${BUILD_DIR}
cmake ${SRC_DIR}/geniac -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -Dap_install_singularity_images=ON

Tip

To have all the available options and help, run cmake -LAH ${SRC_DIR}/geniac in the build directory. The different options are displayed in the Cache values section.

Set options with a file¶

The file geniac/install/cmake-init-default.cmake provides a script to set all the available variables during the configuration step. We recommend that you copy this file into geniac/install/cmake-init.cmake, edit it and set the different variables to match your configuration. Then you can configure the project as follows:

cp ${SRC_DIR}/geniac/install/cmake-init-default.cmake ${SRC_DIR}/geniac/install/cmake-init.cmake

# edit the file ${SRC_DIR}/geniac/install/cmake-init.cmake to set your options

cd ${BUILD_DIR}

cmake -C ${SRC_DIR}/geniac/install/cmake-init.cmake ${SRC_DIR}/geniac

Note

If you use a Linux distribution which is binary-compatible with the Red Hat Enterprise Linux (RHEL) operating system source code (such as AlmaLinux, CentOS or Rocky Linux), Cmake version 3 is available as cmake3 executable, the syntax is cmake3 ${SRC_DIR}/geniac -C ${SRC_DIR}/geniac/install/cmake-init.cmake

Containers¶

Warning

In order to build singularity images, root credentials are required:

either type make if you have fakeroot singularity credentials
or sudo make if you have sudo privileges
then make install

In order to build the containers, you can either pass the required options during the configure stage (see Configure) or use custom targets (see Build recipes and containers).

Custom targets¶

The custom targets will be available once you have performed the configuration step with cmake.

Build configuration files¶

Assume you are in the build directory. The following custom target allows you to build only the configuration files:

make build_config_files

The configuration files will be generated in build/workDir/results/conf.

Build recipes and containers¶

Assume you are in the build directory. The following custom targets allows you to build recipes and containers even if you did not ask for them during the configure stage:

make build_singularity_recipes
make build_singularity_images
make build_docker_recipes
make build_docker_images
make build_podman_recipes
make build_podman_images

For singularity:

Recipes will be generated in build/workDir/results/singularity/deffiles.
Images will be generated in build/workDir/results/singularity/images.

For docker:

Recipes will be generated in build/workDir/results/docker/Dockerfiles.
Images will be created in the registry.

Install and test with different profiles¶

In order to make the deployment and testing of the pipeline easier, several custom targets are provided such that you only need to type one of the following command to install the pipeline:

make test_conda
make test_docker
make test_multiconda
make test_multipath
make test_path
make test_singularity
make test_standard

Assuming that you configured the build directory such that CMAKE_INSTALL_PREFIX=${INSTALL_DIR}, typing make test_conda is similar to:

make
make install
cd ${SRC_DIR}/pipeline
nextflow -c conf/test.config run main.nf -profile conda

If you want to add the cluster profile, just type the following:

make test_conda_cluster
make test_docker_cluster
make test_multiconda_cluster
make test_multipath_cluster
make test_path_cluster
make test_singularity_cluster
make test_standard_cluster

Note

For these custom targets to be available, test data and conf/test.config file have to be provided in the repository.

Structure of the installation directory tree¶

├── annotations
├── containers
│   └── singularity
├── multipath
│   ├── alpine
│   │   └── bin
│   ├── fastqc
│   │   └── bin
│   ├── helloWorld
│   │   └── bin
│   ├── rmarkdown
│   │   └── bin
│   └── trickySoftware
│       └── bin
├── path
│   ├── bin
└── pipeline
    ├── assets
    ├── bin
    │   └── fromSource
    ├── conf
    ├── docs
    ├── env
    ├── modules
    │   └── helloWorld
    ├── recipes
    │   ├── conda
    │   ├── dependencies
    │   ├── docker
    │   └── singularity
    └── test
        └── data

Examples¶

Install and run with conda¶

Important

You must have Conda installed locally, if not, proceed as follows:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

Then, edit your file .bashrc and add $HOME/miniconda3/bin (or the install directory you set) in your PATH.

export WORK_DIR="${HOME}/tmp/myPipeline"
export SRC_DIR="${WORK_DIR}/src"
export INSTALL_DIR="${WORK_DIR}/install"
export BUILD_DIR="${WORK_DIR}/build"
export GIT_URL="https://github.com/bioinfo-pf-curie/geniac-demo.git"

mkdir -p ${INSTALL_DIR} ${BUILD_DIR}

# clone the repository
# the option --recursive is needed if you use geniac as a submodule
# the option --remote-submodules will pull the last geniac version
# using the release branch from https://github.com/bioinfo-pf-curie/geniac
git clone --remote-submodules --recursive ${GIT_URL} ${SRC_DIR}

cd ${BUILD_DIR}
cmake ${SRC_DIR}/geniac -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR}
make
make install

cd ${INSTALL_DIR}/pipeline

nextflow -c conf/test.config run main.nf -profile multiconda

Note

If you use both the conda and cluster profile, check that your master job that launches nextflow has been submitted with enough memory, otherwise the creation of the conda environment may fail.

Install and run with singularity¶

export WORK_DIR="${HOME}/tmp/myPipeline"
export SRC_DIR="${WORK_DIR}/src"
export INSTALL_DIR="${WORK_DIR}/install"
export BUILD_DIR="${WORK_DIR}/build"
export GIT_URL="https://github.com/bioinfo-pf-curie/geniac-demo.git"

mkdir -p ${INSTALL_DIR} ${BUILD_DIR}

# clone the repository
# the option --recursive is needed if you use geniac as a submodule
# the option --remote-submodules will pull the last geniac version
# using the release branch from https://github.com/bioinfo-pf-curie/geniac
git clone --remote-submodules --recursive ${GIT_URL} ${SRC_DIR}

cd ${BUILD_DIR}
cmake ${SRC_DIR}/geniac  -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -Dap_install_singularity_images=ON

### /!\ with sudo, the singularity and nextflow commands must be
### /!\ in the secure_path option declared in the file /etc/sudoers

### build the files needed by the pipeline
sudo make

### change file owner/group to the current user
sudo chown -R $(id -gn):$(id -gn) ${BUILD_DIR}

make install

cd ${INSTALL_DIR}/pipeline

nextflow -c conf/test.config run main.nf -profile singularity

Note

Whenever you explicitely set an option on the command line such as -Dap_install_singularity_images=ON, and then you want to reconfigure your build directory by specifying only another option on the command line such as -DCMAKE_INSTALL_PREFIX=${HOME}/myPipelineNewDir, the ap_install_singularity_images will remain ON unless you specify -Dap_install_singularity_images=OFF.

Generate the recipes for the containers¶

As geniac automatically generates the recipes of the containers, they are not available in the git repository. However, they can be easily retrieved in several ways. We provide here an example.

First, let’s assume you want the Singularity definition files. Then, pass the option -Dap_install_singularity_recipes=ON to cmake during the configuration stage:

export WORK_DIR="${HOME}/tmp/myPipeline"
export SRC_DIR="${WORK_DIR}/src"
export INSTALL_DIR="${WORK_DIR}/install"
export BUILD_DIR="${WORK_DIR}/build"
export GIT_URL="https://github.com/bioinfo-pf-curie/geniac-demo.git"

mkdir -p ${INSTALL_DIR} ${BUILD_DIR}

# clone the repository
# the option --recursive is needed if you use geniac as a submodule
# the option --remote-submodules will pull the last geniac version
# using the release branch from https://github.com/bioinfo-pf-curie/geniac
git clone --remote-submodules --recursive ${GIT_URL} ${SRC_DIR}

cd ${BUILD_DIR}
cmake ${SRC_DIR}/geniac  -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -Dap_install_singularity_recipes=ON
make install

### the Singularity definition files are available here:
ls ${INSTALL_DIR}/pipeline/recipes/singularity

Second, let’s assume you want the Dockerfiles. Then, use a custom target (see Build recipes and containers) as follows:

make build_docker_recipes
make install

### the Dockerfiles are available here:
ls ${INSTALL_DIR}/pipeline/recipes/docker

Obviously, if you want both the Singularity definition files and Dockerfiles you can simply do the following during the configuration stage:

cmake ${SRC_DIR}/geniac  -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -Dap_install_singularity_recipes=ON -Dap_install_docker_recipes=ON