.. include:: substitutions.rst .. _install-page: ************ Installation ************ We describe here how the analysis pipeline can be installed. The pipeline must be available from a git repository and follows the expected organization (see :ref:`overview-source-tree`). An example of installation is provided with the |geniacdemo|_ pipeline. Installation requires |cmake|_ (version 3.0 or above) and consists of the following sequence. :: # Create the geniac conda environment (use conda >= 4.12.0) export GENIAC_CONDA="https://raw.githubusercontent.com/bioinfo-pf-curie/geniac/release/environment.yml" wget ${GENIAC_CONDA} conda env create -f environment.yml conda activate geniac pip install geniac # Prepare the working directory for the use case export WORK_DIR="${HOME}/tmp/myPipeline" export SRC_DIR="${WORK_DIR}/src" export INSTALL_DIR="${WORK_DIR}/install" export BUILD_DIR="${WORK_DIR}/build" export GIT_URL="https://github.com/bioinfo-pf-curie/geniac-demo.git" mkdir -p ${INSTALL_DIR} ${BUILD_DIR} # clone the repository # the option --recursive is needed if you use geniac as a submodule # the option --remote-submodules will pull the last geniac version # using the release branch from https://github.com/bioinfo-pf-curie/geniac git clone --remote-submodules --recursive ${GIT_URL} ${SRC_DIR} cd ${BUILD_DIR} # configure the pipeline cmake ${SRC_DIR}/geniac -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} # build the files needed by the pipeline make # install the pipeline make install .. note:: If you use a Linux distribution which is binary-compatible with the Red Hat Enterprise Linux (RHEL) operating system source code (such as |almalinux|_, |centos|_ or |rockylinux|_), |cmake|_ version 3 is available as ``cmake3`` executable. You can alias ``cmake3`` as ``cmake`` in your ``.bashrc`` if needed. Different options can be passed to |cmake|_ for the configuration step. They are described in the following section. .. _install-configure: Configure ========= List of options --------------- The configure options for the **a**\nalysis **p**\ipeline start with the prefix **ap** and are in lower case. Options in upper case are |cmake|_ language variables. CMAKE_INSTALL_PREFIX ++++++++++++++++++++ This is the |cmake|_ variable to set the install directory. ap_annotation_path ++++++++++++++++++ | STRING | Path to the annotations. A symlink ``annotations`` with the given target will be created in the install directory. This is useful if the annotations are already available. | Default is empty. ap_check_config_file_from_source ++++++++++++++++++++++++++++++++ | BOOL | Check if the nextflow config files generated by geniac are present in the source directory. If this is the case, they are compared with those automatically generated by geniac during the build process. If one is different an ERROR is thrown. | Default is ON. .. _install-ap_conda_release: ap_conda_release ++++++++++++++++ | STRING | When building the docker/singularity images, geniac bootstraps from docker containers available on the docker hub registry |4geniac|_. When a tool is installed with Conda, the container obviously needs Conda. Therefore, this variable defines which Conda release to use from |4geniac|_. For details, about the docker containers see |4geniacgithub|_. | Default is ``py312_25.1.1-2``. See also options :ref:`install-ap_linux_distro` and :ref:`install-ap_docker_registry`. ap_container_list +++++++++++++++++ | STRING | Provide PATH to a text file which contains the list of tool labels for which the containers will be built. This makes it possible to build only a subset of containers instead of building all the containers for all the tools. | Default is none. .. _install-ap_docker_registry: ap_docker_registry ++++++++++++++++++ | STRING | Docker registry used to build the containers from recipes automatically generated by geniac. | Default is |4geniac|_ available on the official docker hub registry (see also |4geniacgithub|_). See also options :ref:`install-ap_linux_distro` and :ref:`install-ap_conda_release`. ap_install_docker_images ++++++++++++++++++++++++ | BOOL | Generate and install Dockerfiles and images for docker if set to ON. | Default is OFF. ap_install_docker_recipes +++++++++++++++++++++++++ | BOOL | Generate and install Dockerfiles for docker if set to ON. | Default is OFF. ap_install_podman_images ++++++++++++++++++++++++ | BOOL | Generate and install Dockerfiles and images for podman if set to ON. | Default is OFF. ap_install_podman_recipes +++++++++++++++++++++++++ | BOOL | Generate and install Dockerfiles for podman if set to ON. | Default is OFF. ap_install_singularity_images +++++++++++++++++++++++++++++ | BOOL | Generate and install Singularity def files and images if set to ON. | Default is OFF. ap_install_singularity_recipes ++++++++++++++++++++++++++++++ | BOOL | Generate and install singularity def files if set to ON. | Default is OFF. ap_keep_envyml_from_source +++++++++++++++++++++++++++ | BOOL | Keep ``environment.yml`` file from the source code if set to ON. The ``environment.yml`` generated by geniac will be discarded. Indeed, it can be impossible to automatically generate the conda recipe. We may have no choice but to write the ``environment.yml`` manually. | Default is OFF. .. _install-ap_linux_distro: ap_linux_distro +++++++++++++++ | STRING | When building the docker/singularity images, geniac bootstraps from docker containers available on the docker hub registry |4geniac|_. This variable defines which Linux distro (i.e. which repository) and which version (i.e. which tag) to use from |4geniac|_. For details, about the docker containers see |4geniacgithub|_. | Default is ``almalinux:9.5``. See also options :ref:`install-ap_conda_release` and :ref:`install-ap_docker_registry`. .. _install-ap_mount_dir: ap_mount_dir ++++++++++++ | Option is deprecated. ap_nf_executor ++++++++++++++ | STRING | `executor `_ used by nextflow (e.g. pbs, slurm, etc.). | Default is pbs. ap_singularity_image_path +++++++++++++++++++++++++ | STRING | Path to the singularity images. A symlink ``containers/singularity`` with the given target will be created in the install directory. This is useful if the singularity containers are already available. | Default is empty. .. warning:: Options ``ap_install_singularity_images`` and ``ap_singularity_image_path`` are exclusive. ap_singularity_build_options ++++++++++++++++++++++++++++ | STRING | Allow to pass specific options when building singularity images. (e.g. --fakeroot). | Default is empty. .. warning:: Options ``ap_install_singularity_images`` and ``ap_singularity_image_path`` are exclusive. Set options in CLI ------------------ All the options can be set on the command line interface. If your want to install the pipeline in ``$HOME/myPipeline`` directory and build and install the singularity images, run: :: cd ${BUILD_DIR} cmake ${SRC_DIR}/geniac -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -Dap_install_singularity_images=ON .. tip:: To have all the available options and help, run ``cmake -LAH ${SRC_DIR}/geniac`` in the ``build`` directory. The different options are displayed in the **Cache values** section. .. _install-configure-file: Set options with a file ----------------------- The file ``geniac/install/cmake-init-default.cmake`` provides a script to set all the available variables during the configuration step. We recommend that you copy this file into ``geniac/install/cmake-init.cmake``, edit it and set the different variables to match your configuration. Then you can configure the project as follows: :: cp ${SRC_DIR}/geniac/install/cmake-init-default.cmake ${SRC_DIR}/geniac/install/cmake-init.cmake # edit the file ${SRC_DIR}/geniac/install/cmake-init.cmake to set your options cd ${BUILD_DIR} cmake -C ${SRC_DIR}/geniac/install/cmake-init.cmake ${SRC_DIR}/geniac .. note:: If you use a Linux distribution which is binary-compatible with the Red Hat Enterprise Linux (RHEL) operating system source code (such as |almalinux|_, |centos|_ or |rockylinux|_), |cmake|_ version 3 is available as ``cmake3`` executable, the syntax is ``cmake3 ${SRC_DIR}/geniac -C ${SRC_DIR}/geniac/install/cmake-init.cmake`` Containers ========== .. warning:: In order to build singularity images, **root** credentials are required: * either type `make` if you have `fakeroot` singularity credentials * or `sudo make` if you have sudo privileges * then `make install` In order to build the containers, you can either pass the required options during the configure stage (see :ref:`install-configure`) or use custom targets (see :ref:`install-target-containers`). Custom targets ============== The custom targets will be available once you have performed the configuration step with ``cmake``. .. _install-target-config: Build configuration files ------------------------- Assume you are in the ``build`` directory. The following custom target allows you to build only the configuration files: * ``make build_config_files`` The configuration files will be generated in ``build/workDir/results/conf``. .. _install-target-containers: Build recipes and containers ---------------------------- Assume you are in the ``build`` directory. The following custom targets allows you to build recipes and containers even if you did not ask for them during the configure stage: * ``make build_singularity_recipes`` * ``make build_singularity_images`` * ``make build_docker_recipes`` * ``make build_docker_images`` * ``make build_podman_recipes`` * ``make build_podman_images`` For singularity: * Recipes will be generated in ``build/workDir/results/singularity/deffiles``. * Images will be generated in ``build/workDir/results/singularity/images``. For docker: * Recipes will be generated in ``build/workDir/results/docker/Dockerfiles``. * Images will be created in the registry. .. _install-test: Install and test with different profiles ---------------------------------------- In order to make the deployment and testing of the pipeline easier, several custom targets are provided such that you only need to type one of the following command to install the pipeline: * ``make test_conda`` * ``make test_docker`` * ``make test_multiconda`` * ``make test_multipath`` * ``make test_path`` * ``make test_singularity`` * ``make test_standard`` Assuming that you configured the build directory such that ``CMAKE_INSTALL_PREFIX=${INSTALL_DIR}``, typing ``make test_conda`` is similar to: :: make make install cd ${SRC_DIR}/pipeline nextflow -c conf/test.config run main.nf -profile conda If you want to add the :ref:`run-profile-cluster` profile, just type the following: * ``make test_conda_cluster`` * ``make test_docker_cluster`` * ``make test_multiconda_cluster`` * ``make test_multipath_cluster`` * ``make test_path_cluster`` * ``make test_singularity_cluster`` * ``make test_standard_cluster`` .. note:: For these custom targets to be available, test data and ``conf/test.config`` file have to be provided in the repository. .. _install-structure-dir-tree: Structure of the installation directory tree ============================================ :: ├── annotations ├── containers │   └── singularity ├── multipath │   ├── alpine │   │   └── bin │   ├── fastqc │   │   └── bin │   ├── helloWorld │   │   └── bin │   ├── rmarkdown │   │   └── bin │   └── trickySoftware │   └── bin ├── path │   ├── bin └── pipeline ├── assets ├── bin │   └── fromSource ├── conf ├── docs ├── env ├── modules │   └── helloWorld ├── recipes │   ├── conda │   ├── dependencies │   ├── docker │   └── singularity └── test └── data Examples ======== Install and run with conda -------------------------- .. important:: You must have |conda|_ installed locally, if not, proceed as follows: :: wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh Then, edit your file ``.bashrc`` and add ``$HOME/miniconda3/bin`` (or the install directory you set) in your PATH. :: export WORK_DIR="${HOME}/tmp/myPipeline" export SRC_DIR="${WORK_DIR}/src" export INSTALL_DIR="${WORK_DIR}/install" export BUILD_DIR="${WORK_DIR}/build" export GIT_URL="https://github.com/bioinfo-pf-curie/geniac-demo.git" mkdir -p ${INSTALL_DIR} ${BUILD_DIR} # clone the repository # the option --recursive is needed if you use geniac as a submodule # the option --remote-submodules will pull the last geniac version # using the release branch from https://github.com/bioinfo-pf-curie/geniac git clone --remote-submodules --recursive ${GIT_URL} ${SRC_DIR} cd ${BUILD_DIR} cmake ${SRC_DIR}/geniac -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} make make install cd ${INSTALL_DIR}/pipeline nextflow -c conf/test.config run main.nf -profile multiconda .. note:: If you use both the :ref:`run-profile-conda` and :ref:`run-profile-cluster` profile, check that your master job that launches nextflow has been submitted with enough memory, otherwise the creation of the conda environment may fail. .. _install-run-singularity: Install and run with singularity -------------------------------- :: export WORK_DIR="${HOME}/tmp/myPipeline" export SRC_DIR="${WORK_DIR}/src" export INSTALL_DIR="${WORK_DIR}/install" export BUILD_DIR="${WORK_DIR}/build" export GIT_URL="https://github.com/bioinfo-pf-curie/geniac-demo.git" mkdir -p ${INSTALL_DIR} ${BUILD_DIR} # clone the repository # the option --recursive is needed if you use geniac as a submodule # the option --remote-submodules will pull the last geniac version # using the release branch from https://github.com/bioinfo-pf-curie/geniac git clone --remote-submodules --recursive ${GIT_URL} ${SRC_DIR} cd ${BUILD_DIR} cmake ${SRC_DIR}/geniac -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -Dap_install_singularity_images=ON ### /!\ with sudo, the singularity and nextflow commands must be ### /!\ in the secure_path option declared in the file /etc/sudoers ### build the files needed by the pipeline sudo make ### change file owner/group to the current user sudo chown -R $(id -gn):$(id -gn) ${BUILD_DIR} make install cd ${INSTALL_DIR}/pipeline nextflow -c conf/test.config run main.nf -profile singularity .. note:: Whenever you explicitely set an option on the command line such as ``-Dap_install_singularity_images=ON``, and then you want to reconfigure your build directory by specifying only another option on the command line such as ``-DCMAKE_INSTALL_PREFIX=${HOME}/myPipelineNewDir``, the ``ap_install_singularity_images`` will remain ``ON`` unless you specify ``-Dap_install_singularity_images=OFF``. .. _install-generate-recipes: Generate the recipes for the containers --------------------------------------- As geniac automatically generates the recipes of the containers, they are not available in the git repository. However, they can be easily retrieved in several ways. We provide here an example. First, let's assume you want the Singularity definition files. Then, pass the option ``-Dap_install_singularity_recipes=ON`` to cmake during the configuration stage: :: export WORK_DIR="${HOME}/tmp/myPipeline" export SRC_DIR="${WORK_DIR}/src" export INSTALL_DIR="${WORK_DIR}/install" export BUILD_DIR="${WORK_DIR}/build" export GIT_URL="https://github.com/bioinfo-pf-curie/geniac-demo.git" mkdir -p ${INSTALL_DIR} ${BUILD_DIR} # clone the repository # the option --recursive is needed if you use geniac as a submodule # the option --remote-submodules will pull the last geniac version # using the release branch from https://github.com/bioinfo-pf-curie/geniac git clone --remote-submodules --recursive ${GIT_URL} ${SRC_DIR} cd ${BUILD_DIR} cmake ${SRC_DIR}/geniac -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -Dap_install_singularity_recipes=ON make install ### the Singularity definition files are available here: ls ${INSTALL_DIR}/pipeline/recipes/singularity Second, let's assume you want the Dockerfiles. Then, use a custom target (see :ref:`install-target-containers`) as follows: :: make build_docker_recipes make install ### the Dockerfiles are available here: ls ${INSTALL_DIR}/pipeline/recipes/docker Obviously, if you want both the Singularity definition files and Dockerfiles you can simply do the following during the configuration stage: :: cmake ${SRC_DIR}/geniac -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} -Dap_install_singularity_recipes=ON -Dap_install_docker_recipes=ON