Development cycle¶

Prototyping¶

First, configure your repository as explained here: Set options with a file. In general, this has to be done only once unless you want to change the options.

When prototyping the pipeline, we advise to use the multiconda profile. At this stage, the containers should not be available thus making impossible to use the singularity or docker profiles.

We suggest that you provide test data and a conf/test.config file such that the pipeline can be tested on any modification of the source code for validation. Whenever possible, the test data must be as small as possible such that running the test does not take too much time.

Then, to install and test your modifications, just type make test_multiconda (see Install and test with different profiles) in the build directory. The first time this command is typed, the config files are automatically generated and installed. The configuration files will be regenerated whenever you modify the conf/geniac.config file (or whenever something is added or modified in both the recipes or modules directories).

Note

You can combine both the multiconda with the multipath profiles as described in the Combine path/multipath profile with conda/multiconda section. This offers the possibility to install on your own all the software you need to setup the analysis methodology for the pipeline you are developing, in particular whenever you fall in any of the cases of as described in the Process types and profiles table.

If you don’t want any test to be started, just type make install.

Whatever you use make test_multiconda (or any custom targets available in Install and test with different profiles) or make install, only the files that have been modified will be installed that allows this step to be just a quick copy of the modified files in the install directory (if it is not necessary to generate the config files).

Important

Why it is essential to deploy the pipeline in a dedicated directory and then test your modifications rather than testing it directly from your source code directory in which you are developing?

The deployment of the pipeline in a dedicated directory makes it possible to keep developing and modifying any file or to checkout any branch while a test is running especially when the test can take time. If you would launch a test from the source code directory the files could be modified while the test is running.

Note

If you really prefer to launch your test in your source code directory (for good or bad reasons) you can still do it. In this case, you can either write and add the config files for the nextflow Profiles as described in Config files for the different nextflow profiles in the conf directory of copy them from config files automatically generated.

Containerizing¶

Building the Singularity or Docker containers should start once the prototyping is over. Thus, the software developers will take care of:

writing the recipes for any process that have a label falling in the Install from source code or Custom install categories,
performing Resource tuning in order to optimize the informatic resource asked by the different processes.

Deployment¶

Whoever you are, follow the guidelines describes in the Installation section.

Git¶

We assume that the reader is familiar with Git (if not refer to the documentation for details).

Branching strategy¶

The management of the different development versions is based on different git branches. Each branch is used depending on the context and the stage in the development cycle. The model we recommend is based on a central git repository that contains 4 branches:

devel: contains the code of the current version under development
release: contains the code with the official releases
hotfix (if needed): this branch is a mirror of the release branch and is used to patch the code that is in production. If a critical anomaly happens in production, this branch is used to fix the issue.
master: this branch is not used for development, it is only used to archive the code from the release and hotfix branches.

The developer is encouraged to create local branches in his local development environment whenever he/she develops new features or hotfixes.

The workflow across the different branches can be summarized as described in the graphic below:

Warning

Do not forget to merge any developements from release to devel, and from hotfix to release and devel such that all the branches are up-to-date with last developments and hotfixes.

Tip

For more details on the branching model and the usage of git, we recommend the user to read the biogitflow documentation and the associated article biogitflow: development workflow protocols for bioinformatics pipelines with git and GitLab (Kamoun et al., 2020).

Tag strategy¶

For official release that can be used in a production environment, use a tag with the prefix version- (for example version-1.2.3).

Warning

This is essential that you use this naming convention. Indeed, this will allow the display of a message if a user runs a pipeline with a development version that can be unstable and thus providing results that are not reliable. For example:

======================================================================
DISCLAIMER

This software is currently under active development and the results
have been generated with a non stable version.
The reliability, reproducibility and the quality of the results are
therefore not guaranteed.

/!\ Do not use the results for any kind of projects /!\
======================================================================

This head will be also displayed in the MultiQC report.