Continuous integration¶

Please note that if one of the following markers appear in the latest commit message, the following actions are taken.

Commit message marker	Action taken by CI
[skip ci]	Github CI is skipped completely. Several other options are also possible, see github documentation.
[skip test]	Skip running the tests.
[skip doc]	Skip building the doc.
[test nightly]	Run tests on the nightly build of Nilearn’s dependencies.
[full doc]	Run a full build of the documentation (meaning that all examples will be run).
[example] name_of_example.py	Run partial documentation build but will run the requested example.
[force download]	Force a download of all the dataset required for the build of the documentation.

Documentation build and deploy¶

        flowchart TD
subgraph Circle-CI
nilearn_CCI(["host_docs job"])
end

subgraph GitHub - nilearn.github.io
nilearn_doc_dev["<code>/dev"]
nilearn_doc_stable["<code>/stable"]
end

subgraph GitHub - nilearn
DocumentationBuilder["build-docs.yml"]
trigger_hosting["trigger_hosting_on_pr.yml"]
artifacts@{ shape: docs, label: "doc build artifacts" }
end

DocumentationBuilder -- on main --> nilearn_doc_dev
DocumentationBuilder -- on GitHub release ---> nilearn_doc_stable
DocumentationBuilder -- on PR --> trigger_hosting & artifacts
trigger_hosting -- triggers --> nilearn_CCI
artifacts -. fetched by .-> nilearn_CCI

The development docs are built using GitHub Actions and Circle-CI.

The workflow .github/workflows/build-docs.yml builds the doc and stores as a Github artifact. On the main branch (or respectively on GitHub release), this workflow will also deploy the doc to https://github.com/nilearn//nilearn.github.io.git on the dev (respectively stable).

On pull-request, a successful build of the doc by .github/workflows/build-docs.yml triggers trigger_hosting_on_pr.yml. This latter workflow will launch the circle-CI host_docs jobs by calling the circle-CI API. CircleCI download the doc artifacts from GitHub and hosts it on circle-CI.

The documentation for a pull request can be accessed on https://app.circleci.com/pipelines/github/nilearn/nilearn, by selecting the host_and_deploy_doc workflow for the corresponding pull-request, then by selecting the host_docs job, clicking on the ARTIFACTS tab, and opening the the landing page of the deployed doc by clikcing on index.html.

GitHub Actions Specification¶

assign.yml¶

Workflow file: assign.yml

Allows anyone to self-assign an issue automatically by commenting the word take on any issue.

See also

Adapted from Scikit-learn.

auto-comment.yml¶

Workflow file: auto-comment.yml

Automatically comments on a newly open pull request to provide some guidelines, useful links and a checklist.

The checklist is only editable by maintainers at the moment.

benchmark.yml¶

Workflow file: benchmark.yml

TL;DR: this workflow runs the benchmarks on the latest commit. It runs on PR. On main, it will also run on a schedule at regular interval or can be triggered manually via the GitHub UI. When run on main it will deploy the updated benchmarks to nilearn.github.io/benchmarks.

Step-wise details:

First it installs asv.
On a pull request, the checkout is pinned to the actual HEAD commit of the PR branch (rather than the synthetic PR merge commit), and the benchmarks are run for that commit as well as for the tip of the upstream main branch, so the two can be compared. Since the PR branch may live on a fork that is not reachable from the repo URL configured in asv.conf.json, asv is temporarily pointed at the already checked out local clone instead (it contains both commits, as the checkout step fetches full history).
On a push to main (typically a merged PR), the benchmarks are run for the main HEAD and for the most recent git tag (the latest stable release), and this comparison overwrites whatever was previously published on the benchmarks repo, rather than being appended to the historical results used by the schedule/manual runs.
Then it sets up the SSH key to access the benchmarks repo, pulls it, and copies the results dir into the current dir. We will append the new results to these results.
asv machine --yes then fetches the machine info like CPU, RAM, OS, etc. and saves it in ~/.asv-machine.json, but it gives a unique name to the machine on every run even if the specs are the same. So we will set a fixed name for the machine so the legend isn’t too overcrowded.
To do this, we edit the ~/.asv-machine.json file to change the machine name to a fixed name fv-az1113-357. The name is arbitrary and has been chosen to match the first run. The script build_tools/github/set_machine_name.py does all this.
Then we run the benchmarks (asv run) such that the results are appended to the old results (via --append-samples parameter). -ev makes sure any errors are printed in detail and all the output can be seen in the logs.
asv run exits 0 and keeps going even when individual benchmarks fail, so we explicitly grep the log for asv: benchmark failed and fail the workflow if it is found.
Then we create the HTML with all the results via asv publish.
We upload the html and results directories as a single artifact so that we can download them later, e.g. to inspect the results of a PR run without waiting for them to be deployed. To view them after downloading and unzipping the artifact, the html directory it contains cannot simply be opened in a browser (it needs its data served over HTTP, not opened as local file:// pages). Instead, either:
- run python -m http.server from inside the unzipped html directory, then open the printed URL (e.g. http://localhost:8000) in a browser; or
- if asv is installed, run asv preview from the unzipped directory that contains both html and results (this also works from a local asv_benchmarks checkout after running asv publish, as described in CONTRIBUTING.rst).
Then we push the new results back to the benchmarks repo. This will automatically deploy the new results to nilearn.github.io/benchmarks.

build-docs.yml¶

Workflow file: build-docs.yml

Workflow to build the documentation.

validate CITATION.CFF file
check the quality of the doc strings
get datasets from cache or from source
build doc and upload it as artifact
on main (or on github release): trigger hosting of dev (or stable) doc on https://github.com/nilearn//nilearn.github.io.git
on main when triggered by schedule or manually via the github UI, this workflow will also attempt to do a full build of the documentation on the latest supported python and dependencies

Control via commit message

This workflow can be controlled if your commit message contains:

[full doc] runs a full build on pull-request.
[skip doc] skip build of the documentation.
[example] name_of_example.py trigger build of some specific examples.
[force download] do not rely of cached datasets and force to redownload them.
[reports] build estimators reports that are embedded in the documentation.

On Pull Requests, “partial builds” are run by default which render all the rst files, but only build examples modified in the Pull Request. This saves a lot of time and resources when working on Pull Requests.

Occasionally, some changes necessitate rebuilding the documentation from scratch, for example to see the full effect of the changes. These are called “full builds”.

You can request a full build from a Pull Request at any time by including the tag [full doc] in your commit message. Note that this will trigger a full build of the documentation which usually takes >90 minutes.

$ git commit --message "[full doc] request full build"

Full builds are always run on “main”. This is done every time there is a push on “main” and every week.

Most of this workflow is skipped if [skip doc] is in the commit message.

Though partial builds will build modified examples, sometimes code changes on the module side could affect the plots in unmodified examples. For this, you can request for the CI to build a specific example by using the tag [example] and the name of the example. This is useful when wanting to get quick feedback from reviewers.

$ git commit --message "[example] plot_nilearn_101.py"

However for quick checks to do yourself you should always opt for local builds following the instructions here: in the Building documentation section.

Dataset caching¶

We also implemented a dataset caching strategy within this workflow such that datasets are only downloaded once every month. Once these datasets are cached, they will be used by all jobs running without requiring any download. This saves a lot of time and avoids potential network errors that can happen when downloading datasets from remote servers.

There is a single cache for the month, shared by all runs of this workflow (scheduled, push and pull request alike). Only the first run of the month populates it; every other run only reads from it and never overwrites it. This keeps a single long-lived ~4GB+ cache entry instead of every run saving its own copy, which used to blow through the repository’s 10GB Actions cache quota and evict the shared cache well before it was reused.

Data is passed from the get_data job to the build_docs job of the same run via a short-lived (1 day) build artifact, not via the cache above.

Note that you can request to download all datasets and ignore the cache at any time by including the tag [force download] in your commit message.

To run a full build and download all datasets, you would then combine both tags:

$ git commit --message "[full doc][force download] request full build"

check_code_complexity.yml¶

Workflow file: check_code_complexity.yml

Check that the cognitive complexity of the code has not increased.

Uses complexipy (https://github.com/rohaquinlop/complexipy) snapshot baselines (see https://rohaquinlop.github.io/complexipy/usage-guide/#snapshot-baselines) to compare the pull request against the base branch:

A snapshot of the base branch is generated on the fly (recording every function currently above MAX_COMPLEXITY_ALLOWED) instead of being committed to the repository, so nothing needs to be kept in sync or auto-committed back by CI.
That snapshot is copied into the pull request checkout, and complexipy is run there: functions already in the snapshot still pass unless they got worse, and new functions above the threshold fail.

check_gha_workflow.yml¶

Workflow file: check_gha_workflow.yml

Workflow to check the time taken to run tests and build the documentation.

Pings Github API to collect information about:

how long each run of the test suite lasted,
how long the build of the doc lasted.

Plots the results and saves it as an artifact to download and manually inspect to see if there is a trend in tests or doc build taking longer.

See also

maint_tools/check_gha_workflow.py

check_skip_flags.yml¶

Workflow file: check_skip_flags.yml

Job to check skip flags in a commit message.

To reuse from a job, use:

uses: ./.github/workflows/check_skip_flags.yml

circle_artifacts.yml¶

Workflow file: circle_artifacts.yml

Display a link to the circle-CI artifacts at the bottom of a pull-request.

label_blank_issue.yml¶

Workflow file: label_blank_issue.yml

Gives ‘Needs Triage’ label to new issues.

linkcheck.yml¶

Workflow file: linkcheck.yml

Check for dead links in the the documentation.

Links to ignore are set in doc/conf.py (see linkcheck_.* variables).

nightly_dependencies.yml¶

Workflow file: nightly_dependencies.yml

Run test suite using the nightly release of Nilearn dependencies.

When running on main, if the workflow fails the action will open an issue using this issue template.

sourcery.yml¶

Workflow file: sourcery.yml

Runs Sourcery refactoring on the entire codebase.

If some potential refactoring are found it opens a pull-request.

Requires a SOURCERY_TOKEN that can be obtained from logging into the sourcery website with your github account.

test_atlases.yml¶

Workflow file: test_atlases.yml

Test label maskers on atlases (from nilearn, templateFlow…)

test_html.yml¶

Workflow file: test_html.yml

Test that HTML output stay visually constant.

HTML figures and reports are generated either via doc/visual_testing/reporter_visual_inspection_suite.py or test/js/generate_html.py
Javascript tests are run to open the HTML documents, take a screenshot of it, and compare it to a reference screenshot

Most of this is handled via tox: tox run -e test_html

Screenshots (reference, current and their difference) are uploaded at the end of the workflow.

test_with_free_threaded_python.yml¶

Workflow file: test_with_free_threaded_python.yml

Runs tests suite with free threaded python.

test_with_tox.yml¶

Workflow file: test_with_tox.yml

Runs pytest in several environments including several Python and dependencies versions as well as on different systems. All environments are defined in tox.ini.

Control via commit message

This workflow can be controlled if your commit message contains:

[skip test] skip this workflow.

Uploads as artifacts:

report.html that gives a browsable overview of the tests results
pytest_output that contains a .csv of the tests results and figures showing the run time of all the tests See maint_tools/plot_test_timing.py.
htmlcov containing the html version of the code coverage.

Not all python version run on pull requests.

When this workflow is triggered by a pull-request, it will run using only the oldest and latest python version we support. When running on main all combinations of operating system and python versions are run.

testing_install.yml¶

Workflow file: testing_install.yml

Runs several checks on the nilearn package:

tries to install Nilearn from wheel & check installation on all operating systems
ensure thath the nilearn wheel is not over a certain size
ensure the whole nilearn install with all its transitive dependencies is not too large

testing_minimum.yml¶

Workflow file: testing_minimum.yml

This workflow is automatically run once a month.

Checks that installing the minimum version of a given dependency of Nilearn along with the latest version of all the other dependencies leads to a successful run of all the tests.

trigger_hosting_on_pr.yml¶

Workflow file: trigger_hosting_on_pr.yml

Trigger hosting of the doc on circle-ci on pull-requests.

We get the url for the artifacts for circle-ci via https://nightly.link/ .

This must be run in a separate workflow because for security reasons the secrets for login to circle-ci cannot be accessed by pull-requests opened from a fork.

So this workflow waits for the doc building workflow to finish successfully before grabbing its output and pushing it to circle-CI.

CIRCLE_CI_TOKEN

The CIRCLE_CI_TOKEN used in this workflow must be a personal token and not a project token: https://support.circleci.com/hc/en-us/articles/360060360811-CircleCI-API-v2-Returns-404-Not-Found

update_authors.yml¶

Workflow file: update_authors.yml

If the CITATION.CFF file is modified, this workflow will run to update the AUTHORS file and the and doc/changes/names.rst file.

Circle-CI workflows¶

Source file: .circleci/config.yml