.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/02_decoding/plot_haxby_multiclass.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. or to run this example in your browser via Binder .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_02_decoding_plot_haxby_multiclass.py: The Haxby dataset: different multi-class strategies =================================================== In this example, we compare ``one vs all`` and ``one vs one`` multi-class strategies: We compare their overall cross-validated accuracy and their confusion matrix. See the scikit-learn documentation about `multiclass classification `_. .. GENERATED FROM PYTHON SOURCE LINES 13-19 .. code-block:: Python import numpy as np import pandas as pd from nilearn import datasets .. GENERATED FROM PYTHON SOURCE LINES 20-23 Load and prepare the Haxby dataset ---------------------------------- By default 2nd subject from haxby datasets will be fetched. .. GENERATED FROM PYTHON SOURCE LINES 23-32 .. code-block:: Python haxby_dataset = datasets.fetch_haxby() func_filename = haxby_dataset.func[0] mask_filename = haxby_dataset.mask print(f"Mask nifti images are located at: {mask_filename}") print(f"Functional nifti images are located at: {func_filename}") .. rst-class:: sphx-glr-script-out .. code-block:: none [fetch_haxby] Dataset directory found: /home/runner/work/nilearn/nilearn/nilearn_data/haxby2001 Mask nifti images are located at: /home/runner/work/nilearn/nilearn/nilearn_data/haxby2001/mask.nii.gz Functional nifti images are located at: /home/runner/work/nilearn/nilearn/nilearn_data/haxby2001/subj2/bold.nii.gz .. GENERATED FROM PYTHON SOURCE LINES 33-41 We load the behavioral data that we will predict and remove the ``"rest"`` condition, as it is of no interest to us. .. seealso:: For more information about the dataset see its :ref:`description `. .. GENERATED FROM PYTHON SOURCE LINES 41-52 .. code-block:: Python labels = pd.read_csv(haxby_dataset.session_target[0], sep=" ") non_rest = labels["labels"] != "rest" y = labels["labels"][non_rest] run = labels["chunks"] n_runs = len(np.unique(run)) .. GENERATED FROM PYTHON SOURCE LINES 53-56 We extract the data with a NiftiMasker. For decoding, standardizing is often very important, so we set ``standardize="zscore_sample"``. .. GENERATED FROM PYTHON SOURCE LINES 56-70 .. code-block:: Python from nilearn.maskers import NiftiMasker nifti_masker = NiftiMasker( mask_img=mask_filename, runs=run, smoothing_fwhm=4, standardize="zscore_sample", memory="nilearn_cache", memory_level=1, verbose=1, ) X = nifti_masker.fit_transform(func_filename) .. rst-class:: sphx-glr-script-out .. code-block:: none [NiftiMasker.wrapped] Loading data from .../subj2/bold.nii.gz [NiftiMasker.wrapped] Loading mask from .../mask.nii.gz [NiftiMasker.wrapped] Resampling mask [NiftiMasker.wrapped] Finished fit ________________________________________________________________________________ [Memory] Calling nilearn.maskers.nifti_masker.filter_and_mask... filter_and_mask('/home/runner/work/nilearn/nilearn/nilearn_data/haxby2001/subj2/bold.nii.gz', , { 'clean_args': None, 'clean_kwargs': {}, 'cmap': 'gray', 'detrend': False, 'dtype': None, 'high_pass': None, 'high_variance_confounds': False, 'low_pass': None, 'reports': True, 'runs': 0 0 1 0 2 0 3 0 4 0 .. 1447 11 1448 11 1449 11 1450 11 1451 11 Name: chunks, Length: 1452, dtype: int64, 'smoothing_fwhm': 4, 'standardize': 'zscore_sample', 'standardize_confounds': True, 't_r': None, 'target_affine': None, 'target_shape': None}, memory_level=1, memory=Memory(location=nilearn_cache/joblib), verbose=1, confounds=None, sample_mask=None, copy=True, sklearn_output_config=None) [NiftiMasker.wrapped] Loading data from [NiftiMasker.wrapped] Smoothing images [NiftiMasker.wrapped] Extracting region signals [NiftiMasker.wrapped] Cleaning extracted signals _________________________________________________filter_and_mask - 10.7s, 0.2min .. GENERATED FROM PYTHON SOURCE LINES 71-72 Remove the ``"rest"`` condition .. GENERATED FROM PYTHON SOURCE LINES 72-75 .. code-block:: Python X = X[non_rest] run = run[non_rest] .. GENERATED FROM PYTHON SOURCE LINES 76-82 Build the decoders, using scikit-learn -------------------------------------- Nilearn does not have dedicated multiclass estimators, as we can directly use those from sklearn. Here we use a Support Vector Classification (:class:`sklearn.svm.SVC`), with a linear kernel, and a simple feature selection step. .. GENERATED FROM PYTHON SOURCE LINES 82-88 .. code-block:: Python from sklearn.feature_selection import SelectKBest, f_classif from sklearn.multiclass import OneVsOneClassifier, OneVsRestClassifier from sklearn.pipeline import Pipeline from sklearn.svm import SVC .. GENERATED FROM PYTHON SOURCE LINES 89-99 .. code-block:: Python svc_ovo = OneVsOneClassifier( Pipeline( [ ("anova", SelectKBest(f_classif, k=500)), ("svc", SVC(kernel="linear")), ] ) ) svc_ovo .. raw:: html 
OneVsOneClassifier(estimator=Pipeline(steps=[('anova', SelectKBest(k=500)),
                                                 ('svc', SVC(kernel='linear'))]))
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. GENERATED FROM PYTHON SOURCE LINES 100-110 .. code-block:: Python svc_ova = OneVsRestClassifier( Pipeline( [ ("anova", SelectKBest(f_classif, k=500)), ("svc", SVC(kernel="linear")), ] ) ) svc_ova .. raw:: html 
OneVsRestClassifier(estimator=Pipeline(steps=[('anova', SelectKBest(k=500)),
                                                  ('svc', SVC(kernel='linear'))]))
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.


.. GENERATED FROM PYTHON SOURCE LINES 111-117 Now we compute cross-validation scores -------------------------------------- The :term:`fMRI` data is acquired by runs, and the noise is autocorrelated in a given run. Hence, it is better to predict across runs when doing cross-validation. Here we do a 5 fold cross-validation. .. GENERATED FROM PYTHON SOURCE LINES 117-122 .. code-block:: Python from sklearn.model_selection import cross_val_score cv = 5 .. GENERATED FROM PYTHON SOURCE LINES 123-126 .. code-block:: Python cv_scores_ovo = cross_val_score(svc_ovo, X, y, cv=cv, verbose=1) cv_scores_ovo .. rst-class:: sphx-glr-script-out .. code-block:: none [Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers. [Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 9.2s finished array([0.50289017, 0.65895954, 0.58959538, 0.65317919, 0.60465116]) .. GENERATED FROM PYTHON SOURCE LINES 127-130 .. code-block:: Python cv_scores_ova = cross_val_score(svc_ova, X, y, cv=cv, verbose=1) cv_scores_ova .. rst-class:: sphx-glr-script-out .. code-block:: none [Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers. [Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 4.8s finished array([0.62427746, 0.71676301, 0.66473988, 0.75722543, 0.59302326]) .. GENERATED FROM PYTHON SOURCE LINES 131-134 .. code-block:: Python print("OvO:", cv_scores_ovo.mean().round(decimals=3)) print("OvA:", cv_scores_ova.mean().round(decimals=3)) .. rst-class:: sphx-glr-script-out .. code-block:: none OvO: 0.602 OvA: 0.671 .. GENERATED FROM PYTHON SOURCE LINES 135-137 Plot barplots of the prediction scores -------------------------------------- .. GENERATED FROM PYTHON SOURCE LINES 137-149 .. code-block:: Python from matplotlib import pyplot as plt from nilearn.plotting import show plt.figure(figsize=(4, 3)) plt.boxplot([cv_scores_ova, cv_scores_ovo]) plt.xticks([1, 2], ["One vs All", "One vs One"]) plt.title("Prediction: accuracy score") show() .. image-sg:: /auto_examples/02_decoding/images/sphx_glr_plot_haxby_multiclass_001.png :alt: Prediction: accuracy score :srcset: /auto_examples/02_decoding/images/sphx_glr_plot_haxby_multiclass_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 150-158 Plot the cross-validated confusion matrices -------------------------------------------- Instead of fitting on a single train/test split, we use :func:`sklearn.model_selection.cross_val_predict` with the same cross-validation as above to get, for each sample, a prediction made by a model that never saw that sample during training. We can then build a single confusion matrix from those out-of-fold predictions for the whole dataset. .. GENERATED FROM PYTHON SOURCE LINES 158-165 .. code-block:: Python from sklearn.model_selection import cross_val_predict y_pred_ovo = cross_val_predict(svc_ovo, X, y, cv=cv, verbose=1) y_pred_ova = cross_val_predict(svc_ova, X, y, cv=cv, verbose=1) .. rst-class:: sphx-glr-script-out .. code-block:: none [Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers. [Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 9.4s finished [Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers. [Parallel(n_jobs=1)]: Done 5 out of 5 | elapsed: 4.8s finished .. GENERATED FROM PYTHON SOURCE LINES 166-168 We get the labels of the numerical conditions represented by the vector y and we sort the conditions by the order of appearance. .. GENERATED FROM PYTHON SOURCE LINES 168-204 .. code-block:: Python from sklearn.metrics import confusion_matrix from nilearn.plotting import plot_matrix unique_conditions, order = np.unique(y, return_index=True) unique_conditions = unique_conditions[np.argsort(order)] im = plot_matrix( confusion_matrix(y_pred_ovo, y), labels=unique_conditions, title="Confusion matrix: One vs One", cmap="inferno", figure=(6, 5), auto_fit=False, vmax=108, ) ax = im.axes ax.set_ylabel("True label") ax.set_xlabel("Predicted label") im = plot_matrix( confusion_matrix(y_pred_ova, y), labels=unique_conditions, title="Confusion matrix: One vs All", cmap="inferno", figure=(6, 5), auto_fit=False, vmax=108, ) ax = im.axes ax.set_ylabel("True label") ax.set_xlabel("Predicted label") show() .. rst-class:: sphx-glr-horizontal * .. image-sg:: /auto_examples/02_decoding/images/sphx_glr_plot_haxby_multiclass_002.png :alt: Confusion matrix: One vs One :srcset: /auto_examples/02_decoding/images/sphx_glr_plot_haxby_multiclass_002.png :class: sphx-glr-multi-img * .. image-sg:: /auto_examples/02_decoding/images/sphx_glr_plot_haxby_multiclass_003.png :alt: Confusion matrix: One vs All :srcset: /auto_examples/02_decoding/images/sphx_glr_plot_haxby_multiclass_003.png :class: sphx-glr-multi-img .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 44.699 seconds) **Estimated memory usage:** 2572 MB .. _sphx_glr_download_auto_examples_02_decoding_plot_haxby_multiclass.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: binder-badge .. image:: images/binder_badge_logo.svg :target: https://mybinder.org/v2/gh/nilearn/nilearn/0.14.1?urlpath=lab/tree/notebooks/auto_examples/02_decoding/plot_haxby_multiclass.ipynb :alt: Launch binder :width: 150 px .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_haxby_multiclass.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_haxby_multiclass.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_haxby_multiclass.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_