.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "tutorials/tutorial_examples/plot_002_bids_layout.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_tutorials_tutorial_examples_plot_002_bids_layout.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_tutorials_tutorial_examples_plot_002_bids_layout.py:


====================
How pyAFQ uses BIDS
====================

The pyAFQ API relies heavily on the
`Brain Imaging Data Standard (BIDS) <https://bids-specification.readthedocs.io/en/stable/>`_,
a widely used standard for organizing and describing neuroimaging data. This
means that the software assumes that its inputs are organized according to the
BIDS specification and its outputs conform where possible with BIDS.

.. note::

    Derivatives of processing diffusion MRI are not currently fully
    described in the existing BIDS specification, but describing these
    is part of an ongoing effort. Wherever possible, we conform with
    the draft implementation of the BIDS DWI derivatives available
    `here <https://bids-specification.readthedocs.io/en/wip-derivatives/05-derivatives/05-diffusion-derivatives.html>`_

In this example, we will explore the use of BIDS in pyAFQ and see
how BIDS allows us to extend and provide flexibility to the users
of the software.

.. GENERATED FROM PYTHON SOURCE LINES 24-33

.. code-block:: Python


    import os
    import os.path as op

    import AFQ.api.bundle_dict as abd
    from AFQ.api.group import GroupAFQ, get_afq_bids_entities_fname
    import AFQ.data.fetch as afd









.. GENERATED FROM PYTHON SOURCE LINES 34-36

To interact with and query BIDS datasets, we use
 `pyBIDS <https://bids-standard.github.io/pybids/>`_, which we import here:

.. GENERATED FROM PYTHON SOURCE LINES 36-40

.. code-block:: Python


    import bids
    from bids.layout import BIDSLayout








.. GENERATED FROM PYTHON SOURCE LINES 41-47

We start with some example data. The data we will use here is
generated from the
`Stanford HARDI dataset <https://purl.stanford.edu/ng782rw8378>`_.
The call below fetches
this dataset and organized it within the `~/AFQ_data` folder in the BIDS
format.

.. GENERATED FROM PYTHON SOURCE LINES 47-50

.. code-block:: Python


    afd.organize_stanford_data(clear_previous_afq="all")








.. GENERATED FROM PYTHON SOURCE LINES 51-93

After doing that, we should have a folder that looks like this:

| stanford_hardi
| ├── dataset_description.json
| └── derivatives
|     ├── freesurfer
|     │   ├── dataset_description.json
|     │   └── sub-01
|     │       └── ses-01
|     │           └── anat
|     │               ├── sub-01_ses-01_T1w.nii.gz
|     │               └── sub-01_ses-01_seg.nii.gz
|     └── vistasoft
|         ├── dataset_description.json
|         └── sub-01
|             └── ses-01
|                 └── dwi
|                     ├── sub-01_ses-01_dwi.bvals
|                     ├── sub-01_ses-01_dwi.bvecs
|                     └── sub-01_ses-01_dwi.nii.gz

The top level directory (`stanford_hardi`) is our overall BIDS dataset folder.
In many cases, this folder will include folders with raw data for each subject
in the dataset. In this case, we do not include the raw data folders and only
have the outputs of pipelines that were used to preprocess the data (e.g.,
correct the data for subject motion, eddy currents, and so forth).
In general, only the preprocessed diffusion data is required for pyAFQ to run.
See the :doc:`"Organizing your data" </howto/usage/organizing>` section of the
documentation for more details.
In this case, one folder contains derivative of the Freesurfer software and
another folder contains the DWI data that has been preprocessed with the
Vistasoft software.
pyAFQ provides facilities to segment tractography results obtained
using other software as well. For example, we often use
`qsiprep <https://qsiprep.readthedocs.io/en/latest/>`_ to preprocess
our data and reconstruct tractographies with software such as
`MRTRIX <https://www.mrtrix.org/>`_. Here, we will demonstrate how to use
these reconstructions in the pyAFQ segmentation and tractometry pipeline
We fetch this data and add it as a separate pipeline
The following code will download a previously-created tractography and
organize it by adding it to the BIDS dataset folder and renaming them to be
BIDS-compliant (e.g., `sub-01_ses_01_dwi_tractography.trk`).

.. GENERATED FROM PYTHON SOURCE LINES 93-118

.. code-block:: Python


    afd.fetch_stanford_hardi_tractography()

    bids_path = op.join(op.expanduser('~'), 'AFQ_data', 'stanford_hardi')
    tractography_path = op.join(bids_path, 'derivatives', 'my_tractography')
    sub_path = op.join(tractography_path, 'sub-01', 'ses-01', 'dwi')

    os.makedirs(sub_path, exist_ok=True)
    os.rename(
        op.join(
            op.expanduser('~'),
            'AFQ_data',
            'stanford_hardi_tractography',
            'full_segmented_cleaned_tractography.trk'),
        op.join(
            sub_path,
            'sub-01_ses-01-dwi_tractography.trk'))

    afd.to_bids_description(
        tractography_path,
        **{"Name": "my_tractography",
           "PipelineDescription": {"Name": "my_tractography"},
           "GeneratedBy": [{"Name": "my_tractography"}]})






.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|          | 0/11337 [00:00<?, ? MB/s]
      0%|          | 2/11337 [00:00<12:45, 14.80 MB/s]
      0%|          | 5/11337 [00:00<09:54, 19.07 MB/s]
      0%|          | 8/11337 [00:00<09:14, 20.42 MB/s]
      0%|          | 19/11337 [00:00<04:15, 44.25 MB/s]
      0%|          | 39/11337 [00:00<02:19, 81.28 MB/s]
      1%|          | 78/11337 [00:00<01:14, 151.03 MB/s]
      1%|▏         | 159/11337 [00:00<00:37, 296.14 MB/s]
      3%|▎         | 319/11337 [00:01<00:19, 575.92 MB/s]
      4%|▍         | 504/11337 [00:01<00:13, 820.55 MB/s]
      6%|▌         | 688/11337 [00:01<00:10, 983.73 MB/s]
      8%|▊         | 871/11337 [00:01<00:09, 1093.94 MB/s]
      9%|▉         | 1056/11337 [00:01<00:08, 1173.70 MB/s]
     11%|█         | 1243/11337 [00:01<00:08, 1233.80 MB/s]
     13%|█▎        | 1426/11337 [00:01<00:07, 1260.04 MB/s]
     14%|█▍        | 1612/11337 [00:02<00:07, 1291.35 MB/s]
     16%|█▌        | 1798/11337 [00:02<00:07, 1312.23 MB/s]
     18%|█▊        | 1987/11337 [00:02<00:07, 1334.97 MB/s]
     19%|█▉        | 2174/11337 [00:02<00:06, 1345.29 MB/s]
     21%|██        | 2366/11337 [00:02<00:06, 1361.47 MB/s]
     23%|██▎       | 2557/11337 [00:02<00:06, 1376.45 MB/s]
     24%|██▍       | 2747/11337 [00:02<00:06, 1381.52 MB/s]
     26%|██▌       | 2937/11337 [00:03<00:06, 1363.77 MB/s]
     28%|██▊       | 3127/11337 [00:03<00:05, 1372.73 MB/s]
     29%|██▉       | 3306/11337 [00:03<00:05, 1347.40 MB/s]
     31%|███       | 3491/11337 [00:03<00:05, 1350.13 MB/s]
     32%|███▏      | 3678/11337 [00:03<00:05, 1357.39 MB/s]
     34%|███▍      | 3866/11337 [00:03<00:05, 1364.37 MB/s]
     36%|███▌      | 4051/11337 [00:03<00:05, 1362.05 MB/s]
     37%|███▋      | 4242/11337 [00:03<00:05, 1374.52 MB/s]
     39%|███▉      | 4428/11337 [00:04<00:05, 1371.14 MB/s]
     41%|████      | 4618/11337 [00:04<00:04, 1378.39 MB/s]
     42%|████▏     | 4804/11337 [00:04<00:04, 1374.10 MB/s]
     44%|████▍     | 4989/11337 [00:04<00:04, 1356.22 MB/s]
     46%|████▌     | 5178/11337 [00:04<00:04, 1365.17 MB/s]
     47%|████▋     | 5368/11337 [00:04<00:04, 1373.88 MB/s]
     49%|████▉     | 5559/11337 [00:04<00:04, 1380.48 MB/s]
     51%|█████     | 5750/11337 [00:05<00:04, 1388.58 MB/s]
     52%|█████▏    | 5940/11337 [00:05<00:03, 1390.38 MB/s]
     54%|█████▍    | 6123/11337 [00:05<00:03, 1368.18 MB/s]
     56%|█████▌    | 6312/11337 [00:05<00:03, 1373.42 MB/s]
     57%|█████▋    | 6500/11337 [00:05<00:03, 1375.46 MB/s]
     59%|█████▉    | 6689/11337 [00:05<00:03, 1374.68 MB/s]
     61%|██████    | 6878/11337 [00:05<00:03, 1377.89 MB/s]
     62%|██████▏   | 7069/11337 [00:06<00:03, 1383.50 MB/s]
     64%|██████▍   | 7260/11337 [00:06<00:02, 1360.26 MB/s]
     66%|██████▌   | 7452/11337 [00:06<00:02, 1369.40 MB/s]
     67%|██████▋   | 7643/11337 [00:06<00:02, 1414.02 MB/s]
     69%|██████▉   | 7822/11337 [00:06<00:02, 1332.45 MB/s]
     71%|███████   | 8012/11337 [00:06<00:02, 1350.60 MB/s]
     72%|███████▏  | 8187/11337 [00:06<00:02, 1327.80 MB/s]
     74%|███████▍  | 8374/11337 [00:06<00:02, 1340.84 MB/s]
     75%|███████▌  | 8551/11337 [00:07<00:02, 1325.75 MB/s]
     77%|███████▋  | 8740/11337 [00:07<00:01, 1343.63 MB/s]
     79%|███████▊  | 8927/11337 [00:07<00:01, 1352.46 MB/s]
     80%|████████  | 9101/11337 [00:07<00:01, 1327.28 MB/s]
     82%|████████▏ | 9285/11337 [00:07<00:01, 1334.24 MB/s]
     84%|████████▎ | 9471/11337 [00:07<00:01, 1343.18 MB/s]
     85%|████████▌ | 9660/11337 [00:07<00:01, 1356.16 MB/s]
     87%|████████▋ | 9841/11337 [00:08<00:01, 1343.05 MB/s]
     88%|████████▊ | 10030/11337 [00:08<00:00, 1356.18 MB/s]
     90%|█████████ | 10213/11337 [00:08<00:00, 1350.90 MB/s]
     92%|█████████▏| 10400/11337 [00:08<00:00, 1355.49 MB/s]
     93%|█████████▎| 10588/11337 [00:08<00:00, 1363.06 MB/s]
     95%|█████████▌| 10780/11337 [00:08<00:00, 1375.96 MB/s]
     97%|█████████▋| 10954/11337 [00:08<00:00, 1335.39 MB/s]
     98%|█████████▊| 11139/11337 [00:09<00:00, 1341.73 MB/s]
    100%|█████████▉| 11322/11337 [00:09<00:00, 1341.89 MB/s]
    100%|██████████| 11337/11337 [00:09<00:00, 1235.09 MB/s]

      0%|          | 0/14 [00:00<?, ? MB/s]
     14%|█▍        | 2/14 [00:00<00:00, 14.74 MB/s]
     36%|███▌      | 5/14 [00:00<00:00, 19.04 MB/s]
     57%|█████▋    | 8/14 [00:00<00:00, 20.42 MB/s]
    100%|██████████| 14/14 [00:00<00:00, 34.26 MB/s]

      0%|          | 0/1037 [00:00<?, ? MB/s]
      0%|          | 2/1037 [00:00<01:10, 14.66 MB/s]
      0%|          | 5/1037 [00:00<00:54, 18.93 MB/s]
      1%|          | 9/1037 [00:00<00:43, 23.65 MB/s]
      2%|▏         | 20/1037 [00:00<00:22, 46.05 MB/s]
      4%|▍         | 41/1037 [00:00<00:11, 84.70 MB/s]
      8%|▊         | 84/1037 [00:00<00:05, 162.63 MB/s]
     16%|█▋        | 170/1037 [00:00<00:02, 314.91 MB/s]
     33%|███▎      | 342/1037 [00:01<00:01, 614.99 MB/s]
     51%|█████     | 529/1037 [00:01<00:00, 850.16 MB/s]
     68%|██████▊   | 707/1037 [00:01<00:00, 989.26 MB/s]
     87%|████████▋ | 899/1037 [00:01<00:00, 1107.00 MB/s]
    100%|██████████| 1037/1037 [00:01<00:00, 682.26 MB/s]




.. GENERATED FROM PYTHON SOURCE LINES 119-148

After we do that, our dataset folder should look like this:

| stanford_hardi
| ├── dataset_description.json
| └── derivatives
|     ├── freesurfer
|     │   ├── dataset_description.json
|     │   └── sub-01
|     │       └── ses-01
|     │           └── anat
|     │               ├── sub-01_ses-01_T1w.nii.gz
|     │               └── sub-01_ses-01_seg.nii.gz
|     ├── my_tractography
|     |   ├── dataset_description.json
|     │   └── sub-01
|     │       └── ses-01
|     │           └── dwi
|     │               └── sub-01_ses-01-dwi_tractography.trk
|     └── vistasoft
|         ├── dataset_description.json
|         └── sub-01
|             └── ses-01
|                 └── dwi
|                     ├── sub-01_ses-01_dwi.bvals
|                     ├── sub-01_ses-01_dwi.bvecs
|                     └── sub-01_ses-01_dwi.nii.gz

To explore the layout of these derivatives, we will initialize a
:class:`BIDSLayout` class instance to help us see what is in this dataset

.. GENERATED FROM PYTHON SOURCE LINES 148-151

.. code-block:: Python


    layout = bids.BIDSLayout(bids_path, derivatives=True)








.. GENERATED FROM PYTHON SOURCE LINES 152-154

Because there is no raw data in this BIDS layout (only derivatives),
pybids will report that there are no subjects and sessions:

.. GENERATED FROM PYTHON SOURCE LINES 154-157

.. code-block:: Python


    print(layout)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    BIDS Layout: ...runner/AFQ_data/stanford_hardi | Subjects: 0 | Sessions: 0 | Runs: 0




.. GENERATED FROM PYTHON SOURCE LINES 158-160

But a query on the derivatives will reveal the different derivatives that
are stored here:

.. GENERATED FROM PYTHON SOURCE LINES 160-163

.. code-block:: Python


    print(layout.derivatives)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {'derivatives/recobundles': BIDS Layout: ..._hardi/derivatives/recobundles | Subjects: 1 | Sessions: 1 | Runs: 0, 'derivatives/vistasoft': BIDS Layout: ...rd_hardi/derivatives/vistasoft | Subjects: 1 | Sessions: 1 | Runs: 0, 'derivatives/freesurfer': BIDS Layout: ...d_hardi/derivatives/freesurfer | Subjects: 1 | Sessions: 1 | Runs: 0, 'derivatives/my_tractography': BIDS Layout: ...di/derivatives/my_tractography | Subjects: 1 | Sessions: 1 | Runs: 0}




.. GENERATED FROM PYTHON SOURCE LINES 164-167

We can use a :class:`bids.BIDSValidator` object to make sure that the
files within our data set are BIDS-compliant. For example, we can
extract the tractography derivatives part of our layout using:

.. GENERATED FROM PYTHON SOURCE LINES 167-170

.. code-block:: Python


    my_tractography = layout.derivatives["my_tractography"]








.. GENERATED FROM PYTHON SOURCE LINES 171-175

This variable is also a BIDS layout object. This object has a ``get``
method, which allows us to query and find specific items within the
layout. For example, we can ask for files that have a suffix consistent
with tractography results:

.. GENERATED FROM PYTHON SOURCE LINES 175-178

.. code-block:: Python


    tractography_files = my_tractography.get(suffix='tractography')








.. GENERATED FROM PYTHON SOURCE LINES 179-180

Or ask for files that have a ``.trk`` extension:

.. GENERATED FROM PYTHON SOURCE LINES 180-183

.. code-block:: Python


    tractography_files = my_tractography.get(extension='.trk')








.. GENERATED FROM PYTHON SOURCE LINES 184-185

In this case, both of these would produce the same result.

.. GENERATED FROM PYTHON SOURCE LINES 185-189

.. code-block:: Python


    tractography_file = tractography_files[0]
    print(tractography_file)





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    <BIDSFile filename='/home/runner/AFQ_data/stanford_hardi/derivatives/my_tractography/sub-01/ses-01/dwi/sub-01_ses-01-dwi_tractography.trk'>




.. GENERATED FROM PYTHON SOURCE LINES 190-191

We can also get some more structured information about this file:

.. GENERATED FROM PYTHON SOURCE LINES 191-195

.. code-block:: Python


    print(tractography_file.get_entities())






.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    {'datatype': 'dwi', 'extension': '.trk', 'session': '01', 'subject': '01', 'suffix': 'tractography'}




.. GENERATED FROM PYTHON SOURCE LINES 196-202

We can use a :class:`bids.BIDSValidator` class instance to validate that
this file is compliant with the specification. Note that the validator
requires that the filename be provided relative to the root of the BIDS
dataset, so we have to split the string that contains the full path
of the tractography to extract only the part that is relative to the
root of the entire BIDS ``layout`` object:

.. GENERATED FROM PYTHON SOURCE LINES 202-209

.. code-block:: Python


    tractography_full_path = tractography_file.path
    tractography_relative_path = tractography_full_path.split(layout.root)[-1]

    validator = bids.BIDSValidator()
    print(validator.is_bids(tractography_relative_path))





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    True




.. GENERATED FROM PYTHON SOURCE LINES 210-217

Next, we specify the information we need to define the bundles that we are
interested in segmenting. In this case, we are going to use a list of
bundle names for the bundle info. These names refer to bundles for
which we already have clear definitions of the information
needed to segment them (e.g., waypoint ROIs and probability maps).
For an example that includes custom definition of bundle info, see the
`plot_callosal_tract_profile example <http://tractometry.org/pyAFQ/auto_examples/plot_callosal_tract_profile.html>`_.

.. GENERATED FROM PYTHON SOURCE LINES 217-227

.. code-block:: Python


    bundle_info = abd.default18_bd()[
        "Left Superior Longitudinal",
        "Right Superior Longitudinal",
        "Left Arcuate",
        "Right Arcuate",
        "Left Corticospinal",
        "Right Corticospinal",
        "Forceps Minor"]








.. GENERATED FROM PYTHON SOURCE LINES 228-244

Now, we can define our GroupAFQ object, pointing to the derivatives of the
`'my_tractography'` pipeline as inputs. This is done by setting the
`import_tract` key-word argument. We pass the
`bundle_info` defined above. We also point to the preprocessed
data that is in a `'dmriprep'` pipeline. Note that the pipeline name
is not necessarily the name of the folder it is in; the pipeline name is
defined in each pipeline's `dataset_description.json`. These data were
preprocessed with 'vistasoft', so this is the pipeline we'll point to
If we were using `'qsiprep'`, this is where we would pass that
string instead. If we did that, AFQ would look for a derivatives
folder called `'stanford_hardi/derivatives/qsiprep'` and find the
preprocessed DWI data within it. Finally, to speed things up
a bit, we also sub-sample the provided tractography. This is
done by defining the segmentation_params dictionary input.
To sub-sample to 10,000 streamlines, we define
`'nb_streamlines' = 10000`.

.. GENERATED FROM PYTHON SOURCE LINES 244-255

.. code-block:: Python


    my_afq = GroupAFQ(
        bids_path,
        preproc_pipeline='vistasoft',
        bundle_info=bundle_info,
        import_tract={
            "suffix": "tractography",
            "scope": "my_tractography"
        },
        segmentation_params={'nb_streamlines': 10000})








.. GENERATED FROM PYTHON SOURCE LINES 256-259

Finally, to run the segmentation and extract tract profiles, we call
The `export_all` method. This creates all of the derivative outputs of
AFQ within the 'stanford_hardi/derivatives/afq' folder.

.. GENERATED FROM PYTHON SOURCE LINES 259-263

.. code-block:: Python


    my_afq.export_all()






.. rst-class:: sphx-glr-script-out

 .. code-block:: none


      0%|          | 0/179864.0 [00:00<?, ?it/s]
      0%|          | 363/179864.0 [00:00<00:49, 3624.41it/s]
      0%|          | 789/179864.0 [00:00<00:44, 3993.54it/s]
      1%|          | 1203/179864.0 [00:00<00:44, 4059.21it/s]
      1%|          | 1629/179864.0 [00:00<00:43, 4136.25it/s]
      1%|          | 2043/179864.0 [00:00<00:43, 4125.73it/s]
      1%|▏         | 2471/179864.0 [00:00<00:42, 4174.84it/s]
      2%|▏         | 2896/179864.0 [00:00<00:42, 4199.00it/s]
      2%|▏         | 3329/179864.0 [00:00<00:41, 4238.23it/s]
      2%|▏         | 3766/179864.0 [00:00<00:41, 4279.18it/s]
      2%|▏         | 4199/179864.0 [00:01<00:40, 4294.23it/s]
      3%|▎         | 4640/179864.0 [00:01<00:40, 4325.33it/s]
      3%|▎         | 5083/179864.0 [00:01<00:40, 4355.50it/s]
      3%|▎         | 5519/179864.0 [00:01<00:40, 4326.65it/s]
      3%|▎         | 5952/179864.0 [00:01<00:40, 4323.25it/s]
      4%|▎         | 6397/179864.0 [00:01<00:39, 4360.04it/s]
      4%|▍         | 6845/179864.0 [00:01<00:39, 4394.17it/s]
      4%|▍         | 7285/179864.0 [00:01<00:39, 4385.35it/s]
      4%|▍         | 7742/179864.0 [00:01<00:38, 4439.07it/s]
      5%|▍         | 8192/179864.0 [00:01<00:38, 4457.12it/s]
      5%|▍         | 8644/179864.0 [00:02<00:38, 4473.88it/s]
      5%|▌         | 9092/179864.0 [00:02<00:38, 4393.69it/s]
      5%|▌         | 9545/179864.0 [00:02<00:38, 4433.30it/s]
      6%|▌         | 9991/179864.0 [00:02<00:38, 4438.49it/s]
      6%|▌         | 10451/179864.0 [00:02<00:37, 4484.17it/s]
      6%|▌         | 10900/179864.0 [00:02<00:38, 4383.81it/s]
      6%|▋         | 11348/179864.0 [00:02<00:38, 4411.72it/s]
      7%|▋         | 11805/179864.0 [00:02<00:37, 4457.99it/s]
      7%|▋         | 12268/179864.0 [00:02<00:37, 4505.26it/s]
      7%|▋         | 12719/179864.0 [00:02<00:37, 4478.96it/s]
      7%|▋         | 13168/179864.0 [00:03<00:37, 4407.78it/s]
      8%|▊         | 13620/179864.0 [00:03<00:37, 4438.65it/s]
      8%|▊         | 14078/179864.0 [00:03<00:37, 4477.82it/s]
      8%|▊         | 14552/179864.0 [00:03<00:36, 4554.04it/s]
      8%|▊         | 15008/179864.0 [00:03<00:36, 4506.08it/s]
      9%|▊         | 15459/179864.0 [00:03<00:37, 4434.50it/s]
      9%|▉         | 15914/179864.0 [00:03<00:36, 4466.86it/s]
      9%|▉         | 16394/179864.0 [00:03<00:35, 4562.90it/s]
      9%|▉         | 16873/179864.0 [00:03<00:35, 4626.47it/s]
     10%|▉         | 17336/179864.0 [00:03<00:35, 4619.41it/s]
     10%|▉         | 17799/179864.0 [00:04<00:35, 4544.60it/s]
     10%|█         | 18254/179864.0 [00:04<00:36, 4478.85it/s]
     10%|█         | 18721/179864.0 [00:04<00:35, 4533.06it/s]
     11%|█         | 19194/179864.0 [00:04<00:35, 4590.49it/s]
     11%|█         | 19654/179864.0 [00:04<00:34, 4589.74it/s]
     11%|█         | 20114/179864.0 [00:04<00:34, 4581.19it/s]
     11%|█▏        | 20573/179864.0 [00:04<00:35, 4506.31it/s]
     12%|█▏        | 21026/179864.0 [00:04<00:35, 4510.86it/s]
     12%|█▏        | 21502/179864.0 [00:04<00:34, 4583.89it/s]
     12%|█▏        | 21967/179864.0 [00:04<00:34, 4603.45it/s]
     12%|█▏        | 22428/179864.0 [00:05<00:34, 4604.57it/s]
     13%|█▎        | 22889/179864.0 [00:05<00:34, 4571.02it/s]
     13%|█▎        | 23347/179864.0 [00:05<00:34, 4504.66it/s]
     13%|█▎        | 23810/179864.0 [00:05<00:34, 4541.26it/s]
     13%|█▎        | 24265/179864.0 [00:05<00:35, 4414.21it/s]
     14%|█▎        | 24728/179864.0 [00:05<00:34, 4476.61it/s]
     14%|█▍        | 25196/179864.0 [00:05<00:34, 4536.25it/s]
     14%|█▍        | 25658/179864.0 [00:05<00:33, 4561.01it/s]
     15%|█▍        | 26115/179864.0 [00:05<00:33, 4542.99it/s]
     15%|█▍        | 26570/179864.0 [00:05<00:33, 4537.27it/s]
     15%|█▌        | 27050/179864.0 [00:06<00:33, 4613.10it/s]
     15%|█▌        | 27530/179864.0 [00:06<00:32, 4667.77it/s]
     16%|█▌        | 28003/179864.0 [00:06<00:32, 4682.84it/s]
     16%|█▌        | 28478/179864.0 [00:06<00:32, 4701.80it/s]
     16%|█▌        | 28949/179864.0 [00:06<00:32, 4637.90it/s]
     16%|█▋        | 29414/179864.0 [00:06<00:33, 4552.09it/s]
     17%|█▋        | 29879/179864.0 [00:06<00:32, 4578.48it/s]
     17%|█▋        | 30361/179864.0 [00:06<00:32, 4645.71it/s]
     17%|█▋        | 30837/179864.0 [00:06<00:31, 4678.79it/s]
     17%|█▋        | 31320/179864.0 [00:06<00:31, 4722.88it/s]
     18%|█▊        | 31793/179864.0 [00:07<00:31, 4655.13it/s]
     18%|█▊        | 32259/179864.0 [00:07<00:31, 4627.47it/s]
     18%|█▊        | 32722/179864.0 [00:07<00:32, 4552.75it/s]
     18%|█▊        | 33180/179864.0 [00:07<00:32, 4556.58it/s]
     19%|█▊        | 33655/179864.0 [00:07<00:31, 4611.84it/s]
     19%|█▉        | 34117/179864.0 [00:07<00:32, 4519.94it/s]
     19%|█▉        | 34600/179864.0 [00:07<00:31, 4609.85it/s]
     19%|█▉        | 35065/179864.0 [00:07<00:31, 4620.13it/s]
     20%|█▉        | 35528/179864.0 [00:07<00:31, 4587.17it/s]
     20%|██        | 35988/179864.0 [00:08<00:31, 4585.39it/s]
     20%|██        | 36448/179864.0 [00:08<00:31, 4587.08it/s]
     21%|██        | 36937/179864.0 [00:08<00:30, 4676.66it/s]
     21%|██        | 37420/179864.0 [00:08<00:30, 4721.38it/s]
     21%|██        | 37908/179864.0 [00:08<00:29, 4768.07it/s]
     21%|██▏       | 38385/179864.0 [00:08<00:29, 4765.07it/s]
     22%|██▏       | 38862/179864.0 [00:08<00:30, 4580.11it/s]
     22%|██▏       | 39322/179864.0 [00:08<00:30, 4567.21it/s]
     22%|██▏       | 39781/179864.0 [00:08<00:30, 4572.23it/s]
     22%|██▏       | 40272/179864.0 [00:08<00:29, 4671.61it/s]
     23%|██▎       | 40759/179864.0 [00:09<00:29, 4727.43it/s]
     23%|██▎       | 41237/179864.0 [00:09<00:29, 4743.05it/s]
     23%|██▎       | 41712/179864.0 [00:09<00:29, 4733.53it/s]
     23%|██▎       | 42186/179864.0 [00:09<00:29, 4665.40it/s]
     24%|██▎       | 42654/179864.0 [00:09<00:29, 4668.41it/s]
     24%|██▍       | 43122/179864.0 [00:09<00:29, 4620.53it/s]
     24%|██▍       | 43586/179864.0 [00:09<00:29, 4623.23it/s]
     25%|██▍       | 44099/179864.0 [00:09<00:28, 4770.33it/s]
     25%|██▍       | 44577/179864.0 [00:09<00:28, 4746.28it/s]
     25%|██▌       | 45052/179864.0 [00:09<00:28, 4739.28it/s]
     25%|██▌       | 45527/179864.0 [00:10<00:28, 4649.42it/s]
     26%|██▌       | 45993/179864.0 [00:10<00:28, 4621.66it/s]
     26%|██▌       | 46469/179864.0 [00:10<00:28, 4659.35it/s]
     26%|██▌       | 46960/179864.0 [00:10<00:28, 4730.70it/s]
     26%|██▋       | 47472/179864.0 [00:10<00:27, 4844.52it/s]
     27%|██▋       | 47957/179864.0 [00:10<00:27, 4817.10it/s]
     27%|██▋       | 48439/179864.0 [00:10<00:27, 4781.34it/s]
     27%|██▋       | 48918/179864.0 [00:10<00:27, 4711.44it/s]
     27%|██▋       | 49390/179864.0 [00:10<00:27, 4665.62it/s]
     28%|██▊       | 49857/179864.0 [00:10<00:27, 4653.62it/s]
     28%|██▊       | 50355/179864.0 [00:11<00:27, 4749.89it/s]
     28%|██▊       | 50855/179864.0 [00:11<00:26, 4821.58it/s]
     29%|██▊       | 51338/179864.0 [00:11<00:27, 4743.93it/s]
     29%|██▉       | 51813/179864.0 [00:11<00:27, 4726.12it/s]
     29%|██▉       | 52286/179864.0 [00:11<00:27, 4710.14it/s]
     29%|██▉       | 52758/179864.0 [00:11<00:27, 4640.60it/s]
     30%|██▉       | 53223/179864.0 [00:11<00:27, 4568.93it/s]
     30%|██▉       | 53681/179864.0 [00:12<01:05, 1922.13it/s]
     30%|███       | 54165/179864.0 [00:12<00:53, 2357.50it/s]
     30%|███       | 54641/179864.0 [00:12<00:45, 2778.74it/s]
     31%|███       | 55116/179864.0 [00:12<00:39, 3174.02it/s]
     31%|███       | 55583/179864.0 [00:12<00:35, 3506.72it/s]
     31%|███       | 56025/179864.0 [00:12<00:33, 3667.99it/s]
     31%|███▏      | 56465/179864.0 [00:12<00:32, 3853.00it/s]
     32%|███▏      | 56921/179864.0 [00:12<00:30, 4040.65it/s]
     32%|███▏      | 57420/179864.0 [00:13<00:28, 4300.93it/s]
     32%|███▏      | 57902/179864.0 [00:13<00:27, 4446.27it/s]
     32%|███▏      | 58388/179864.0 [00:13<00:26, 4562.49it/s]
     33%|███▎      | 58872/179864.0 [00:13<00:26, 4642.91it/s]
     33%|███▎      | 59349/179864.0 [00:13<00:25, 4679.75it/s]
     33%|███▎      | 59825/179864.0 [00:13<00:26, 4595.68it/s]
     34%|███▎      | 60291/179864.0 [00:13<00:25, 4604.84it/s]
     34%|███▍      | 60796/179864.0 [00:13<00:25, 4735.05it/s]
     34%|███▍      | 61295/179864.0 [00:13<00:24, 4807.83it/s]
     34%|███▍      | 61786/179864.0 [00:13<00:24, 4837.65it/s]
     35%|███▍      | 62272/179864.0 [00:14<00:24, 4748.68it/s]
     35%|███▍      | 62760/179864.0 [00:14<00:24, 4786.77it/s]
     35%|███▌      | 63240/179864.0 [00:14<00:24, 4672.39it/s]
     35%|███▌      | 63709/179864.0 [00:14<00:25, 4633.18it/s]
     36%|███▌      | 64204/179864.0 [00:14<00:24, 4724.74it/s]
     36%|███▌      | 64697/179864.0 [00:14<00:24, 4780.43it/s]
     36%|███▌      | 65197/179864.0 [00:14<00:23, 4845.06it/s]
     37%|███▋      | 65683/179864.0 [00:14<00:23, 4836.04it/s]
     37%|███▋      | 66172/179864.0 [00:14<00:23, 4851.21it/s]
     37%|███▋      | 66658/179864.0 [00:15<00:23, 4732.43it/s]
     37%|███▋      | 67133/179864.0 [00:15<00:24, 4628.47it/s]
     38%|███▊      | 67597/179864.0 [00:15<00:24, 4622.83it/s]
     38%|███▊      | 68078/179864.0 [00:15<00:23, 4675.36it/s]
     38%|███▊      | 68562/179864.0 [00:15<00:23, 4722.00it/s]
     38%|███▊      | 69039/179864.0 [00:15<00:23, 4733.60it/s]
     39%|███▊      | 69525/179864.0 [00:15<00:23, 4770.85it/s]
     39%|███▉      | 70003/179864.0 [00:15<00:23, 4701.13it/s]
     39%|███▉      | 70474/179864.0 [00:15<00:23, 4617.85it/s]
     39%|███▉      | 70937/179864.0 [00:15<00:23, 4590.65it/s]
     40%|███▉      | 71419/179864.0 [00:16<00:23, 4655.91it/s]
     40%|███▉      | 71894/179864.0 [00:16<00:23, 4683.46it/s]
     40%|████      | 72370/179864.0 [00:16<00:22, 4706.04it/s]
     40%|████      | 72843/179864.0 [00:16<00:22, 4712.16it/s]
     41%|████      | 73315/179864.0 [00:16<00:23, 4623.95it/s]
     41%|████      | 73778/179864.0 [00:16<00:23, 4561.88it/s]
     41%|████▏     | 74235/179864.0 [00:16<00:23, 4515.24it/s]
     42%|████▏     | 74703/179864.0 [00:16<00:23, 4562.36it/s]
     42%|████▏     | 75173/179864.0 [00:16<00:22, 4601.84it/s]
     42%|████▏     | 75647/179864.0 [00:16<00:22, 4640.81it/s]
     42%|████▏     | 76119/179864.0 [00:17<00:22, 4662.75it/s]
     43%|████▎     | 76586/179864.0 [00:17<00:22, 4627.77it/s]
     43%|████▎     | 77049/179864.0 [00:17<00:22, 4528.23it/s]
     43%|████▎     | 77503/179864.0 [00:17<00:22, 4469.17it/s]
     43%|████▎     | 77963/179864.0 [00:17<00:22, 4506.95it/s]
     44%|████▎     | 78445/179864.0 [00:17<00:22, 4598.49it/s]
     44%|████▍     | 78906/179864.0 [00:17<00:21, 4591.28it/s]
     44%|████▍     | 79379/179864.0 [00:17<00:21, 4632.24it/s]
     44%|████▍     | 79843/179864.0 [00:17<00:21, 4594.56it/s]
     45%|████▍     | 80303/179864.0 [00:17<00:21, 4535.56it/s]
     45%|████▍     | 80757/179864.0 [00:18<00:22, 4467.81it/s]
     45%|████▌     | 81205/179864.0 [00:18<00:22, 4382.33it/s]
     45%|████▌     | 81658/179864.0 [00:18<00:22, 4425.20it/s]
     46%|████▌     | 82122/179864.0 [00:18<00:21, 4486.37it/s]
     46%|████▌     | 82586/179864.0 [00:18<00:21, 4530.16it/s]
     46%|████▌     | 83040/179864.0 [00:18<00:21, 4498.90it/s]
     46%|████▋     | 83491/179864.0 [00:18<00:21, 4416.82it/s]
     47%|████▋     | 83961/179864.0 [00:18<00:21, 4498.47it/s]
     47%|████▋     | 84413/179864.0 [00:18<00:21, 4502.32it/s]
     47%|████▋     | 84867/179864.0 [00:18<00:21, 4510.47it/s]
     47%|████▋     | 85320/179864.0 [00:19<00:20, 4512.98it/s]
     48%|████▊     | 85777/179864.0 [00:19<00:20, 4527.65it/s]
     48%|████▊     | 86238/179864.0 [00:19<00:20, 4551.79it/s]
     48%|████▊     | 86694/179864.0 [00:19<00:20, 4507.83it/s]
     48%|████▊     | 87145/179864.0 [00:19<00:20, 4442.18it/s]
     49%|████▊     | 87593/179864.0 [00:19<00:20, 4449.99it/s]
     49%|████▉     | 88039/179864.0 [00:19<00:20, 4446.20it/s]
     49%|████▉     | 88518/179864.0 [00:19<00:20, 4547.90it/s]
     49%|████▉     | 89011/179864.0 [00:19<00:19, 4660.37it/s]
     50%|████▉     | 89499/179864.0 [00:19<00:19, 4723.61it/s]
     50%|█████     | 89983/179864.0 [00:20<00:18, 4756.55it/s]
     50%|█████     | 90459/179864.0 [00:20<00:19, 4622.65it/s]
     51%|█████     | 90923/179864.0 [00:20<00:19, 4578.71it/s]
     51%|█████     | 91382/179864.0 [00:20<00:19, 4494.44it/s]
     51%|█████     | 91833/179864.0 [00:20<00:19, 4475.84it/s]
     51%|█████▏    | 92312/179864.0 [00:20<00:19, 4566.31it/s]
     52%|█████▏    | 92805/179864.0 [00:20<00:18, 4671.23it/s]
     52%|█████▏    | 93277/179864.0 [00:20<00:18, 4685.13it/s]
     52%|█████▏    | 93754/179864.0 [00:20<00:18, 4708.42it/s]
     52%|█████▏    | 94226/179864.0 [00:21<00:18, 4693.93it/s]
     53%|█████▎    | 94696/179864.0 [00:21<00:18, 4573.88it/s]
     53%|█████▎    | 95155/179864.0 [00:21<00:18, 4546.43it/s]
     53%|█████▎    | 95621/179864.0 [00:21<00:18, 4579.20it/s]
     53%|█████▎    | 96098/179864.0 [00:21<00:18, 4633.24it/s]
     54%|█████▎    | 96562/179864.0 [00:21<00:17, 4632.19it/s]
     54%|█████▍    | 97026/179864.0 [00:21<00:17, 4623.25it/s]
     54%|█████▍    | 97489/179864.0 [00:21<00:18, 4530.71it/s]
     54%|█████▍    | 97943/179864.0 [00:21<00:18, 4492.67it/s]
     55%|█████▍    | 98393/179864.0 [00:21<00:18, 4454.99it/s]
     55%|█████▍    | 98839/179864.0 [00:22<00:18, 4450.19it/s]
     55%|█████▌    | 99307/179864.0 [00:22<00:17, 4517.98it/s]
     55%|█████▌    | 99782/179864.0 [00:22<00:17, 4585.91it/s]
     56%|█████▌    | 100254/179864.0 [00:22<00:17, 4625.08it/s]
     56%|█████▌    | 100722/179864.0 [00:22<00:17, 4639.18it/s]
     56%|█████▋    | 101187/179864.0 [00:22<00:17, 4405.52it/s]
     57%|█████▋    | 101643/179864.0 [00:22<00:17, 4449.69it/s]
     57%|█████▋    | 102092/179864.0 [00:22<00:17, 4460.65it/s]
     57%|█████▋    | 102557/179864.0 [00:22<00:17, 4516.13it/s]
     57%|█████▋    | 103036/179864.0 [00:22<00:16, 4594.16it/s]
     58%|█████▊    | 103509/179864.0 [00:23<00:16, 4632.51it/s]
     58%|█████▊    | 103973/179864.0 [00:23<00:16, 4558.61it/s]
     58%|█████▊    | 104435/179864.0 [00:23<00:16, 4575.35it/s]
     58%|█████▊    | 104894/179864.0 [00:23<00:16, 4494.42it/s]
     59%|█████▊    | 105346/179864.0 [00:23<00:16, 4500.80it/s]
     59%|█████▉    | 105808/179864.0 [00:23<00:16, 4532.78it/s]
     59%|█████▉    | 106291/179864.0 [00:23<00:15, 4620.14it/s]
     59%|█████▉    | 106774/179864.0 [00:23<00:15, 4680.80it/s]
     60%|█████▉    | 107250/179864.0 [00:23<00:15, 4704.24it/s]
     60%|█████▉    | 107721/179864.0 [00:23<00:15, 4698.27it/s]
     60%|██████    | 108191/179864.0 [00:24<00:15, 4638.67it/s]
     60%|██████    | 108656/179864.0 [00:24<00:15, 4604.49it/s]
     61%|██████    | 109126/179864.0 [00:24<00:15, 4629.61it/s]
     61%|██████    | 109606/179864.0 [00:24<00:15, 4678.68it/s]
     61%|██████    | 110090/179864.0 [00:24<00:14, 4724.96it/s]
     61%|██████▏   | 110579/179864.0 [00:24<00:14, 4772.50it/s]
     62%|██████▏   | 111062/179864.0 [00:24<00:14, 4788.66it/s]
     62%|██████▏   | 111547/179864.0 [00:24<00:14, 4806.71it/s]
     62%|██████▏   | 112028/179864.0 [00:24<00:14, 4761.36it/s]
     63%|██████▎   | 112505/179864.0 [00:24<00:14, 4672.09it/s]
     63%|██████▎   | 112985/179864.0 [00:25<00:14, 4708.04it/s]
     63%|██████▎   | 113469/179864.0 [00:25<00:13, 4745.99it/s]
     63%|██████▎   | 113971/179864.0 [00:25<00:13, 4825.66it/s]
     64%|██████▎   | 114454/179864.0 [00:25<00:13, 4808.14it/s]
     64%|██████▍   | 114938/179864.0 [00:25<00:13, 4816.84it/s]
     64%|██████▍   | 115420/179864.0 [00:25<00:13, 4773.38it/s]
     64%|██████▍   | 115898/179864.0 [00:25<00:13, 4730.58it/s]
     65%|██████▍   | 116384/179864.0 [00:25<00:13, 4765.54it/s]
     65%|██████▍   | 116877/179864.0 [00:25<00:13, 4813.02it/s]
     65%|██████▌   | 117363/179864.0 [00:26<00:12, 4826.84it/s]
     66%|██████▌   | 117855/179864.0 [00:26<00:12, 4854.57it/s]
     66%|██████▌   | 118359/179864.0 [00:26<00:12, 4909.23it/s]
     66%|██████▌   | 118851/179864.0 [00:26<00:12, 4823.94it/s]
     66%|██████▋   | 119334/179864.0 [00:26<00:12, 4798.02it/s]
     67%|██████▋   | 119828/179864.0 [00:26<00:12, 4837.99it/s]
     67%|██████▋   | 120336/179864.0 [00:26<00:12, 4909.16it/s]
     67%|██████▋   | 120828/179864.0 [00:26<00:12, 4886.04it/s]
     67%|██████▋   | 121317/179864.0 [00:26<00:12, 4870.14it/s]
     68%|██████▊   | 121829/179864.0 [00:26<00:11, 4943.17it/s]
     68%|██████▊   | 122324/179864.0 [00:27<00:11, 4835.66it/s]
     68%|██████▊   | 122809/179864.0 [00:27<00:11, 4806.19it/s]
     69%|██████▊   | 123299/179864.0 [00:27<00:11, 4833.47it/s]
     69%|██████▉   | 123799/179864.0 [00:27<00:11, 4879.98it/s]
     69%|██████▉   | 124293/179864.0 [00:27<00:11, 4896.21it/s]
     69%|██████▉   | 124783/179864.0 [00:27<00:11, 4881.98it/s]
     70%|██████▉   | 125274/179864.0 [00:27<00:11, 4888.14it/s]
     70%|██████▉   | 125763/179864.0 [00:27<00:11, 4802.73it/s]
     70%|███████   | 126244/179864.0 [00:27<00:11, 4771.38it/s]
     70%|███████   | 126742/179864.0 [00:27<00:10, 4831.74it/s]
     71%|███████   | 127242/179864.0 [00:28<00:10, 4879.21it/s]
     71%|███████   | 127733/179864.0 [00:28<00:10, 4886.56it/s]
     71%|███████▏  | 128222/179864.0 [00:28<00:10, 4827.53it/s]
     72%|███████▏  | 128706/179864.0 [00:28<00:10, 4811.09it/s]
     72%|███████▏  | 129188/179864.0 [00:28<00:10, 4707.46it/s]
     72%|███████▏  | 129660/179864.0 [00:28<00:10, 4679.15it/s]
     72%|███████▏  | 130147/179864.0 [00:28<00:10, 4733.55it/s]
     73%|███████▎  | 130630/179864.0 [00:28<00:10, 4760.77it/s]
     73%|███████▎  | 131131/179864.0 [00:28<00:10, 4829.57it/s]
     73%|███████▎  | 131615/179864.0 [00:28<00:10, 4788.45it/s]
     73%|███████▎  | 132095/179864.0 [00:29<00:10, 4772.67it/s]
     74%|███████▎  | 132573/179864.0 [00:29<00:10, 4643.47it/s]
     74%|███████▍  | 133039/179864.0 [00:29<00:10, 4603.51it/s]
     74%|███████▍  | 133503/179864.0 [00:29<00:10, 4611.89it/s]
     74%|███████▍  | 133974/179864.0 [00:29<00:09, 4639.53it/s]
     75%|███████▍  | 134453/179864.0 [00:29<00:09, 4683.14it/s]
     75%|███████▌  | 134927/179864.0 [00:29<00:09, 4697.99it/s]
     75%|███████▌  | 135418/179864.0 [00:29<00:09, 4759.66it/s]
     76%|███████▌  | 135895/179864.0 [00:29<00:09, 4662.36it/s]
     76%|███████▌  | 136362/179864.0 [00:29<00:09, 4576.02it/s]
     76%|███████▌  | 136834/179864.0 [00:30<00:09, 4615.48it/s]
     76%|███████▋  | 137325/179864.0 [00:30<00:09, 4700.19it/s]
     77%|███████▋  | 137800/179864.0 [00:30<00:08, 4714.40it/s]
     77%|███████▋  | 138278/179864.0 [00:30<00:08, 4731.88it/s]
     77%|███████▋  | 138752/179864.0 [00:30<00:08, 4677.27it/s]
     77%|███████▋  | 139221/179864.0 [00:30<00:08, 4576.57it/s]
     78%|███████▊  | 139680/179864.0 [00:30<00:08, 4487.83it/s]
     78%|███████▊  | 140157/179864.0 [00:30<00:08, 4569.19it/s]
     78%|███████▊  | 140636/179864.0 [00:30<00:08, 4631.64it/s]
     78%|███████▊  | 141111/179864.0 [00:31<00:08, 4664.44it/s]
     79%|███████▊  | 141580/179864.0 [00:31<00:08, 4669.94it/s]
     79%|███████▉  | 142048/179864.0 [00:31<00:08, 4657.66it/s]
     79%|███████▉  | 142515/179864.0 [00:31<00:08, 4515.97it/s]
     79%|███████▉  | 142973/179864.0 [00:31<00:08, 4533.22it/s]
     80%|███████▉  | 143436/179864.0 [00:31<00:07, 4561.59it/s]
     80%|████████  | 143904/179864.0 [00:31<00:07, 4596.60it/s]
     80%|████████  | 144394/179864.0 [00:31<00:07, 4686.00it/s]
     81%|████████  | 144877/179864.0 [00:31<00:07, 4728.70it/s]
     81%|████████  | 145351/179864.0 [00:31<00:07, 4666.95it/s]
     81%|████████  | 145819/179864.0 [00:32<00:07, 4620.51it/s]
     81%|████████▏ | 146317/179864.0 [00:32<00:07, 4725.21it/s]
     82%|████████▏ | 146796/179864.0 [00:32<00:06, 4744.22it/s]
     82%|████████▏ | 147284/179864.0 [00:32<00:06, 4783.39it/s]
     82%|████████▏ | 147765/179864.0 [00:32<00:06, 4790.63it/s]
     82%|████████▏ | 148245/179864.0 [00:32<00:06, 4738.83it/s]
     83%|████████▎ | 148720/179864.0 [00:32<00:06, 4632.90it/s]
     83%|████████▎ | 149202/179864.0 [00:32<00:06, 4686.99it/s]
     83%|████████▎ | 149682/179864.0 [00:32<00:06, 4716.99it/s]
     83%|████████▎ | 150162/179864.0 [00:32<00:06, 4738.46it/s]
     84%|████████▍ | 150640/179864.0 [00:33<00:06, 4748.88it/s]
     84%|████████▍ | 151116/179864.0 [00:33<00:06, 4695.35it/s]
     84%|████████▍ | 151586/179864.0 [00:33<00:06, 4657.71it/s]
     85%|████████▍ | 152052/179864.0 [00:33<00:05, 4655.87it/s]
     85%|████████▍ | 152518/179864.0 [00:33<00:05, 4624.64it/s]
     85%|████████▌ | 152991/179864.0 [00:33<00:05, 4655.61it/s]
     85%|████████▌ | 153457/179864.0 [00:33<00:05, 4632.48it/s]
     86%|████████▌ | 153921/179864.0 [00:33<00:05, 4594.78it/s]
     86%|████████▌ | 154381/179864.0 [00:33<00:05, 4547.24it/s]
     86%|████████▌ | 154836/179864.0 [00:33<00:05, 4487.23it/s]
     86%|████████▋ | 155293/179864.0 [00:34<00:05, 4510.01it/s]
     87%|████████▋ | 155765/179864.0 [00:34<00:05, 4571.10it/s]
     87%|████████▋ | 156223/179864.0 [00:34<00:05, 4561.63it/s]
     87%|████████▋ | 156680/179864.0 [00:34<00:05, 4548.65it/s]
     87%|████████▋ | 157135/179864.0 [00:34<00:05, 4537.48it/s]
     88%|████████▊ | 157589/179864.0 [00:34<00:04, 4456.98it/s]
     88%|████████▊ | 158050/179864.0 [00:34<00:04, 4501.63it/s]
     88%|████████▊ | 158534/179864.0 [00:34<00:04, 4601.35it/s]
     88%|████████▊ | 159008/179864.0 [00:34<00:04, 4639.39it/s]
     89%|████████▊ | 159473/179864.0 [00:34<00:04, 4587.73it/s]
     89%|████████▉ | 159933/179864.0 [00:35<00:04, 4482.36it/s]
     89%|████████▉ | 160382/179864.0 [00:35<00:04, 4422.53it/s]
     89%|████████▉ | 160846/179864.0 [00:35<00:04, 4485.33it/s]
     90%|████████▉ | 161325/179864.0 [00:35<00:04, 4572.94it/s]
     90%|████████▉ | 161784/179864.0 [00:35<00:03, 4576.61it/s]
     90%|█████████ | 162243/179864.0 [00:35<00:03, 4542.07it/s]
     90%|█████████ | 162698/179864.0 [00:35<00:03, 4454.55it/s]
     91%|█████████ | 163152/179864.0 [00:35<00:03, 4476.80it/s]
     91%|█████████ | 163619/179864.0 [00:35<00:03, 4532.49it/s]
     91%|█████████ | 164078/179864.0 [00:35<00:03, 4547.94it/s]
     91%|█████████▏| 164534/179864.0 [00:36<00:03, 4502.53it/s]
     92%|█████████▏| 164985/179864.0 [00:36<00:03, 4453.56it/s]
     92%|█████████▏| 165455/179864.0 [00:36<00:03, 4524.73it/s]
     92%|█████████▏| 165919/179864.0 [00:36<00:03, 4557.40it/s]
     93%|█████████▎| 166383/179864.0 [00:36<00:02, 4580.00it/s]
     93%|█████████▎| 166842/179864.0 [00:36<00:02, 4542.95it/s]
     93%|█████████▎| 167297/179864.0 [00:36<00:02, 4384.94it/s]
     93%|█████████▎| 167762/179864.0 [00:36<00:02, 4459.95it/s]
     94%|█████████▎| 168228/179864.0 [00:36<00:02, 4518.40it/s]
     94%|█████████▍| 168681/179864.0 [00:37<00:02, 4492.52it/s]
     94%|█████████▍| 169131/179864.0 [00:37<00:02, 4445.88it/s]
     94%|█████████▍| 169577/179864.0 [00:37<00:02, 4446.14it/s]
     95%|█████████▍| 170040/179864.0 [00:37<00:02, 4500.25it/s]
     95%|█████████▍| 170494/179864.0 [00:37<00:02, 4511.82it/s]
     95%|█████████▌| 170946/179864.0 [00:37<00:01, 4470.01it/s]
     95%|█████████▌| 171394/179864.0 [00:37<00:01, 4424.23it/s]
     96%|█████████▌| 171842/179864.0 [00:37<00:01, 4438.44it/s]
     96%|█████████▌| 172287/179864.0 [00:37<00:01, 4402.93it/s]
     96%|█████████▌| 172728/179864.0 [00:37<00:01, 4380.76it/s]
     96%|█████████▋| 173167/179864.0 [00:38<00:01, 4368.72it/s]
     97%|█████████▋| 173607/179864.0 [00:38<00:01, 4377.25it/s]
     97%|█████████▋| 174058/179864.0 [00:38<00:01, 4415.64it/s]
     97%|█████████▋| 174500/179864.0 [00:38<00:01, 4392.89it/s]
     97%|█████████▋| 174950/179864.0 [00:38<00:01, 4423.87it/s]
     98%|█████████▊| 175406/179864.0 [00:38<00:00, 4463.60it/s]
     98%|█████████▊| 175853/179864.0 [00:38<00:00, 4431.88it/s]
     98%|█████████▊| 176297/179864.0 [00:38<00:00, 4400.14it/s]
     98%|█████████▊| 176741/179864.0 [00:38<00:00, 4410.46it/s]
     99%|█████████▊| 177183/179864.0 [00:38<00:00, 4388.46it/s]
     99%|█████████▉| 177622/179864.0 [00:39<00:00, 4344.20it/s]
     99%|█████████▉| 178089/179864.0 [00:39<00:00, 4437.03it/s]
     99%|█████████▉| 178535/179864.0 [00:39<00:00, 4442.82it/s]
    100%|█████████▉| 178997/179864.0 [00:39<00:00, 4495.45it/s]
    100%|█████████▉| 179493/179864.0 [00:39<00:00, 4632.78it/s]
    100%|██████████| 179864/179864.0 [00:39<00:00, 4549.02it/s]
    Optimizing level 2 [max iter: 10000]
    Optimizing level 1 [max iter: 1000]
    Optimizing level 0 [max iter: 100]
    Optimizing level 2 [max iter: 10000]
    Optimizing level 1 [max iter: 1000]
    Optimizing level 0 [max iter: 100]
    Optimizing level 2 [max iter: 10000]
    Optimizing level 1 [max iter: 1000]
    Optimizing level 0 [max iter: 100]

      0%|          | 0/7 [00:00<?, ?it/s]
     71%|███████▏  | 5/7 [00:00<00:00, 43.98it/s]
    100%|██████████| 7/7 [00:00<00:00, 46.68it/s]

      0%|          | 0/7 [00:00<?, ?it/s]
     14%|█▍        | 1/7 [00:00<00:02,  2.77it/s]
     86%|████████▌ | 6/7 [00:00<00:00, 15.76it/s]
    100%|██████████| 7/7 [00:00<00:00, 14.35it/s]




.. GENERATED FROM PYTHON SOURCE LINES 264-270

A few common issues that can hinder BIDS from working properly are:

1. Faulty `dataset_description.json` file. You need to make sure that the
   file contains the right names for the pipeline. See above for an example
   of that.
2. File naming convention doesn't uniquely identify file with bids filters.

.. GENERATED FROM PYTHON SOURCE LINES 273-277

The outputs of AFQ are also BIDS compatible. However, we use some
custom entities, which need to be loaded. Here we demonstrate how
to load the afq entities with get_afq_bids_entities_fname()
and then show all files with the key-value pair recogmethod-AFQ

.. GENERATED FROM PYTHON SOURCE LINES 277-283

.. code-block:: Python


    layout = BIDSLayout(bids_path)
    layout.add_derivatives(
        f'{bids_path}/derivatives/afq',
        config=['bids', 'derivatives', get_afq_bids_entities_fname()])
    print(layout.get(recogmethod="AFQ", return_type="filename"))




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    ['/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-ForcepsMinor_tractography.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-ForcepsMinor_tractography.trk', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftArcuate_tractography.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftArcuate_tractography.trk', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftCorticospinal_tractography.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftCorticospinal_tractography.trk', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftSuperiorLongitudinal_tractography.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftSuperiorLongitudinal_tractography.trk', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightArcuate_tractography.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightArcuate_tractography.trk', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightCorticospinal_tractography.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightCorticospinal_tractography.trk', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightSuperiorLongitudinal_tractography.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightSuperiorLongitudinal_tractography.trk', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-ForcepsMinorviz_dwi', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-medianBundleLengths_dwi.csv', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-medianBundleLengths_dwi.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-profiles_dwi.csv', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-profiles_dwi.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-slCount_dwi.csv', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-slCount_dwi.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-viz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-viz_dwi.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_tractography.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_tractography.trk', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/tract_profile_plots/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_model-dtifa_desc-vizprofile_dwi.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/tract_profile_plots/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_model-dtifa_desc-vizprofile_dwi.png', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/tract_profile_plots/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_model-dtifa_desc-vizprofile_dwi.svg', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/tract_profile_plots/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_model-dtimd_desc-vizprofile_dwi.json', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/tract_profile_plots/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_model-dtimd_desc-vizprofile_dwi.png', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/tract_profile_plots/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_model-dtimd_desc-vizprofile_dwi.svg', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-ForcepsMinorviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftArcuateviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftCorticospinalviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftSuperiorLongitudinalviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightArcuateviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightCorticospinalviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightSuperiorLongitudinalviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_core_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-ForcepsMinorviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_core_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftArcuateviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_core_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftCorticospinalviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_core_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-LeftSuperiorLongitudinalviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_core_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightArcuateviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_core_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightCorticospinalviz_dwi.html', '/home/runner/AFQ_data/stanford_hardi/derivatives/afq/sub-01/ses-01/viz_core_bundles/sub-01_ses-01_coordsys-RASMM_trkmethod-probCSD_recogmethod-AFQ_desc-RightSuperiorLongitudinalviz_dwi.html']





.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (16 minutes 9.340 seconds)

**Estimated memory usage:**  2176 MB


.. _sphx_glr_download_tutorials_tutorial_examples_plot_002_bids_layout.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_002_bids_layout.ipynb <plot_002_bids_layout.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_002_bids_layout.py <plot_002_bids_layout.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_002_bids_layout.zip <plot_002_bids_layout.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_