{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "abd1da02",
   "metadata": {},
   "source": [
    "# Re-running pyAFQ\n",
    "Sometimes you want to change arguments and re-run pyAFQ. This could be to\n",
    "try different parameters, to run on changed data, or re-run after updating\n",
    "parameters due to an error.\n",
    "\n",
    "pyAFQ saves derivatives as it goes, so if you re-run pyAFQ after changing\n",
    "parameters, it could use derivatives from previous runs with the\n",
    "old parameters.\n",
    "\n",
    "To solve this, use the `myafq.clobber()` or `myafq.cmd_outputs()` methods.\n",
    "These are equivalent methods that delete previous derivatives so you can\n",
    "re-run your pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b97a99c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from AFQ.api.group import GroupAFQ\n",
    "import AFQ.data.fetch as afd\n",
    "import AFQ.definitions.image as afm\n",
    "import os.path as op\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79398355",
   "metadata": {},
   "source": [
    "We start with some example data generated from HBN, then setup our\n",
    "`myafq` object to demonstrate the clobber method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8b5d9ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "afd.fetch_hbn_preproc([\"NDARAA948VFH\"])\n",
    "\n",
    "tracking_params = dict(n_seeds=100,\n",
    "                       random_seeds=True,\n",
    "                       rng_seed=2022,\n",
    "                       trx=True)\n",
    "\n",
    "pve = afm.PVEImages(\n",
    "    afm.ImageFile(\n",
    "        suffix=\"probseg\", filters={\"scope\": \"qsiprep\", \"label\": \"CSF\"}),\n",
    "    afm.ImageFile(\n",
    "        suffix=\"probseg\", filters={\"scope\": \"qsiprep\", \"label\": \"GM\"}),\n",
    "    afm.ImageFile(\n",
    "        suffix=\"probseg\", filters={\"scope\": \"qsiprep\", \"label\": \"WM\"}))\n",
    "\n",
    "myafq = GroupAFQ(\n",
    "    bids_path=op.join(afd.afq_home, 'HBN'),\n",
    "    dwi_preproc_pipeline='qsiprep',\n",
    "    t1_preproc_pipeline='qsiprep',\n",
    "    participant_labels=['NDARAA948VFH'],\n",
    "    pve=pve,\n",
    "    tracking_params=tracking_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02cd5379",
   "metadata": {},
   "source": [
    "## Delete Everything\n",
    "\n",
    "To delete all pyAFQ outputs in the output directory, simply call:\n",
    "\n",
    "```python\n",
    "myafq.cmd_outputs()\n",
    "```\n",
    "\n",
    "or:\n",
    "\n",
    "```python\n",
    "myafq.clobber()\n",
    "```\n",
    "\n",
    "This is equivalent to running `rm -r` on all pyAFQ outputs. After this,\n",
    "you can re-run your pipeline from scratch.\n",
    "\n",
    "Here, we will delete everything and re-run with a different b0 threshold.\n",
    "The `b0_threshold` determines which b-values are considered b0.\n",
    "The default is 50."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27a9f555",
   "metadata": {},
   "outputs": [],
   "source": [
    "myafq.cmd_outputs()\n",
    "\n",
    "myafq = GroupAFQ(\n",
    "    bids_path=op.join(afd.afq_home, 'HBN'),\n",
    "    dwi_preproc_pipeline='qsiprep',\n",
    "    t1_preproc_pipeline='qsiprep',\n",
    "    participant_labels=['NDARAA948VFH'],\n",
    "    b0_threshold=100,\n",
    "    tracking_params=tracking_params,\n",
    "    pve=pve)\n",
    "\n",
    "myafq.export(\"b0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b13f9e43",
   "metadata": {},
   "source": [
    "## Delete Some Things\n",
    "\n",
    "To delete only specific types of derivatives while preserving others,\n",
    "use the `dependent_on` parameter:\n",
    "\n",
    "```python\n",
    "# Delete only tractography-dependent files\n",
    "myafq.cmd_outputs(dependent_on=\"track\")\n",
    "\n",
    "# Delete only bundle recognition-dependent files\n",
    "myafq.cmd_outputs(dependent_on=\"recog\")\n",
    "\n",
    "# Delete only profiling-dependent files\n",
    "myafq.cmd_outputs(dependent_on=\"prof\")\n",
    "```\n",
    "\n",
    "You can also specify exceptions — files to preserve:\n",
    "\n",
    "```python\n",
    "# Delete all outputs except the tractography\n",
    "myafq.cmd_outputs(exceptions=[\"streamlines\"])\n",
    "```\n",
    "\n",
    "Here, we will change the tractography parameters, but keep all derivatives\n",
    "not dependent on tractography. Typically this means keeping the mapping from\n",
    "MNI space and fitted models, but deleting recognized bundles and tract profiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92d09976",
   "metadata": {},
   "outputs": [],
   "source": [
    "myafq.clobber(dependent_on=\"track\")\n",
    "\n",
    "tracking_params = dict(n_seeds=100,\n",
    "                       random_seeds=True,\n",
    "                       max_angle=60,\n",
    "                       rng_seed=12,\n",
    "                       trx=True)\n",
    "\n",
    "myafq = GroupAFQ(\n",
    "    bids_path=op.join(afd.afq_home, 'HBN'),\n",
    "    dwi_preproc_pipeline='qsiprep',\n",
    "    t1_preproc_pipeline='qsiprep',\n",
    "    participant_labels=['NDARAA948VFH'],\n",
    "    b0_threshold=100,\n",
    "    tracking_params=tracking_params,\n",
    "    pve=pve)\n",
    "\n",
    "myafq.export(\"streamlines\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7f79e51",
   "metadata": {},
   "source": [
    "## Move Some Things\n",
    "\n",
    "The `cmd_outputs` method is flexible and can perform other file operations\n",
    "besides deletion. For example, to copy files:\n",
    "\n",
    "```python\n",
    "# Copy only files dependent on tractography\n",
    "myafq.cmd_outputs(\n",
    "        cmd=\"cp\",\n",
    "        dependent_on=\"track\",\n",
    "        suffix=\"/path/to/backup/\")\n",
    "\n",
    "# Note: The method automatically adds `-r` for \"cp\" and \"rm\" operations.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c70e393",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create backup directory\n",
    "backup_dir = op.join(afd.afq_home, \"HBN_backup\")\n",
    "os.makedirs(backup_dir, exist_ok=True)\n",
    "\n",
    "# Move the outputs of AFQ to this directory\n",
    "myafq.cmd_outputs(cmd=\"mv\", suffix=backup_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "935d45f9",
   "metadata": {},
   "source": [
    "## How It Works\n",
    "\n",
    "The method works by:\n",
    "1. Identifying all pyAFQ outputs in the output directory\n",
    "2. Filtering based on the `dependent_on` parameter (if provided)\n",
    "3. Removing any files listed in `exceptions`\n",
    "4. Executing the specified command (`rm`, `mv`, `cp` etc.) on the remaining files\n",
    "5. Resetting the workflow to ensure subsequent runs regenerate affected derivatives\n",
    "\n",
    "We plan to automate this process in the future.\n",
    "\n",
    ":::{only} html\n",
    "{download}`Download as Jupyter Notebook <plot_003_rerun.ipynb>`\n",
    ":::"
   ]
  }
 ],
 "metadata": {
  "file_format": "mystnb",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  },
  "mystnb": {
   "execution_mode": "off"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}