06_fine_tuning_bert.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uk1Wm4ht7L8I"
      },
      "source": [
        "## Fine Tuning BERT (GLUE - MRPC)\n",
        "\n",
        "This notebook demonstrates the process of fine-tuning [BERT-base (Bidirectional Encoder Representations from Transformers)](https://arxiv.org/abs/1810.04805) for the Microsoft Research Paraphrase Corpus (MRPC) task, part of the [General Language Understanding Evaluation (GLUE)](https://gluebenchmark.com/) benchmark. BERT-base is a transformer model pre-trained on a large corpus of English text using self-supervised learning. Its pre-training involves two key tasks: **Masked Language Modeling (MLM)**, where it predicts randomly masked words in a sentence, and **Next Sentence Prediction (NSP)**, where it determines if two sentences are consecutive in the original text. This approach allows BERT to learn bidirectional representations of language, capturing complex contextual relationships.\n",
        "\n",
        "While BERT's pre-training provides a robust understanding of language, it requires fine-tuning on specific tasks that use the whole sentence (potentially masked) such as sequence classification, token classification, question answering, and paraphrase identification - as in our implementation. This fine-tuning process adapts BERT's general language understanding to the specific nuances of the MRPC task, which involves determining whether two given sentences are paraphrases of each other.\n",
        "\n",
        "In this notebook, we'll walk through the steps of preparing the MRPC dataset (incl. tokenization and dynamic padding), training the model with [Hugging Face Transformers](https://huggingface.co/docs/transformers/index), and tracking its performance on the paraphrase identification task with the [Weights & Biases](https://wandb.ai/site) framework. Note that more advanced techniques for fine-tuning will be covered in another notebook."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<p align=\"center\">\n",
        "  <img src=\"./static/fine_tuning_process.png\">\n",
        "</p>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ouAsqU6E7xGp"
      },
      "outputs": [],
      "source": [
        "%pip install transformers datasets evaluate --quiet | tail -n 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cq2S2AbNErzx"
      },
      "source": [
        "We will use [Weights & Biases](https://wandb.ai/site) to track and visualize our fine tuning process:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ak1kAlaRE3zP",
        "outputId": "642cef6b-5b73-427d-facf-72752ed710be"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.7/62.7 kB 6.6 MB/s eta 0:00:00\n"
          ]
        }
      ],
      "source": [
        "%pip install wandb --quiet | tail -n 1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Krh6_aAYFJrU"
      },
      "outputs": [],
      "source": [
        "import wandb\n",
        "wandb.login()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ulGxR50Y7L8K"
      },
      "source": [
        "### 1. Preparing Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OsA0nGVJ7L8K"
      },
      "source": [
        "#### 1.1 Loading MRPC"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "31w_3PC47L8L"
      },
      "source": [
        "The MRPC dataset is one of the 10 datasets composing the [GLUE benchmark](https://gluebenchmark.com/), an academic benchmark used to measure the performance of ML models across 10 different text classification tasks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lsUpUlen7L8L"
      },
      "outputs": [],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "raw_datasets = load_dataset(\"glue\", \"mrpc\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QC_6q_pjGzz0",
        "outputId": "fdafce1f-433e-4840-be74-8426744a027a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    train: Dataset({\n",
              "        features: ['sentence1', 'sentence2', 'label', 'idx'],\n",
              "        num_rows: 3668\n",
              "    })\n",
              "    validation: Dataset({\n",
              "        features: ['sentence1', 'sentence2', 'label', 'idx'],\n",
              "        num_rows: 408\n",
              "    })\n",
              "    test: Dataset({\n",
              "        features: ['sentence1', 'sentence2', 'label', 'idx'],\n",
              "        num_rows: 1725\n",
              "    })\n",
              "})"
            ]
          },
          "execution_count": 18,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "raw_datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bIxukcUD-Q-j"
      },
      "source": [
        "In the random sample [42], we can see that `sentence1` and `sentence2` are equivalent (label == 1):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q1_FSsvX7L8M",
        "outputId": "58ec72f6-3c14-4cb1-a966-cd33fa988284"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'sentence1': 'The company has said it plans to restate its earnings for 2000 through 2002 .',\n",
              " 'sentence2': 'The company had announced in January that it would have to restate earnings for 2002 , 2001 and perhaps 2000 .',\n",
              " 'label': 1,\n",
              " 'idx': 47}"
            ]
          },
          "execution_count": 19,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "raw_train_dataset = raw_datasets[\"train\"]\n",
        "raw_train_dataset[42]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h-9Wrt6p7L8M",
        "outputId": "364ee9f3-4ea0-4190-ffbc-c958009b3769"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'sentence1': Value(dtype='string', id=None),\n",
              " 'sentence2': Value(dtype='string', id=None),\n",
              " 'label': ClassLabel(names=['not_equivalent', 'equivalent'], id=None),\n",
              " 'idx': Value(dtype='int32', id=None)}"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "raw_train_dataset.features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "klv5YI8D7L8M"
      },
      "source": [
        "#### 1.2 Tokenization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mS8fsUtY7L8M"
      },
      "source": [
        "Tokenization is the process of breaking down text into smaller units called tokens. These tokens can be words, subwords, or characters, depending on the tokenization strategy.\n",
        "\n",
        "For BERT, we use a subword tokenization method that can handle out-of-vocabulary words by breaking them into familiar subword units. In this section, we use the `AutoTokenizer` from the `Transformers` library to tokenize our input sentences. The tokenizer is initialized with the same checkpoint as our model (\"bert-base-uncased\") to ensure compatibility."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y1vKih3-7L8N"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "checkpoint = \"bert-base-uncased\"\n",
        "tokenizer = AutoTokenizer.from_pretrained(checkpoint)\n",
        "\n",
        "def tokenize(x):\n",
        "    # defer padding to batch creation (however, with TPUs it is more efficient to apply padding here!)\n",
        "    # in bert, max_sequence_length = 512\n",
        "    return tokenizer(x[\"sentence1\"], x[\"sentence2\"], truncation=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G-fBkO9i7L8N"
      },
      "source": [
        "The `tokenize` function processeses both sentences in each example, applying truncation to fit BERT's maximum sequence length of 512 tokens. The map function is then used to apply this tokenization to our entire dataset efficiently."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 292,
          "referenced_widgets": [
            "49cfed6de62b4f0da717121f554f5566",
            "6c72e011ed2b45f7a1f2aadbf106162b",
            "fecb6d266c9c4cc885bdc3d7cf4eaff5",
            "7d22a3a44e034c078b51f39fd706a329",
            "437cefb239a647b3a0346a9ba293c386",
            "235b0ac3c3d6497e8068b51d435b57f6",
            "5579e9638896437aa81c356158b83bd1",
            "bc44931728da430b9a0d72a150a175ee",
            "d5b8cafa73684d17af1fbb2bab678ecf",
            "1b45598114ab43499abfe71de4ae32b9",
            "cc085cf818f3475696c5e1e1289e0ed4"
          ]
        },
        "id": "R0NM-QtO7L8N",
        "outputId": "4ca90db2-c655-4500-d48f-2d16b340d2d5"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "49cfed6de62b4f0da717121f554f5566",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Map:   0%|          | 0/408 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "DatasetDict({\n",
              "    train: Dataset({\n",
              "        features: ['sentence1', 'sentence2', 'label', 'idx', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
              "        num_rows: 3668\n",
              "    })\n",
              "    validation: Dataset({\n",
              "        features: ['sentence1', 'sentence2', 'label', 'idx', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
              "        num_rows: 408\n",
              "    })\n",
              "    test: Dataset({\n",
              "        features: ['sentence1', 'sentence2', 'label', 'idx', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
              "        num_rows: 1725\n",
              "    })\n",
              "})"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "tokenized_datasets = raw_datasets.map(tokenize, batched=True)\n",
        "tokenized_datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gY3foMej7L8N"
      },
      "source": [
        "#### 1.3 Dynamic Padding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G4CrRLnI7L8N"
      },
      "source": [
        "Dynamic padding is an efficient technique used to handle variable-length sequences within a batch. Instead of padding all sequences to a fixed maximum length, which can waste computational resources, dynamic padding adds padding only up to the length of the longest sequence in each batch. This approach optimizes memory usage and computation time.\n",
        "\n",
        "In our BERT fine-tuning process, we'll use the `DataCollatorWithPadding` class from the Transformers library, which automatically pads the inputs in each batch to the maximum length in that specific batch."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_2G5ZXe17L8O"
      },
      "outputs": [],
      "source": [
        "from transformers import DataCollatorWithPadding\n",
        "\n",
        "data_collator = DataCollatorWithPadding(tokenizer=tokenizer)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nbbuvCJE7L8O",
        "outputId": "b35d06d8-6d7e-4095-ec7a-10a9ecee41b9"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[50, 59, 47, 67, 59, 50, 62, 32]"
            ]
          },
          "execution_count": 24,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "samples = tokenized_datasets[\"train\"][:8]\n",
        "samples = {k: v for k, v in samples.items() if k not in [\"idx\", \"sentence1\", \"sentence2\"]}\n",
        "[len(x) for x in samples[\"input_ids\"]]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Wkzkoam27L8O",
        "outputId": "597b1c48-c663-4525-d78c-4893dde06b0c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'input_ids': torch.Size([8, 67]),\n",
              " 'token_type_ids': torch.Size([8, 67]),\n",
              " 'attention_mask': torch.Size([8, 67]),\n",
              " 'labels': torch.Size([8])}"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "batch = data_collator(samples)\n",
        "{k: v.shape for k, v in batch.items()}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xprTnnn57L8O"
      },
      "source": [
        "### 2. Training and Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g2Qjfu5l7L8O"
      },
      "source": [
        "In this section, we set up the training and evaluation pipeline for our BERT model on the MRPC task. We define a `compute_metrics` function that uses the `evaluate` library to calculate performance metrics specific to the MRPC task."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "suXFWgQUFbNF"
      },
      "source": [
        "#### 2.1 Init 'Weights and Biases' project"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lZ46hSNuFho3"
      },
      "outputs": [],
      "source": [
        "run = wandb.init(project=\"bert-uncased-finetuned-mrpc\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "an-HL5sbAF6D"
      },
      "source": [
        "#### 2.1 Evaluate\n",
        "\n",
        "The `Trainer`class does not automatically evaluate model performance during training. We need to pass it a function to compute and report metrics. The `Evaluate` library provides dozens of evaluation methods for different domains such as for the GLUE MRPC task. For this subset it will report:\n",
        "\n",
        "- `accuracy`: the proportion of correct predictions among the total number of cases processed, with a range between 0 and 1, and:\n",
        "\n",
        "- `f1`: the harmonic mean of the precision and recall. Its range is 0-1 – its lowest possible value is 0, if either the precision or the recall is 0, and its highest possible value is 1.0, which means perfect precision and recall."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rygqRSJx7L8O"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import evaluate\n",
        "\n",
        "def compute_metrics(eval_preds):\n",
        "    metric = evaluate.load(\"glue\", \"mrpc\")\n",
        "    logits, labels = eval_preds\n",
        "    predictions = np.argmax(logits, axis=-1)\n",
        "    return metric.compute(predictions=predictions, references=labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2lBdboAhBZht"
      },
      "source": [
        "#### 2.2 Training Hyperparameters"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vDHweZrg7L8O"
      },
      "source": [
        "We then initialize the `TrainingArguments` with basic settings, including the output directory and evaluation strategy. In our case, the `compute_metrics` function will be called at the end of each epoch. This class contains many more hyperparameters that can be tuned as well as flags for activating different training options:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oDbMUkGs7L8P",
        "outputId": "97f4e82b-7dd4-48ef-968c-3c016f3d9c61"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/training_args.py:1494: FutureWarning: `evaluation_strategy` is deprecated and will be removed in version 4.46 of 🤗 Transformers. Use `eval_strategy` instead\n",
            "  warnings.warn(\n",
            "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        }
      ],
      "source": [
        "from transformers import Trainer, TrainingArguments, AutoModelForSequenceClassification\n",
        "\n",
        "training_args = TrainingArguments(\n",
        "    output_dir=\"results\",\n",
        "    evaluation_strategy=\"epoch\",\n",
        "    save_strategy=\"epoch\",\n",
        "    load_best_model_at_end=True,\n",
        "    learning_rate=2e-5,\n",
        "    per_device_train_batch_size=16,\n",
        "    per_device_eval_batch_size=16,\n",
        "    num_train_epochs=3,\n",
        "    weight_decay=0.1,\n",
        "    push_to_hub=False,\n",
        ")\n",
        "\n",
        "model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)\n",
        "\n",
        "trainer = Trainer(\n",
        "    model,\n",
        "    training_args,\n",
        "    train_dataset=tokenized_datasets[\"train\"],\n",
        "    eval_dataset=tokenized_datasets[\"validation\"],\n",
        "    data_collator=data_collator,\n",
        "    tokenizer=tokenizer,\n",
        "    compute_metrics=compute_metrics,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "6NPGFAyL7L8P",
        "outputId": "07c911b3-4582-42c1-f695-314fbc4fba86"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='690' max='690' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [690/690 01:43, Epoch 3/3]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "      <th>Accuracy</th>\n",
              "      <th>F1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>No log</td>\n",
              "      <td>0.432218</td>\n",
              "      <td>0.803922</td>\n",
              "      <td>0.851852</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>No log</td>\n",
              "      <td>0.369652</td>\n",
              "      <td>0.857843</td>\n",
              "      <td>0.898246</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>0.429800</td>\n",
              "      <td>0.413086</td>\n",
              "      <td>0.867647</td>\n",
              "      <td>0.906574</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "TrainOutput(global_step=690, training_loss=0.3674315521682518, metrics={'train_runtime': 104.2954, 'train_samples_per_second': 105.508, 'train_steps_per_second': 6.616, 'total_flos': 428577075854640.0, 'train_loss': 0.3674315521682518, 'epoch': 3.0})"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "trainer.train()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iWF0Rr1-7L8Q"
      },
      "source": [
        "### 3. Results"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![wandb results](./static/fine_tuning_bert_wandb.png)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7upQf34jFsnU"
      },
      "outputs": [],
      "source": [
        "wandb.finish()"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "L4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "1b45598114ab43499abfe71de4ae32b9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "235b0ac3c3d6497e8068b51d435b57f6": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "437cefb239a647b3a0346a9ba293c386": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "49cfed6de62b4f0da717121f554f5566": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6c72e011ed2b45f7a1f2aadbf106162b",
              "IPY_MODEL_fecb6d266c9c4cc885bdc3d7cf4eaff5",
              "IPY_MODEL_7d22a3a44e034c078b51f39fd706a329"
            ],
            "layout": "IPY_MODEL_437cefb239a647b3a0346a9ba293c386"
          }
        },
        "5579e9638896437aa81c356158b83bd1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6c72e011ed2b45f7a1f2aadbf106162b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_235b0ac3c3d6497e8068b51d435b57f6",
            "placeholder": "​",
            "style": "IPY_MODEL_5579e9638896437aa81c356158b83bd1",
            "value": "Map: 100%"
          }
        },
        "7d22a3a44e034c078b51f39fd706a329": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1b45598114ab43499abfe71de4ae32b9",
            "placeholder": "​",
            "style": "IPY_MODEL_cc085cf818f3475696c5e1e1289e0ed4",
            "value": " 408/408 [00:00&lt;00:00, 8761.67 examples/s]"
          }
        },
        "bc44931728da430b9a0d72a150a175ee": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cc085cf818f3475696c5e1e1289e0ed4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d5b8cafa73684d17af1fbb2bab678ecf": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "fecb6d266c9c4cc885bdc3d7cf4eaff5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bc44931728da430b9a0d72a150a175ee",
            "max": 408,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_d5b8cafa73684d17af1fbb2bab678ecf",
            "value": 408
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}