notebooks/vertex_nas_detection_tfvision.ipynb

{
  "cells": [
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# NAS (Neural Architecture Search) for Object-Detection on Vertex AI with TF-vision\n",
     "\n",
     "Make sure that you have read the [required documentations](https://cloud.google.com/vertex-ai/docs/training/neural-architecture-search/overview#reading_order)\n",
     "before executing this notebook.\n",
     "NOTE: This notebook is meant to run pre-built trainer code with pre-built search-spaces. If you want to run your own trainer\n",
     "code or create your own NAS search-space from scratch, then do not use this notebook.\n",
     "\n",
     "This notebook shows example of [SpineNet](https://arxiv.org/pdf/1912.05027.pdf) paper result on CoCo data.\n",
     "According to Table 3 in this paper, SpineNet-49 achieves 40.8 AP score with 85.4B FLOPs, \n",
     "which is higher performance than R50-FPN which achieves 37.8 AP score with 96.8B FLOPs.\n",
     "However, this notebook uses GPUs instead of TPUs for training and significantly less number of 1000 total trials for search.\n",
     "With this notebook, the expected Stage2 AP score on Spinenet is 39.1 AP with 94.43 FLOPS which is still higher performance than R50-FPN.\n",
     "The detailed settings for this notebook are:\n",
     "- Stage-1 search\n",
     "    - Number of trials: 1000\n",
     "    - Number of GPUs per trial: 2\n",
     "    - GPU type: TESLA_V100\n",
     "    - Avg single trial training time: 2.5 hours\n",
     "    - Number of parallel trials: 10\n",
     "    - GPU quota used: 10*2 = 20 V100 GPUs\n",
     "    - Time to run: 10.4 days\n",
     "    If you have higher GPU quota, then the runtime will decrease proportionately.\n",
     "    - GPU hours: 5000 V100 GPU hours\n",
     "- Stage-2 full training with top 5 models\n",
     "    - Number of trials: 5\n",
     "    - Number of GPUs per trial: 8\n",
     "    - GPU type: TESLA_V100\n",
     "    - Avg single trial training time: 3 days 17 hrs\n",
     "    - Number of parallel trials: 5\n",
     "    - GPU quota used: 5*8 = 40 V100 GPUs\n",
     "    You can also run this with just 24 V100 GPUs by running the job twice\n",
     "    with just 3 models at a time instead of all 5 in parallel.\n",
     "    - Time to run: 3 days 17 hrs\n",
     "    - GPU hours: 3560 V100 GPU hours\n",
     "\n",
     "Stage1 search cost: ~$22,000\n",
     "Stage2 full-training cost: ~$14,000\n",
     "Total cost: ~$36,000\n",
     "\n",
     "\n",
     "Here are the **pre-requisites** before you can start using this notebook: \n",
     "1. Your GCP project should have been (a) allow-listed and (b) [a GPU quota should have been allocated](https://cloud.google.com/vertex-ai/docs/training/neural-architecture-search/environment-setup#device-quota) for the NAS jobs.\n",
     "2. You have selected a python3 kernel to run this notebook."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Install required libraries"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "%%sh\n",
     "\n",
     "pip install tensorflow==2.7.0 --user\n",
     "pip install tf-models-official==2.7.1\n",
     "pip install pyglove==0.1.0"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "**NOTE: Please restart the notebook after installing above libraries successfully.**"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Download source code\n",
     "This needs to be done just once.\n"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "%%sh\n",
     "\n",
     "# NOTE: It is ok for this step to fail if the directory exists.\n",
     "mkdir ~/nas_experiment"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "%%sh\n",
     "rm -r -f ~/nas_experiment/nas_codes\n",
     "git clone https://github.com/google/vertex-ai-nas.git ~/nas_experiment/nas_codes"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Set code path"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "import os\n",
     "os.chdir('/home/jupyter/nas_experiment/nas_codes')"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Set up environment variables\n",
     "Here we set up the environment variables.\n",
     "NOTE: These have to be set-up every time you run a new session because the later code-blocks use them.\n"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "# Set a unique USER name. This is used for creating a unique job-name identifier.\n",
     "%env USER=<fill>\n",
     "# Set a region to launch jobs into.\n",
     "%env REGION=<fill>\n",
     "# Set any unique docker-id for this run. When the next section builds a docker, then this id will be used to tag it.\n",
     "%env TRAINER_DOCKER_ID=<fill>\n",
     "%env LATENCY_CALCULATOR_DOCKER_ID=<fill>\n",
     "# The GCP project-id must be the one that has been clear-listed for the NAS jobs. \n",
     "%env PROJECT_ID=<fill>\n",
     "# Set an output working directory for the NAS jobs. The GCP project should have write access to \n",
     "# this GCS output directory. A simple way to ensure this is to use a bucket inside the same GCP project.\n",
     "# NOTE: The region of the bucket must be the same as job's.\n",
     "%env GCS_ROOT_DIR=<fill>\n",
     "# Set the accelerator device type.\n",
     "%env DEVICE=<fill>\n",
     "\n",
     "\n",
     "# Set the GCS paths to the training and validation datasets. The GCP project should have read access to the data-location.\n",
     "# Please read the \"Data-Preparation\" section \n",
     "# (https://cloud.google.com/vertex-ai/docs/training/neural-architecture-search/pre-built-trainer#data-preparation)\n",
     "# in the documentation to ensure that the data is in an appropriate format\n",
     "# suitable for the NAS pipeline. You can run the \"Validate and Visualize data format\" section in this notebook \n",
     "# to verify that the data can be loaded properly.\n",
     "# Update the path to CoCo data below.\n",
     "%env STAGE1_TRAINING_DATA_PATH=<gs://path-to>/coco/train-00[0-1]??-of-00256.tfrecord,gs://cloud-nas-public-eu/detection/coco/train-002[0-3]?-of-00256.tfrecord\n",
     "%env STAGE1_VALIDATION_DATA_PATH=<gs://path-to>/coco/train-002[4-5]?-of-00256.tfrecord\n",
     "%env STAGE2_TRAINING_DATA_PATH=<gs://path-to>/coco/train*\n",
     "%env STAGE2_VALIDATION_DATA_PATH=<gs://path-to>/coco/val*"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "**NOTE:** The following set up steps need to be done just once."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "%%sh\n",
     "\n",
     "# Authenticate docker for your artifact registry.\n",
     "gcloud auth configure-docker ${REGION}-docker.pkg.dev"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "%%sh\n",
     "\n",
     "# NOTE: This needs to be just once for the first time. It is ok for this to FAIL if the GCS bucket already exists.\n",
     "\n",
     "# Create the output directory. \n",
     "gsutil mkdir $GCS_ROOT_DIR"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Validate and Visualize data format"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "The following code verifies that the data can be loaded properly before you run the experiments."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "import tensorflow as tf\n",
     "import matplotlib.pyplot as plt\n",
     "from official.vision.beta.dataloaders import tf_example_decoder\n",
     "import cloud_nas_utils\n",
     "\n",
     "dataset = tf.data.Dataset.list_files(os.environ.get('STAGE1_TRAINING_DATA_PATH'), shuffle=False).apply(tf.data.TFRecordDataset)\n",
     "dataset = dataset.map(tf_example_decoder.TfExampleDecoder().decode).batch(1)\n",
     "\n",
     "num_examples = 4\n",
     "for (i, example) in enumerate(dataset.take(num_examples)):\n",
     "    for k, v in example.items():\n",
     "        example[k] = v.numpy()[0]\n",
     "    \n",
     "    image_with_boxes = cloud_nas_utils.draw_boxes(example['image'].copy(), example['groundtruth_boxes'], example['groundtruth_classes'], [1.0] * len(example['groundtruth_classes']), max_boxes=10, min_score=0.01)\n",
     "\n",
     "    _, ax = plt.subplots(1, 1, figsize=(100, 64))\n",
     "    ax.imshow(image_with_boxes)\n",
     "    ax.grid(False)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Build container\n",
     "The container must be built the first time and then every time the source-code is modified. Otherwise, there is no need to run this step. This step internally builds the *Dockerfile* in the source-code directory and then pushes the docker to the cloud. "
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "%%sh\n",
     "\n",
     "# NOTE: This step can take several minutes when run for the first time.\n",
     "\n",
     "python3 vertex_nas_cli.py build \\\n",
     "--project_id=${PROJECT_ID} \\\n",
     "--trainer_docker_id=${TRAINER_DOCKER_ID} \\\n",
     "--region=${REGION} \\\n",
     "--trainer_docker_file=\"tf_vision/nas_multi_trial.Dockerfile\" \\\n",
     "--latency_calculator_docker_id=${LATENCY_CALCULATOR_DOCKER_ID} \\\n",
     "--latency_calculator_docker_file=\"tf_vision/latency_computation_using_saved_model.Dockerfile\""
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
      "# Launch NAS stage 1 job\n",
      "If you want to customize this notebook for your own dataset other than CoCo, then you must\n",
      "read the [required documentations](https://cloud.google.com/vertex-ai/docs/training/neural-architecture-search/overview#reading_order)\n",
      "to ensure that you set up the proxy-task and other settings properly."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "%%sh\n",
     "\n",
     "DATE=\"$(date '+%Y%m%d_%H%M%S')\"\n",
     "JOB_ID=\"${USER}_nas_tfvision_iod_${DATE}\"\n",
     "\n",
     "CMD=\"\n",
     "python3 vertex_nas_cli.py search \\\n",
     "--project_id=${PROJECT_ID} \\\n",
     "--region=${REGION} \\\n",
     "--trainer_docker_id=${TRAINER_DOCKER_ID} \\\n",
     "--job_name=${JOB_ID} \\\n",
     "--max_nas_trial=1000 \\\n",
     "--max_parallel_nas_trial=10 \\\n",
     "--max_failed_nas_trial=200 \\\n",
     "--use_prebuilt_trainer=True \\\n",
     "--prebuilt_search_space=\"spinenet\" \\\n",
     "--accelerator_type=${DEVICE} \\\n",
     "--num_gpus=2 \\\n",
     "--root_output_dir=${GCS_ROOT_DIR} \\\n",
     "--search_docker_flags \\\n",
     "params_override=\"tf_vision/configs/experiments/spinenet_search_gpu.yaml\" \\\n",
     "training_data_path=${STAGE1_TRAINING_DATA_PATH} \\\n",
     "validation_data_path=${STAGE1_VALIDATION_DATA_PATH} \\\n",
     "model=\"retinanet\"\n",
     "\"\n",
     "\n",
     "echo Executing command: ${CMD}\n",
     "    \n",
     "${CMD}"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Inspect NAS search progress\n",
     "A periodic evaluation while the search is going on can help decide if the search job has converged. This code-block shows how to generate summary of top N trials so far."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "# Set the stage1 search-job id. It's a numeric value returned by the Vertex service.\n",
     "%env JOB_ID=<fill>"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "%%sh\n",
     "\n",
     "mkdir -p /home/jupyter/nas_experiment/jobs\n",
     "python3 vertex_nas_cli.py list_trials \\\n",
     "--project_id=${PROJECT_ID} \\\n",
     "--job_id=${JOB_ID} \\\n",
     "--region=${REGION} \\\n",
     "--trials_output_file=/home/jupyter/nas_experiment/jobs/${JOB_ID}.yaml\n",
     "\n",
     "cat /home/jupyter/nas_experiment/jobs/${JOB_ID}.yaml"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Launch NAS stage 2 job"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "%%sh\n",
     "\n",
     "DATE=\"$(date '+%Y%m%d_%H%M%S')\"\n",
     "\n",
     "# Please modify the \"JOB_ID\", \"TRIAL_ID\", and the finetuning config file before running.\n",
     "# JOB_ID is numeric value you can find from the job UI in Pantheon.\n",
     "JOB_ID=<fill>\n",
     "# TRIAL_ID is one of the best performing trials which has to be finetuned.\n",
     "TRIAL_IDS=<fill> # The top trials chosen for training to converge.\n",
     "\n",
     "CMD=\"\n",
     "\n",
     "python3 vertex_nas_cli.py train \\\n",
     "--project_id=${PROJECT_ID} \\\n",
     "--region=${REGION} \\\n",
     "--trainer_docker_id=${TRAINER_DOCKER_ID} \\\n",
     "--use_prebuilt_trainer=True \\\n",
     "--prebuilt_search_space=\"spinenet\" \\\n",
     "--train_accelerator_type=${DEVICE} \\\n",
     "--train_num_gpus=8 \\\n",
     "--root_output_dir=${GCS_ROOT_DIR} \\\n",
     "--search_job_id=${JOB_ID} \\\n",
     "--search_job_region=${REGION} \\\n",
     "--train_nas_trial_numbers=${TRIAL_IDS} \\\n",
     "--train_job_suffix=\"stage2_${DATE}\" \\\n",
     "--train_docker_flags \\\n",
     "params_override=\"tf_vision/configs/experiments/spinenet_search_finetune_gpu.yaml\" \\\n",
     "training_data_path=${STAGE2_TRAINING_DATA_PATH} \\\n",
     "validation_data_path=${STAGE2_VALIDATION_DATA_PATH} \\\n",
     "model=\"retinanet\"\n",
     "\"\n",
     "\n",
     "echo Executing command: ${CMD}\n",
     "    \n",
     "${CMD}"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "# Post Training Export and Analysis"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "### Create SavedModel"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "# Set the output path for the saved-model.\n",
     "%env SAVED_MODEL_DIR=/home/jupyter/nas_experiment/saved_models/\n",
     "# The directory of the finished job.\n",
     "%env JOB_DIR=<fill>\n",
     "# The trial you want to export.\n",
     "%env TRIAL=<fill>\n",
     "# The config file you used to launch the job, like tf_vision/configs/experiments/spinenet_search_gpu.yaml.\n",
     "%env CONFIG=<fill>"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "import dataclasses\n",
     "import tensorflow as tf\n",
     "\n",
     "# Import from nas_codes.\n",
     "import pyglove as pg\n",
     "import cloud_nas_utils\n",
     "import search_spaces\n",
     "from tf_vision import registry_imports\n",
     "from tf_vision import config_utils\n",
     "\n",
     "# Import from tf-vision library.\n",
     "from official.vision.beta.serving import export_saved_model_lib\n",
     "\n",
     "@dataclasses.dataclass\n",
     "class Args():\n",
     "    model = 'retinanet'\n",
     "    config_file = os.environ.get('CONFIG')\n",
     "    params_override = None\n",
     "    job_dir = os.environ.get('JOB_DIR')\n",
     "    training_data_path = os.environ.get('STAGE1_TRAINING_DATA_PATH')\n",
     "    validation_data_path = os.environ.get('STAGE1_VALIDATION_DATA_PATH')\n",
     "    use_tpu = None\n",
     "\n",
     "args = Args()\n",
     "search_space = 'spinenet'\n",
     "job_dir = os.environ.get('JOB_DIR')\n",
     "trial_dir = os.path.join(job_dir, os.environ.get('TRIAL'))\n",
     "nas_params_str = os.path.join(trial_dir, 'nas_params_str.json')\n",
     "\n",
     "tunable_functor_or_object = cloud_nas_utils.parse_and_save_nas_params_str(\n",
     "    search_spaces.get_search_space(search_space), nas_params_str, job_dir)\n",
     "tunable_object = tunable_functor_or_object()\n",
     "serialized_tunable_object = pg.to_json_str(tunable_object, json_indent=2, hide_default_values=False)\n",
     "params = config_utils.create_params(\n",
     "    args,\n",
     "    search_space,\n",
     "    serialized_tunable_object,\n",
     "    None)\n",
     "latest_checkpoint = tf.train.latest_checkpoint(trial_dir)\n",
     "\n",
     "export_saved_model_lib.export_inference_graph(\n",
     "    input_type='image_bytes',\n",
     "    batch_size=1,\n",
     "    input_image_size=params.task.model.input_size[0:2],\n",
     "    params=params,\n",
     "    checkpoint_path=latest_checkpoint,\n",
     "    export_dir=os.environ.get('SAVED_MODEL_DIR'))"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "%%sh\n",
     "\n",
     "saved_model_cli show --dir=${SAVED_MODEL_DIR} --all"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "### Run Prediction and Visualize results"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "import matplotlib.pyplot as plt\n",
     "import numpy as np\n",
     "import tensorflow as tf\n",
     "import cloud_nas_utils\n",
     "\n",
     "model = tf.saved_model.load(os.environ.get('SAVED_MODEL_DIR'))\n",
     "detect_fn = model.signatures['serving_default']"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "keys_to_features = {\n",
     "    'image/encoded': tf.io.FixedLenFeature((), tf.string),\n",
     "    'image/height': tf.io.FixedLenFeature((), tf.int64),\n",
     "    'image/width': tf.io.FixedLenFeature((), tf.int64),\n",
     "    'image/object/bbox/xmin': tf.io.VarLenFeature(tf.float32),\n",
     "    'image/object/bbox/xmax': tf.io.VarLenFeature(tf.float32),\n",
     "    'image/object/bbox/ymin': tf.io.VarLenFeature(tf.float32),\n",
     "    'image/object/bbox/ymax': tf.io.VarLenFeature(tf.float32),\n",
     "    'image/object/class/label': tf.io.VarLenFeature(tf.int64),\n",
     "    'image/object/area': tf.io.VarLenFeature(tf.float32),\n",
     "    'image/object/is_crowd': tf.io.VarLenFeature(tf.int64),\n",
     "}\n",
     "def _parse_function(serialized_example):\n",
     "    parsed_tensors = tf.io.parse_single_example(serialized_example, keys_to_features)\n",
     "    for k in parsed_tensors:\n",
     "        if isinstance(parsed_tensors[k], tf.SparseTensor):\n",
     "            if parsed_tensors[k].dtype == tf.string:\n",
     "                parsed_tensors[k] = tf.sparse.to_dense(parsed_tensors[k], default_value='')\n",
     "            else:\n",
     "                parsed_tensors[k] = tf.sparse.to_dense(parsed_tensors[k], default_value=0)\n",
     "\n",
     "    image = tf.io.decode_image(parsed_tensors['image/encoded'], channels=3)\n",
     "    image.set_shape([None, None, 3])\n",
     "    xmin = parsed_tensors['image/object/bbox/xmin']\n",
     "    xmax = parsed_tensors['image/object/bbox/xmax']\n",
     "    ymin = parsed_tensors['image/object/bbox/ymin']\n",
     "    ymax = parsed_tensors['image/object/bbox/ymax']\n",
     "    boxes =  tf.stack([ymin, xmin, ymax, xmax], axis=-1)\n",
     "    decoded_tensors = {\n",
     "      'height': parsed_tensors['image/height'],\n",
     "      'width': parsed_tensors['image/width'],\n",
     "      'groundtruth_classes': parsed_tensors['image/object/class/label'],\n",
     "      'groundtruth_boxes': boxes,\n",
     "      'image': image,\n",
     "      'image_bytes': parsed_tensors['image/encoded'],\n",
     "    }\n",
     "    return decoded_tensors\n",
     "\n",
     "dataset = tf.data.Dataset.list_files(os.environ.get('VALIDATION_DATA_PATH'), shuffle=False).apply(tf.data.TFRecordDataset)\n",
     "dataset = dataset.map(_parse_function).batch(1)\n",
     "\n",
     "num_examples = 1\n",
     "for (i, example) in enumerate(dataset.take(num_examples)):\n",
     "    outputs = detect_fn(example['image_bytes'])\n",
     "    \n",
     "    for k, v in example.items():\n",
     "        example[k] = v.numpy()[0]\n",
     "    for k, v in outputs.items():\n",
     "        outputs[k] = v.numpy()[0]\n",
     "    \n",
     "    print('Got {} predicted bboxes.'.format(outputs['num_detections']))\n",
     "    image_with_groundtruth_bbox = cloud_nas_utils.draw_boxes(example['image'].copy(), example['groundtruth_boxes'], example['groundtruth_classes'], [1.0] * len(example['groundtruth_classes']), max_boxes=10, min_score=0.01)\n",
     "    image_with_prediction_bbox = cloud_nas_utils.draw_boxes(example['image'].copy(), outputs['detection_boxes'], outputs['detection_classes'], outputs['detection_scores'], max_boxes=10, min_score=0.01)\n",
     "\n",
     "    _, ax = plt.subplots(1, 2, figsize=(100, 64))\n",
     "    ax[0].imshow(image_with_groundtruth_bbox)\n",
     "    ax[0].grid(False)\n",
     "    ax[0].set_title('Groundtruth')\n",
     "    ax[1].imshow(image_with_prediction_bbox)\n",
     "    ax[1].grid(False)\n",
     "    ax[1].set_title('Prediction')"
    ]
   }
  ],
  "metadata": {
   "environment": {
    "kernel": "python3",
    "name": "common-cu110.m100",
    "type": "gcloud",
    "uri": "gcr.io/deeplearning-platform-release/base-cu110:m100"
   },
   "kernelspec": {
    "display_name": "Python 3",
    "language": "python",
    "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
     "version": 3
    },
    "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
    "version": "3.7.12"
   }
  },
  "nbformat": 4,
  "nbformat_minor": 4
 }