target_encoding_Coursera.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Version 1.1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mean encodings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this programming assignment you will be working with `1C` dataset from the final competition. You are asked to encode `item_id` in 4 different ways:\n",
    "\n",
    "    1) Via KFold scheme;  \n",
    "    2) Via Leave-one-out scheme;\n",
    "    3) Via smoothing scheme;\n",
    "    4) Via expanding mean scheme.\n",
    "\n",
    "**You will need to submit** the correlation coefficient between resulting encoding and target variable up to 4 decimal places.\n",
    "\n",
    "### General tips\n",
    "\n",
    "* Fill NANs in the encoding with `0.3343`.\n",
    "* Some encoding schemes depend on sorting order, so in order to avoid confusion, please use the following code snippet to construct the data frame. This snippet also implements mean encoding without regularization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from  sklearn import model_selection\n",
    "from itertools import product\n",
    "from grader import Grader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Read data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales = pd.read_csv('../readonly/final_project_data/sales_train.csv.gz')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Aggregate data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the competition task is to make a monthly prediction, we need to aggregate the data to montly level before doing any encodings. The following code-cell serves just that purpose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/pandas/core/groupby.py:4036: FutureWarning: using a dict with renaming is deprecated and will be removed in a future version\n",
      "  return super(DataFrameGroupBy, self).aggregate(arg, *args, **kwargs)\n"
     ]
    }
   ],
   "source": [
    "index_cols = ['shop_id', 'item_id', 'date_block_num']\n",
    "\n",
    "# For every month we create a grid from all shops/items combinations from that month\n",
    "grid = [] \n",
    "for block_num in sales['date_block_num'].unique():\n",
    "    cur_shops = sales[sales['date_block_num']==block_num]['shop_id'].unique()\n",
    "    cur_items = sales[sales['date_block_num']==block_num]['item_id'].unique()\n",
    "    grid.append(np.array(list(product(*[cur_shops, cur_items, [block_num]])),dtype='int32'))\n",
    "\n",
    "#turn the grid into pandas dataframe\n",
    "grid = pd.DataFrame(np.vstack(grid), columns = index_cols,dtype=np.int32)\n",
    "\n",
    "#get aggregated values for (shop_id, item_id, month)\n",
    "gb = sales.groupby(index_cols,as_index=False).agg({'item_cnt_day':{'target':'sum'}})\n",
    "\n",
    "#fix column names\n",
    "gb.columns = [col[0] if col[-1]=='' else col[-1] for col in gb.columns.values]\n",
    "#join aggregated data to the grid\n",
    "all_data = pd.merge(grid,gb,how='left',on=index_cols).fillna(0)\n",
    "#sort the data\n",
    "all_data.sort_values(['date_block_num','shop_id','item_id'],inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mean encodings without regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we did the techinical work, we are ready to actually *mean encode* the desired `item_id` variable. \n",
    "\n",
    "Here are two ways to implement mean encoding features *without* any regularization. You can use this code as a starting point to implement regularized techniques. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.483038698862\n"
     ]
    }
   ],
   "source": [
    "# Calculate a mapping: {item_id: target_mean}\n",
    "item_id_target_mean = all_data.groupby('item_id').target.mean()\n",
    "\n",
    "# In our non-regularized case we just *map* the computed means to the `item_id`'s\n",
    "all_data['item_target_enc'] = all_data['item_id'].map(item_id_target_mean)\n",
    "\n",
    "# Fill NaNs\n",
    "all_data['item_target_enc'].fillna(0.3343, inplace=True) \n",
    "\n",
    "# Print correlation\n",
    "encoded_feature = all_data['item_target_enc'].values\n",
    "print(np.corrcoef(all_data['target'].values, encoded_feature)[0][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Method 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.483038698862\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "     Differently to `.target.mean()` function `transform` \n",
    "   will return a dataframe with an index like in `all_data`.\n",
    "   Basically this single line of code is equivalent to the first two lines from of Method 1.\n",
    "'''\n",
    "all_data['item_target_enc'] = all_data.groupby('item_id')['target'].transform('mean')\n",
    "\n",
    "# Fill NaNs\n",
    "all_data['item_target_enc'].fillna(0.3343, inplace=True) \n",
    "\n",
    "# Print correlation\n",
    "encoded_feature = all_data['item_target_enc'].values\n",
    "print(np.corrcoef(all_data['target'].values, encoded_feature)[0][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the printed value? It is the correlation coefficient between the target variable and your new encoded feature. You need to **compute correlation coefficient** between the encodings, that you will implement and **submit those to coursera**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "grader = Grader()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. KFold scheme"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explained starting at 41 sec of [Regularization video](https://www.coursera.org/learn/competitive-data-science/lecture/LGYQ2/regularization)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now it's your turn to write the code!** \n",
    "\n",
    "You may use 'Regularization' video as a reference for all further tasks.\n",
    "\n",
    "First, implement KFold scheme with five folds. Use KFold(5) from sklearn.model_selection. \n",
    "\n",
    "1. Split your data in 5 folds with `sklearn.model_selection.KFold` with `shuffle=False` argument.\n",
    "2. Iterate through folds: use all but the current fold to calculate mean target for each level `item_id`, and  fill the current fold.\n",
    "\n",
    "    *  See the **Method 1** from the example implementation. In particular learn what `map` and pd.Series.map functions do. They are pretty handy in many situations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel/__main__.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.41645907128\n",
      "Current answer for task KFold_scheme is: 0.41645907128\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE GOES HERE\n",
    "kf = model_selection.KFold(n_splits = 5, shuffle = False) \n",
    "all_data['item_target_enc'] = np.nan\n",
    "\n",
    "for tr_ind, val_ind in kf.split(all_data):\n",
    "    X_tr, X_val = all_data.iloc[tr_ind], all_data.iloc[val_ind]\n",
    "    X_val['item_target_enc'] =   X_val['item_id'].map(X_tr.groupby('item_id').target.mean())\n",
    "    all_data.iloc[val_ind,:] = X_val\n",
    "    \n",
    "\n",
    "all_data['item_target_enc'].fillna(0.3343, inplace = True)\n",
    "encoded_feature = all_data['item_target_enc'].values\n",
    "\n",
    "# You will need to compute correlation like that\n",
    "corr = np.corrcoef(all_data['target'].values, encoded_feature)[0][1]\n",
    "print(corr)\n",
    "grader.submit_tag('KFold_scheme', corr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Leave-one-out scheme"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Now, implement leave-one-out scheme. Note that if you just simply set the number of folds to the number of samples and run the code from the **KFold scheme**, you will probably wait for a very long time. \n",
    "\n",
    "To implement a faster version, note, that to calculate mean target value using all the objects but one *given object*, you can:\n",
    "\n",
    "1. Calculate sum of the target values using all the objects.\n",
    "2. Then subtract the target of the *given object* and divide the resulting value by `n_objects - 1`. \n",
    "\n",
    "Note that you do not need to perform `1.` for every object. And `2.` can be implemented without any `for` loop.\n",
    "\n",
    "It is the most convenient to use `.transform` function as in **Method 2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.480384831129\n",
      "Current answer for task Leave-one-out_scheme is: 0.480384831129\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE GOES HERE\n",
    "leave_one_out_sum = all_data['item_id'].map(all_data.groupby('item_id').target.sum())\n",
    "leave_one_out_count = all_data['item_id'].map(all_data.groupby('item_id').target.count())\n",
    "all_data['item_target_enc'] = (leave_one_out_sum - all_data['target']) / (leave_one_out_count - 1)\n",
    "\n",
    "all_data['item_target_enc'].fillna(0.3343, inplace = True)\n",
    "encoded_feature = all_data['item_target_enc'].values\n",
    "\n",
    "corr = np.corrcoef(all_data['target'].values, encoded_feature)[0][1]\n",
    "print(corr)\n",
    "grader.submit_tag('Leave-one-out_scheme', corr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Smoothing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explained starting at 4:03 of [Regularization video](https://www.coursera.org/learn/competitive-data-science/lecture/LGYQ2/regularization)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, implement smoothing scheme with $\\alpha = 100$. Use the formula from the first slide in the video and $0.3343$ as `globalmean`. Note that `nrows` is the number of objects that belong to a certain category (not the number of rows in the dataset)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.48181987971\n",
      "Current answer for task Smoothing_scheme is: 0.48181987971\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE GOES HERE\n",
    "alpha = 100\n",
    "globalmean = 0.3343\n",
    "train_new = all_data.copy()\n",
    "nrows = train_new.groupby('item_id').size()\n",
    "\n",
    "# all_data['item_id_target_mean'] = all_data.groupby('item_id')['target'].transform('mean')\n",
    "\n",
    "means = train_new.groupby('item_id').target.agg('mean')\n",
    "\n",
    "score = (np.multiply(means,nrows) + globalmean*alpha) / (nrows+alpha)\n",
    "\n",
    "train_new['smooth'] = train_new['item_id']\n",
    "train_new['smooth'] = train_new['smooth'].map(score)\n",
    "\n",
    "encoded_feature = train_new['smooth'].values\n",
    "\n",
    "corr = np.corrcoef(all_data['target'].values, encoded_feature)[0][1]\n",
    "print(corr)\n",
    "grader.submit_tag('Smoothing_scheme', corr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Expanding mean scheme"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explained starting at 5:50 of [Regularization video](https://www.coursera.org/learn/competitive-data-science/lecture/LGYQ2/regularization)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Finally, implement the *expanding mean* scheme. It is basically already implemented for you in the video, but you can challenge yourself and try to implement it yourself. You will need [`cumsum`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.core.groupby.DataFrameGroupBy.cumsum.html) and [`cumcount`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.core.groupby.GroupBy.cumcount.html) functions from pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.502524521108\n",
      "Current answer for task Expanding_mean_scheme is: 0.502524521108\n"
     ]
    }
   ],
   "source": [
    "# YOUR CODE GOES HERE\n",
    "cumsum = all_data.groupby('item_id')['target'].cumsum() - all_data['target']\n",
    "cumcnt = all_data.groupby('item_id').cumcount()\n",
    "\n",
    "all_data['item_target_enc_expand'] = cumsum / cumcnt\n",
    "\n",
    "all_data['item_target_enc_expand'].fillna(0.3343, inplace = True)\n",
    "encoded_feature = all_data['item_target_enc_expand'].values\n",
    "\n",
    "corr = np.corrcoef(all_data['target'].values, encoded_feature)[0][1]\n",
    "print(corr)\n",
    "grader.submit_tag('Expanding_mean_scheme', corr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authorization & Submission\n",
    "To submit assignment parts to Cousera platform, please, enter your e-mail and token into variables below. You can generate token on this programming assignment page. Note: Token expires 30 minutes after generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You want to submit these numbers:\n",
      "Task KFold_scheme: 0.41645907128\n",
      "Task Leave-one-out_scheme: 0.480384831129\n",
      "Task Smoothing_scheme: 0.48181987971\n",
      "Task Expanding_mean_scheme: 0.502524521108\n"
     ]
    }
   ],
   "source": [
    "STUDENT_EMAIL = 'attajinkya@gmail.com'# EMAIL HERE\n",
    "STUDENT_TOKEN = 'uAmbymf3V53JYBuG'# TOKEN HERE\n",
    "grader.status()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitted to Coursera platform. See results on assignment page!\n"
     ]
    }
   ],
   "source": [
    "grader.submit(STUDENT_EMAIL, STUDENT_TOKEN)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}