docs/sources/user_guide/evaluate/bias_variance_decomp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# bias_variance_decomp: Bias-variance decomposition for classification and regression losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bias variance decomposition of machine learning algorithms for various loss functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> `from mlxtend.evaluate import bias_variance_decomp`    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often, researchers use the terms *bias* and *variance* or \"bias-variance tradeoff\" to describe the performance of a model -- i.e., you may stumble upon talks, books, or articles where people say that a model has a high variance or high bias. So, what does that mean? In general, we might say that \"high variance\" is proportional to overfitting, and \"high bias\" is proportional to underfitting.\n",
    "\n",
    "Anyways, why are we attempting to do this bias-variance decomposition in the first place? The decomposition of the loss into bias and variance helps us understand learning algorithms, as these concepts are correlated to underfitting and overfitting.\n",
    "\n",
    "\n",
    "To use the more formal terms for bias and variance, assume we have a point estimator $\\hat{\\theta}$ of some parameter or function $\\theta$. Then, the bias is commonly defined as the difference between the expected value of the estimator and the parameter that we want to estimate:\n",
    "\n",
    "\n",
    "$$\n",
    "\\text{Bias} = E[\\hat{\\theta}] - \\theta.\n",
    "$$\n",
    "\n",
    "If the bias is larger than zero, we also say that the estimator is positively biased, if the bias is smaller than zero, the estimator is negatively biased, and if the bias is exactly zero, the estimator is unbiased. Similarly, we define the variance as the difference between the expected value of the squared estimator minus the squared expectation of the estimator:\n",
    "\n",
    "\n",
    "$$\n",
    "\\text{Var}(\\hat{\\theta}) = E\\big[\\hat{\\theta}^2\\big] - \\bigg(E\\big[\\hat{\\theta}\\big]\\bigg)^2.\n",
    "$$\n",
    "\n",
    "Note that in the context of this lecture, it will be more convenient to write the variance in its alternative form:\n",
    "\n",
    "$$\n",
    "\\text{Var}(\\hat{\\theta}) = E[(E[{\\hat{\\theta}}] - \\hat{\\theta})^2].\n",
    "$$\n",
    "\n",
    "To illustrate the concept further in context of machine learning ...\n",
    "\n",
    "Suppose there is an unknown target function or \"true function\" to which we do want to approximate. Now, suppose we have different training sets drawn from an unknown distribution defined as \"true function + noise.\" The following plot shows different linear regression models, each fit to a different training set. None of these hypotheses approximate the true function well, except at two points (around x=-10 and x=6). Here, we can say that the bias is large because the difference between the true value and the predicted value, on average (here, average means \"expectation of the training sets\" not \"expectation over examples in the training set\"), is large:\n",
    "\n",
    "\n",
    "![](bias_variance_decomp_files/high-bias-plot.png)\n",
    "\n",
    "The next plot shows different unpruned decision tree models, each fit to a different training set. Note that these hypotheses fit the training data very closely. However, if we would consider the expectation over training sets, the average hypothesis would fit the true function perfectly (given that the noise is unbiased and has an expected value of 0). As we can see, the variance is very large, since on average, a prediction differs a lot from the expectation value of the prediction:\n",
    "\n",
    "\n",
    "![](bias_variance_decomp_files/varianceplot.png)\n",
    "\n",
    "\n",
    "### Bias-Variance Decomposition of the Squared Loss\n",
    "\n",
    "\n",
    "We can decompose a loss function such as the squared loss into three terms, a variance, bias, and a noise term (and the same is true for the decomposition of the 0-1 loss later). However, for simplicity, we will ignore the noise term.\n",
    "\n",
    "Before we introduce the bias-variance decomposition of the 0-1 loss for classification, let us start with the decomposition of the squared loss as an easy warm-up exercise to get familiar with the overall concept.\n",
    "\n",
    "The previous section already listed the common formal definitions of bias and variance, however, let us define them again for convenience:\n",
    "\n",
    "\n",
    "\n",
    "$$\n",
    "\\text{Bias}(\\hat{\\theta}) = E[\\hat{\\theta}] - \\theta, \\quad \\text{Var}(\\hat{\\theta}) = E[(E[{\\hat{\\theta}}] - \\hat{\\theta})^2].\n",
    "$$\n",
    "\n",
    "Recall that in the context of these machine learning lecture (notes), we defined \n",
    "\n",
    "- the true or target function as $y = f(x)$,\n",
    "- the predicted target value as $\\hat{y} = \\hat{f}(x) = h(x)$,\n",
    "- and the squared loss as $S = (y - \\hat{y})^2$. (I use $S$ here because it will be easier to tell it apart from the $E$, which we use for the *expectation* in this lecture.)\n",
    "\n",
    "**Note that unless noted otherwise, the expectation is over training sets!**\n",
    "\n",
    "To get started with the squared error loss decomposition into bias and variance, let use do some algebraic manipulation, i.e., adding and subtracting the expected value of $\\hat{y}$ and then expanding the expression using the quadratic formula $(a+b)^2 = a^2 + b^2 + 2ab)$:\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\begin{split}\n",
    "S = (y - \\hat{y})^2 \\\\\n",
    "(y - \\hat{y})^2 &= (y - E[{\\hat{y}}] + E[{\\hat{y}}] - \\hat{y})^2 \\\\\n",
    "&= (y-E[{\\hat{y}}])^2 + (E[{\\hat{y}}] - y)^2 + 2(y - E[\\hat{y}])(E[\\hat{y}] - \\hat{y}). \n",
    "\\end{split}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "Next, we just use the expectation on both sides, and we are already done:\n",
    "\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "E[S] &= E[(y - \\hat{y})^2] \\\\\n",
    "E[(y - \\hat{y})^2]\n",
    "&= (y-E[{\\hat{y}}])^2 + E[(E[{\\hat{y}}] - \\hat{y})^2]\\\\\n",
    "&= \\text{[Bias]}^2 + \\text{Variance}.\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "You may wonder what happened to the \"$2ab$\" term ($2(y - E[\\hat{y}])(E[\\hat{y}] - \\hat{y})$) when we used the expectation. It turns that it evaluates to zero and hence vanishes from the equation, which can be shown as follows:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "E[2(y - E[{\\hat{y}}])(E[{\\hat{y}}] - \\hat{y})]  &= 2 E[(y - E[{\\hat{y}}])(E[{\\hat{y}}] - \\hat{y})] \\\\\n",
    "&=  2(y - E[{\\hat{y}}])E[(E[{\\hat{y}}] - \\hat{y})] \\\\\n",
    "&= 2(y - E[{\\hat{y}}])(E[E[{\\hat{y}}]] - E[\\hat{y}])\\\\\n",
    "&= 2(y - E[{\\hat{y}}])(E[{\\hat{y}}] - E[{\\hat{y}}]) \\\\\n",
    "&= 0.\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "\n",
    "So, this is the canonical decomposition of the squared error loss into bias and variance. The next section will discuss some approaches that have been made to decompose the 0-1 loss that we commonly use for classification accuracy or error.\n",
    "\n",
    "\n",
    "\n",
    "The following figure is a sketch of variance and bias in relation to the training error and generalization error -- how high variance related to overfitting, and how large bias relates to underfitting:\n",
    "\n",
    "![](bias_variance_decomp_files/image-20181029010428686.png)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Bias-Variance Decomposition of the 0-1 Loss\n",
    "\n",
    "\n",
    "\n",
    "Note that decomposing the 0-1 loss into bias and variance components is not as straight-forward as for the squared error loss. To quote Pedro Domingos, a well-known machine learning researcher and professor at University of Washington: \n",
    "\n",
    "> \"several authors have proposed bias-variance decompositions related to zero-one loss (Kong & Dietterich, 1995; Breiman, 1996b; Kohavi & Wolpert, 1996; Tibshirani, 1996; Friedman, 1997). However, each of these decompositions has significant shortcomings.\". [1] \n",
    "\n",
    "\n",
    "In fact, the paper this quote was taken from may offer the most intuitive and general formulation at this point. However, we will first, for simplicity, go over Kong & Dietterich formulation [2] of the 0-1 loss decomposition, which is the same as Domingos's but excluding the noise term (for simplicity). \n",
    "\n",
    "The table below summarizes the relevant terms we used for the squared loss in relation to the 0-1 loss. Recall that the 0-1 loss, $L$, is 0 if a class label is predicted correctly, and one otherwise. The main prediction for the squared error loss is simply the average over the predictions $E[\\hat{y}]$ (the expectation is over training sets), for the 0-1 loss Kong & Dietterich and Domingos defined it as the mode. I.e., if a model predicts the label one more than 50% of the time (considering all possible training sets), then the main prediction is 1, and 0 otherwise.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "|               -              | Squared Loss                    | 0-1 Loss                    |\n",
    "|------------------------------|---------------------------------|-----------------------------|\n",
    "| Single loss                  | $(y - \\hat{y})^2$               | $L(y, \\hat{y})$             |\n",
    "| Expected loss                | $E[(y - \\hat{y})^2]$            | $E[L(y, \\hat{y})]$          |\n",
    "| Main prediction $E[\\hat{y}]$ | mean (average)                  | mode                        |\n",
    "| Bias$^2$                     | $(y-E[{\\hat{y}}])^2$            | $L(y, E[\\hat{y}])$          |\n",
    "| Variance                     | $E[(E[{\\hat{y}}] - \\hat{y})^2]$ | $E[L(\\hat{y}, E[\\hat{y}])]$ |\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Hence, as result from using the mode to define the main prediction of the 0-1 loss, the bias is 1 if the main prediction does not agree with the true label $y$, and 0 otherwise:\n",
    "\n",
    "\n",
    "$$\n",
    "Bias = \\begin{cases}\n",
    "1 \\text{ if }  y \\neq E[{\\hat{y}}], \\\\\n",
    "0 \\text{ otherwise}.\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "The variance of the 0-1 loss is defined as the probability that the predicted label does not match the main prediction: \n",
    "\n",
    "$$\n",
    "Variance = P(\\hat{y} \\neq E[\\hat{{y}}]).\n",
    "$$\n",
    "\n",
    "Next, let us take a look at what happens to the loss if the bias is 0. Given the general definition of the loss, loss = bias + variance, if the bias is 0, then we define the loss as the variance: \n",
    "\n",
    "\n",
    "$$\n",
    "Loss = 0 + Variance = Loss = P(\\hat{y} \\neq y) = Variance = P(\\hat{y} \\neq E[\\hat{{y}}]).\n",
    "$$\n",
    "\n",
    "In other words, if a model has zero bias, it's loss is entirely defined by the variance, which is intuitive if we think of variance in the context of being proportional overfitting.\n",
    "\n",
    "The more surprising scenario is if the bias is equal to 1. If the bias is equal to 1, as explained by Pedro Domingos, the increasing the variance can decrease the loss, which is an interesting observation. This can be seen by first rewriting the 0-1 loss function as \n",
    "\n",
    "$$\n",
    "Loss = P(\\hat{y} \\neq y) = 1 - P(\\hat{y} = y).\n",
    "$$\n",
    "\n",
    "(Note that we have not done anything new, yet.) Now, if we look at the previous equation of the bias, if the bias is 1, we have  $ y \\neq E[{\\hat{y}}]$. If $y$ is not equal to the main prediction, but $y$ is also is equal to $\\hat{y}$, then $\\hat{y}$ must be equal to the main prediction. Using the \"inverse\" (\"1 minus\"), we can then write the loss as\n",
    "\n",
    "\n",
    "$$\n",
    "Loss = P(\\hat{y} \\neq y) = 1 - P(\\hat{y} = y) = 1 - P(\\hat{y} \\neq E[{\\hat{y}}]).\n",
    "$$\n",
    "\n",
    "Since the bias is 1, the loss is hence defined as \"loss = bias - variance\" if the bias is 1 (or \"loss = 1 - variance\"). This might be quite unintuitive at first, but the explanations Kong, Dietterich, and Domingos offer was that if a model has a very high bias such that it main prediction is always wrong, increasing the variance can be beneficial, since increasing the variance would push the decision boundary, which might lead to some correct predictions just by chance then. In other words, for scenarios with high bias, increasing the variance can improve (decrease) the loss!\n",
    "\n",
    "### References\n",
    "\n",
    "- [1] Domingos, Pedro. \"A unified bias-variance decomposition.\" Proceedings of 17th International Conference on Machine Learning. 2000.\n",
    "- [2] Dietterich, Thomas G., and Eun Bae Kong. Machine learning bias, statistical bias, and statistical variance of decision tree algorithms. Technical report, Department of Computer Science, Oregon State University, 1995."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1 -- Bias Variance Decomposition of a Decision Tree Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average expected loss: 0.062\n",
      "Average bias: 0.022\n",
      "Average variance: 0.040\n"
     ]
    }
   ],
   "source": [
    "from mlxtend.evaluate import bias_variance_decomp\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from mlxtend.data import iris_data\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "X, y = iris_data()\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
    "                                                    test_size=0.3,\n",
    "                                                    random_state=123,\n",
    "                                                    shuffle=True,\n",
    "                                                    stratify=y)\n",
    "\n",
    "\n",
    "\n",
    "tree = DecisionTreeClassifier(random_state=123)\n",
    "\n",
    "avg_expected_loss, avg_bias, avg_var = bias_variance_decomp(\n",
    "        tree, X_train, y_train, X_test, y_test, \n",
    "        loss='0-1_loss',\n",
    "        random_seed=123)\n",
    "\n",
    "print('Average expected loss: %.3f' % avg_expected_loss)\n",
    "print('Average bias: %.3f' % avg_bias)\n",
    "print('Average variance: %.3f' % avg_var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For comparison, the bias-variance decomposition of a bagging classifier, which should intuitively have a lower variance compared than a single decision tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average expected loss: 0.048\n",
      "Average bias: 0.022\n",
      "Average variance: 0.026\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import BaggingClassifier\n",
    "\n",
    "tree = DecisionTreeClassifier(random_state=123)\n",
    "bag = BaggingClassifier(base_estimator=tree,\n",
    "                        n_estimators=100,\n",
    "                        random_state=123)\n",
    "\n",
    "avg_expected_loss, avg_bias, avg_var = bias_variance_decomp(\n",
    "        bag, X_train, y_train, X_test, y_test, \n",
    "        loss='0-1_loss',\n",
    "        random_seed=123)\n",
    "\n",
    "print('Average expected loss: %.3f' % avg_expected_loss)\n",
    "print('Average bias: %.3f' % avg_bias)\n",
    "print('Average variance: %.3f' % avg_var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 2 -- Bias Variance Decomposition of a Decision Tree Regressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average expected loss: 31.536\n",
      "Average bias: 14.096\n",
      "Average variance: 17.440\n"
     ]
    }
   ],
   "source": [
    "from mlxtend.evaluate import bias_variance_decomp\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from mlxtend.data import boston_housing_data\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "X, y = boston_housing_data()\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
    "                                                    test_size=0.3,\n",
    "                                                    random_state=123,\n",
    "                                                    shuffle=True)\n",
    "\n",
    "\n",
    "\n",
    "tree = DecisionTreeRegressor(random_state=123)\n",
    "\n",
    "avg_expected_loss, avg_bias, avg_var = bias_variance_decomp(\n",
    "        tree, X_train, y_train, X_test, y_test, \n",
    "        loss='mse',\n",
    "        random_seed=123)\n",
    "\n",
    "print('Average expected loss: %.3f' % avg_expected_loss)\n",
    "print('Average bias: %.3f' % avg_bias)\n",
    "print('Average variance: %.3f' % avg_var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For comparison, the bias-variance decomposition of a bagging regressor is shown below, which should intuitively have a lower variance than a single decision tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average expected loss: 18.620\n",
      "Average bias: 15.461\n",
      "Average variance: 3.159\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import BaggingRegressor\n",
    "\n",
    "tree = DecisionTreeRegressor(random_state=123)\n",
    "bag = BaggingRegressor(base_estimator=tree,\n",
    "                       n_estimators=100,\n",
    "                       random_state=123)\n",
    "\n",
    "avg_expected_loss, avg_bias, avg_var = bias_variance_decomp(\n",
    "        bag, X_train, y_train, X_test, y_test, \n",
    "        loss='mse',\n",
    "        random_seed=123)\n",
    "\n",
    "print('Average expected loss: %.3f' % avg_expected_loss)\n",
    "print('Average bias: %.3f' % avg_bias)\n",
    "print('Average variance: %.3f' % avg_var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 3 -- TensorFlow/Keras Support"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since mlxtend v0.18.0, the `bias_variance_decomp` now supports Keras models. Note that the original model is reset in each round (before refitting it to the bootstrap samples)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32.69300595184836"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from mlxtend.evaluate import bias_variance_decomp\n",
    "from mlxtend.data import boston_housing_data\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_squared_error\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "np.random.seed(1)\n",
    "tf.random.set_seed(1)\n",
    "\n",
    "\n",
    "X, y = boston_housing_data()\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y,\n",
    "                                                    test_size=0.3,\n",
    "                                                    random_state=123,\n",
    "                                                    shuffle=True)\n",
    "\n",
    "\n",
    "model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Dense(32, activation=tf.nn.relu),\n",
    "    tf.keras.layers.Dense(1)\n",
    "  ])\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam()\n",
    "model.compile(loss='mean_squared_error', optimizer=optimizer)\n",
    "\n",
    "model.fit(X_train, y_train, epochs=100, verbose=0)\n",
    "\n",
    "mean_squared_error(model.predict(X_test), y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that it is highly recommended to use the same number of training epochs that you would use on the original training set to ensure convergence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average expected loss: 32.740\n",
      "Average bias: 27.474\n",
      "Average variance: 5.265\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "tf.random.set_seed(1)\n",
    "\n",
    "\n",
    "avg_expected_loss, avg_bias, avg_var = bias_variance_decomp(\n",
    "        model, X_train, y_train, X_test, y_test, \n",
    "        loss='mse',\n",
    "        num_rounds=100,\n",
    "        random_seed=123,\n",
    "        epochs=200, # fit_param\n",
    "        verbose=0) # fit_param\n",
    "\n",
    "\n",
    "print('Average expected loss: %.3f' % avg_expected_loss)\n",
    "print('Average bias: %.3f' % avg_bias)\n",
    "print('Average variance: %.3f' % avg_var)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## bias_variance_decomp\n",
      "\n",
      "*bias_variance_decomp(estimator, X_train, y_train, X_test, y_test, loss='0-1_loss', num_rounds=200, random_seed=None, **fit_params)*\n",
      "\n",
      "estimator : object\n",
      "    A classifier or regressor object or class implementing both a\n",
      "    `fit` and `predict` method similar to the scikit-learn API.\n",
      "\n",
      "\n",
      "- `X_train` : array-like, shape=(num_examples, num_features)\n",
      "\n",
      "    A training dataset for drawing the bootstrap samples to carry\n",
      "    out the bias-variance decomposition.\n",
      "\n",
      "\n",
      "- `y_train` : array-like, shape=(num_examples)\n",
      "\n",
      "    Targets (class labels, continuous values in case of regression)\n",
      "    associated with the `X_train` examples.\n",
      "\n",
      "\n",
      "- `X_test` : array-like, shape=(num_examples, num_features)\n",
      "\n",
      "    The test dataset for computing the average loss, bias,\n",
      "    and variance.\n",
      "\n",
      "\n",
      "- `y_test` : array-like, shape=(num_examples)\n",
      "\n",
      "    Targets (class labels, continuous values in case of regression)\n",
      "    associated with the `X_test` examples.\n",
      "\n",
      "\n",
      "- `loss` : str (default='0-1_loss')\n",
      "\n",
      "    Loss function for performing the bias-variance decomposition.\n",
      "    Currently allowed values are '0-1_loss' and 'mse'.\n",
      "\n",
      "\n",
      "- `num_rounds` : int (default=200)\n",
      "\n",
      "    Number of bootstrap rounds (sampling from the training set)\n",
      "    for performing the bias-variance decomposition. Each bootstrap\n",
      "    sample has the same size as the original training set.\n",
      "\n",
      "\n",
      "- `random_seed` : int (default=None)\n",
      "\n",
      "    Random seed for the bootstrap sampling used for the\n",
      "    bias-variance decomposition.\n",
      "\n",
      "\n",
      "- `fit_params` : additional parameters\n",
      "\n",
      "    Additional parameters to be passed to the .fit() function of the\n",
      "    estimator when it is fit to the bootstrap samples.\n",
      "\n",
      "**Returns**\n",
      "\n",
      "- `avg_expected_loss, avg_bias, avg_var` : returns the average expected\n",
      "\n",
      "    average bias, and average bias (all floats), where the average\n",
      "    is computed over the data points in the test set.\n",
      "\n",
      "**Examples**\n",
      "\n",
      "For usage examples, please see\n",
      "    https://rasbt.github.io/mlxtend/user_guide/evaluate/bias_variance_decomp/\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open('../../api_modules/mlxtend.evaluate/bias_variance_decomp.md', 'r') as f:\n",
    "    s = f.read() \n",
    "print(s)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}