{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Stopping CRFE early with a callback on superconductor data\n\n:class:`~tagarnina.crfe.CRFE` calls ``callback`` after each fitted support. A\ntruthy return value stops the elimination and keeps that support. Here the\ncallback stops the run once the support is in a range of acceptable sizes, the\ntime budget has been used, and the score has started to drop.\n\nThe dataset is the same as in the advanced example: the critical temperature of\n21,263 superconductors, available on OpenML (ID 44964) under the CC BY 4.0\nlicence. Please cite it as: Hamidieh, K., A data-driven statistical model for\npredicting the critical temperature of a superconductor, Computational Materials\nScience, 154, 346-354 (2018).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Load the data\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.datasets import fetch_openml\n\nX, y = fetch_openml(data_id=44964, as_frame=True, return_X_y=True, parser=\"auto\")\n\nprint(X.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Constraints\nThe constraints are the same as in the advanced example: bounds on the range\nand mean statistics, a group bound on the weighted entropy statistics, and a\nlink between each weighted mean and its unweighted mean.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from tagarnina.optimization.constraints import (\n    AtLeastKConstraint,\n    AtMostKConstraint,\n    Constraints,\n    GroupConstraint,\n    OnlyEnforceIfConstraint,\n)\n\n\ndef features_with_prefix(prefix):\n    return [c for c in X.columns if c.startswith(prefix)]\n\n\nrange_features = features_with_prefix(\"range_\")\nmean_features = features_with_prefix(\"mean_\")\nwtd_entropy_features = features_with_prefix(\"wtd_entropy_\")\nproperties = [c.removeprefix(\"mean_\") for c in mean_features]\n\nconstraints = Constraints(\n    constraints=[\n        AtMostKConstraint(features=range_features, max_features=3),\n        AtLeastKConstraint(features=mean_features, min_features=2),\n        GroupConstraint(\n            features=wtd_entropy_features,\n            min_features=1,\n            max_features=4,\n        ),\n        *[\n            OnlyEnforceIfConstraint(feature1=f\"wtd_mean_{p}\", feature2=f\"mean_{p}\")\n            for p in properties\n        ],\n    ],\n)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Stop rule\nThe callback returns ``True`` only when all three conditions hold:\n\n* the support has between ``MIN_FEATURES`` and ``MAX_FEATURES`` features,\n* at least ``TIME_CAP`` seconds have elapsed, and\n* the score is more than ``TOLERANCE`` below the best score seen so far.\n\nThe window keeps the selection no smaller than the target and no larger than\nthe upper bound. The time cap is a budget: the run is not stopped before it\nhas been used. The tolerance stops the run when removing features starts to\ncost score. The best score is tracked over all supports, including those\noutside the window.\n\nThe score is computed on the training data, so it is optimistic.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "MIN_FEATURES = 20\nMAX_FEATURES = 40\nTIME_CAP = 60  # seconds\nTOLERANCE = 0.002\n\n\ndef make_stop_rule():\n    best_score = -float(\"inf\")\n\n    def callback(row):\n        nonlocal best_score\n        best_score = max(best_score, row[\"score\"])\n        in_window = MIN_FEATURES <= row[\"n_features\"] <= MAX_FEATURES\n        dropped = best_score - row[\"score\"] > TOLERANCE\n        return in_window and row[\"elapsed\"] >= TIME_CAP and dropped\n\n    return callback"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Feature elimination\nA smaller ``step`` removes fewer features at each iteration, so there are more\nsupports to check against the stop rule. ``verbose=1`` prints one line per\nfitted support.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestRegressor\n\nfrom tagarnina.crfe import CRFE\n\nestimator = RandomForestRegressor(\n    n_estimators=100,\n    min_samples_leaf=5,\n    n_jobs=-1,\n    random_state=0,\n)\n\ncrfe = CRFE(\n    estimator=estimator,\n    constraints=constraints,\n    step=0.05,\n    n_features_to_select=MIN_FEATURES,\n    scoring=\"r2\",\n    callback=make_stop_rule(),\n    verbose=1,\n)\ncrfe.fit(X, y)\n\nn_selected = len(crfe.get_feature_names_out())\nprint(crfe.summary_[\"termination\"])\nprint(f\"Selected {n_selected} features after {crfe.summary_['wall_time']:.0f} s\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The plot shows the score at each support size. The shaded band is the window\nwhere the callback may stop, and the dashed line marks the selected support.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n\nfig, ax = plt.subplots(figsize=(7, 4))\nax.axvspan(\n    MIN_FEATURES,\n    MAX_FEATURES,\n    color=\"tab:green\",\n    alpha=0.1,\n    label=\"stop window\",\n)\nax.plot(crfe.history_[\"n_features\"], crfe.history_[\"score\"], marker=\"o\")\nax.axvline(n_selected, color=\"tab:red\", linestyle=\"--\", label=\"selected\")\nax.set_xlabel(\"Number of features\")\nax.set_ylabel(\"R2 (training data)\")\nax.legend()\nfig.tight_layout()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.13.17"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}