Note
Go to the end to download the full example code.
Stopping CRFE early with a callback on superconductor data#
CRFE calls callback after each fitted support. A
truthy return value stops the elimination and keeps that support. Here the
callback stops the run once the support is in a range of acceptable sizes, the
time budget has been used, and the score has started to drop.
The dataset is the same as in the advanced example: the critical temperature of 21,263 superconductors, available on OpenML (ID 44964) under the CC BY 4.0 licence. Please cite it as: Hamidieh, K., A data-driven statistical model for predicting the critical temperature of a superconductor, Computational Materials Science, 154, 346-354 (2018).
Load the data#
from sklearn.datasets import fetch_openml
X, y = fetch_openml(data_id=44964, as_frame=True, return_X_y=True, parser="auto")
print(X.shape)
(21263, 81)
Constraints#
The constraints are the same as in the advanced example: bounds on the range and mean statistics, a group bound on the weighted entropy statistics, and a link between each weighted mean and its unweighted mean.
from tagarnina.optimization.constraints import (
AtLeastKConstraint,
AtMostKConstraint,
Constraints,
GroupConstraint,
OnlyEnforceIfConstraint,
)
def features_with_prefix(prefix):
return [c for c in X.columns if c.startswith(prefix)]
range_features = features_with_prefix("range_")
mean_features = features_with_prefix("mean_")
wtd_entropy_features = features_with_prefix("wtd_entropy_")
properties = [c.removeprefix("mean_") for c in mean_features]
constraints = Constraints(
constraints=[
AtMostKConstraint(features=range_features, max_features=3),
AtLeastKConstraint(features=mean_features, min_features=2),
GroupConstraint(
features=wtd_entropy_features,
min_features=1,
max_features=4,
),
*[
OnlyEnforceIfConstraint(feature1=f"wtd_mean_{p}", feature2=f"mean_{p}")
for p in properties
],
],
)
Stop rule#
The callback returns True only when all three conditions hold:
the support has between
MIN_FEATURESandMAX_FEATURESfeatures,at least
TIME_CAPseconds have elapsed, andthe score is more than
TOLERANCEbelow the best score seen so far.
The window keeps the selection no smaller than the target and no larger than the upper bound. The time cap is a budget: the run is not stopped before it has been used. The tolerance stops the run when removing features starts to cost score. The best score is tracked over all supports, including those outside the window.
The score is computed on the training data, so it is optimistic.
MIN_FEATURES = 20
MAX_FEATURES = 40
TIME_CAP = 60 # seconds
TOLERANCE = 0.002
def make_stop_rule():
best_score = -float("inf")
def callback(row):
nonlocal best_score
best_score = max(best_score, row["score"])
in_window = MIN_FEATURES <= row["n_features"] <= MAX_FEATURES
dropped = best_score - row["score"] > TOLERANCE
return in_window and row["elapsed"] >= TIME_CAP and dropped
return callback
Feature elimination#
A smaller step removes fewer features at each iteration, so there are more
supports to check against the stop rule. verbose=1 prints one line per
fitted support.
from sklearn.ensemble import RandomForestRegressor
from tagarnina.crfe import CRFE
estimator = RandomForestRegressor(
n_estimators=100,
min_samples_leaf=5,
n_jobs=-1,
random_state=0,
)
crfe = CRFE(
estimator=estimator,
constraints=constraints,
step=0.05,
n_features_to_select=MIN_FEATURES,
scoring="r2",
callback=make_stop_rule(),
verbose=1,
)
crfe.fit(X, y)
n_selected = len(crfe.get_feature_names_out())
print(crfe.summary_["termination"])
print(f"Selected {n_selected} features after {crfe.summary_['wall_time']:.0f} s")
----------------------------------------------------------------------------------------------------------
tagarnina v0.1.0
Constrained Recursive Feature Elimination
(c) Guillermo Navas-Palencia, 2026, Apache License 2.0
----------------------------------------------------------------------------------------------------------
Estimator : RandomForestRegressor
Data : 21263 samples, 81 features
Settings : step=0.05, n_features_to_select=20, strategy=probing, scoring=r2
Constraints : 11 (AtMostK 1, AtLeastK 1, Group 1, OnlyEnforceIf 8)
----------------------------------------------------------------------------------------------------------
iter n_features removed probes fit(s) solve(s) score retained gap elapsed(s)
----------------------------------------------------------------------------------------------------------
0 81 12 0|0 12.603 0.013 9.6305e-01 0.9993 0.0007 13.186
1 69 4 0|0 11.610 0.006 9.6264e-01 1.0000 0.0000 25.012
2 65 5 0|0 10.899 0.005 9.6249e-01 1.0000 0.0000 36.069
3 60 4 0|0 10.231 0.010 9.6236e-01 1.0000 0.0000 46.555
4 56 6 0|0 9.541 0.006 9.6229e-01 0.9998 0.0002 56.304
5 50 5 0|0 8.797 0.009 9.6215e-01 0.9999 0.0000 65.340
6 45 5 0|0 7.734 0.004 9.6193e-01 0.9999 0.0000 73.231
7 40 4 4|1 6.596 0.047 9.6169e-01 0.9999 0.0000 80.097
8 36 4 0|0 6.939 0.005 9.6144e-01 0.9998 0.0000 87.271
9 32 4 0|0 6.014 0.017 9.6100e-01 0.9995 0.0001 96.876
10 32 - - 85.449 - 9.6100e-01 - - 182.469
----------------------------------------------------------------------------------------------------------
Termination : stopped by the callback
Selected : 32 of 81 features
Probes : 4 feasible, 1 infeasible, 1 blocked
Time : wall 182.470s (fit 176.412, solve 0.121, score 4.629), cpu 1001.750s
==========================================================================================================
stopped by the callback
Selected 32 features after 182 s
The plot shows the score at each support size. The shaded band is the window where the callback may stop, and the dashed line marks the selected support.
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(7, 4))
ax.axvspan(
MIN_FEATURES,
MAX_FEATURES,
color="tab:green",
alpha=0.1,
label="stop window",
)
ax.plot(crfe.history_["n_features"], crfe.history_["score"], marker="o")
ax.axvline(n_selected, color="tab:red", linestyle="--", label="selected")
ax.set_xlabel("Number of features")
ax.set_ylabel("R2 (training data)")
ax.legend()
fig.tight_layout()

Total running time of the script: (3 minutes 3.648 seconds)