Note
Go to the end to download the full example code.
Advanced constraints in CRFE on superconductor data#
Recursive feature elimination with CRFE for a continuous
target: the critical temperature of 21,263 superconductors. The constraints
combine group bounds with implications between features. The verbose output
shows the progress of the elimination.
The dataset is public and is available on OpenML (ID 44964), under the CC BY 4.0 licence. Please cite it as: Hamidieh, K., A data-driven statistical model for predicting the critical temperature of a superconductor, Computational Materials Science, 154, 346-354 (2018).
Load the data#
The target is the critical temperature in kelvin. Each feature is a statistic
(for example mean_, range_ or wtd_entropy_) of an elemental
property, named by the statistic followed by the property, such as
range_Density. The weighted statistics (wtd_ prefix) weight each
element by its fraction in the compound.
from sklearn.datasets import fetch_openml
X, y = fetch_openml(data_id=44964, as_frame=True, return_X_y=True, parser="auto")
print(X.shape)
(21263, 81)
Constraints#
Features are grouped by their statistic, which is the prefix of the name. A
AtMostKConstraint limits the range
statistics, an AtLeastKConstraint
requires some mean statistics, and a
GroupConstraint sets both bounds on
the weighted entropy statistics.
An OnlyEnforceIfConstraint links
each weighted mean to the unweighted mean of the same property: the weighted
mean can be selected only if the unweighted mean is too. This is a modelling
choice that keeps the two versions of a property together.
from tagarnina.optimization.constraints import (
AtLeastKConstraint,
AtMostKConstraint,
Constraints,
GroupConstraint,
OnlyEnforceIfConstraint,
)
def features_with_prefix(prefix):
return [c for c in X.columns if c.startswith(prefix)]
range_features = features_with_prefix("range_")
mean_features = features_with_prefix("mean_")
wtd_entropy_features = features_with_prefix("wtd_entropy_")
properties = [c.removeprefix("mean_") for c in mean_features]
constraints = Constraints(
constraints=[
AtMostKConstraint(features=range_features, max_features=3),
AtLeastKConstraint(features=mean_features, min_features=2),
GroupConstraint(
features=wtd_entropy_features,
min_features=1,
max_features=4,
),
*[
OnlyEnforceIfConstraint(feature1=f"wtd_mean_{p}", feature2=f"mean_{p}")
for p in properties
],
],
)
Feature elimination#
CRFE keeps 15 features. verbose=1 prints the
settings, one line per fitted support, and a summary. verbose=2 also prints
the details of the solver and the model.
from sklearn.ensemble import RandomForestRegressor
from tagarnina.crfe import CRFE
estimator = RandomForestRegressor(
n_estimators=100,
min_samples_leaf=5,
n_jobs=-1,
random_state=0,
)
crfe = CRFE(
estimator=estimator,
constraints=constraints,
step=0.1,
n_features_to_select=15,
scoring="r2",
verbose=1,
)
crfe.fit(X, y)
selected_features = crfe.get_feature_names_out()
print(selected_features)
----------------------------------------------------------------------------------------------------------
tagarnina v0.1.0
Constrained Recursive Feature Elimination
(c) Guillermo Navas-Palencia, 2026, Apache License 2.0
----------------------------------------------------------------------------------------------------------
Estimator : RandomForestRegressor
Data : 21263 samples, 81 features
Settings : step=0.1, n_features_to_select=15, strategy=probing, scoring=r2
Constraints : 11 (AtMostK 1, AtLeastK 1, Group 1, OnlyEnforceIf 8)
----------------------------------------------------------------------------------------------------------
iter n_features removed probes fit(s) solve(s) score retained gap elapsed(s)
----------------------------------------------------------------------------------------------------------
0 81 15 0|0 13.649 0.008 9.6305e-01 0.9993 0.0007 13.973
1 66 10 0|0 11.187 0.008 9.6249e-01 0.9998 0.0001 25.441
2 56 9 0|0 9.857 0.011 9.6232e-01 0.9999 0.0000 35.573
3 47 8 0|0 8.574 0.011 9.6214e-01 0.9998 0.0000 44.403
4 39 8 8|1 7.113 0.056 9.6168e-01 0.9996 0.0000 51.790
5 31 8 0|0 5.571 0.009 9.6074e-01 0.9987 0.0001 57.660
6 23 8 0|0 4.085 0.004 9.5926e-01 0.9967 0.0004 61.961
7 15 - - 2.768 - 9.5635e-01 - - 64.860
----------------------------------------------------------------------------------------------------------
Termination : reached the target number of features
Selected : 15 of 81 features
Probes : 8 feasible, 1 infeasible, 1 blocked
Time : wall 64.860s (fit 62.805, solve 0.107, score 1.175), cpu 633.156s
==========================================================================================================
['wtd_range_atomic_mass' 'std_atomic_mass' 'range_atomic_radius'
'mean_Density' 'std_Density' 'gmean_ElectronAffinity'
'wtd_std_ElectronAffinity' 'mean_ThermalConductivity'
'wtd_gmean_ThermalConductivity' 'wtd_entropy_ThermalConductivity'
'range_ThermalConductivity' 'wtd_std_ThermalConductivity'
'wtd_gmean_Valence' 'wtd_range_Valence' 'wtd_std_Valence']
Check the constraints on the selected features. The counts below are compared with the bounds set above.
selected = set(selected_features)
n_range = sum(c in selected for c in range_features)
n_mean = sum(c in selected for c in mean_features)
n_wtd_entropy = sum(c in selected for c in wtd_entropy_features)
print(f"range statistics: {n_range} selected (at most 3)")
print(f"mean statistics: {n_mean} selected (at least 2)")
print(f"weighted entropy statistics: {n_wtd_entropy} selected (between 1 and 4)")
violations = [
p for p in properties if f"wtd_mean_{p}" in selected and f"mean_{p}" not in selected
]
print(f"Weighted means without their unweighted mean: {violations}")
range statistics: 2 selected (at most 3)
mean statistics: 2 selected (at least 2)
weighted entropy statistics: 1 selected (between 1 and 4)
Weighted means without their unweighted mean: []
The run summary records how the elimination ended and how many probes were infeasible.
print(crfe.summary_["termination"])
print(f"Features that could not be removed: {crfe.summary_['n_blocked']}")
reached the target number of features
Features that could not be removed: 1
The plots show the score and the importance gap at each support size. The score is computed on the training data, so it is optimistic and tends to increase with the number of features. The importance gap is the importance lost because of the constraints, compared with standard RFE with the same number of features. See the basic example for the full history.
import matplotlib.pyplot as plt
n_features = crfe.history_["n_features"]
fig, (ax_score, ax_gap) = plt.subplots(1, 2, figsize=(10, 4))
ax_score.plot(n_features, crfe.history_["score"], marker="o")
ax_score.set_xlabel("Number of features")
ax_score.set_ylabel("R2 (training data)")
ax_score.set_title("Score")
ax_gap.plot(n_features, crfe.history_["importance_gap"], marker="o")
ax_gap.set_xlabel("Number of features")
ax_gap.set_ylabel("Importance gap")
ax_gap.set_title("Importance lost to constraints")
fig.tight_layout()

Total running time of the script: (1 minutes 6.008 seconds)