Note
Go to the end to download the full example code.
CRFECV on German Credit Data#
Choosing the number of features with CRFECV on the German
credit dataset (1,000 applicants, 20 features). CRFE runs
inside each cross-validation fold, and every size it reaches is scored on the
held-out rows. The size with the best mean score is selected, and the final run
uses all the data.
The dataset is public and is available on OpenML (ID 31). Please cite it as: Dr. Hans Hofmann, Statlog (German Credit Data), UCI Machine Learning Repository, 1994.
Load the data#
The target is good or bad credit risk. It is encoded as 1 for good.
from sklearn.datasets import fetch_openml
X, y = fetch_openml(data_id=31, as_frame=True, return_X_y=True, parser="auto")
y = (y == "good").astype(int)
categorical_features = X.select_dtypes("category").columns.tolist()
numerical_features = X.select_dtypes("number").columns.tolist()
One-hot encoding#
Each categorical feature is one-hot encoded. The encoded columns keep the original feature name as a prefix, so they can be grouped by name.
import pandas as pd
from sklearn.compose import ColumnTransformer, make_column_selector
from sklearn.preprocessing import OneHotEncoder
preprocessor = ColumnTransformer(
transformers=[
("cat", OneHotEncoder(), make_column_selector(dtype_include="category"))
],
remainder="passthrough",
verbose_feature_names_out=False,
)
X_transformed = pd.DataFrame(
data=preprocessor.fit_transform(X),
columns=preprocessor.get_feature_names_out(),
)
Constraints#
An AllOrNoneConstraint on the
encoded columns of each categorical feature keeps the whole feature or removes
it. A MaxCorrelationConstraint on
limits the pairwise Pearson correlation of the numerical features to 0.5.
from tagarnina.optimization.constraints import (
AllOrNoneConstraint,
Constraints,
MaxCorrelationConstraint,
)
all_or_none_constraints = [
AllOrNoneConstraint(
features=[c for c in X_transformed.columns if c.startswith(f"{feature}_")]
)
for feature in categorical_features
]
max_corr_constraint = MaxCorrelationConstraint(
features=numerical_features, max_corr=0.5, method="pearson"
)
constraints = Constraints(constraints=[*all_or_none_constraints, max_corr_constraint])
The selector#
CRFECV uses 5 stratified folds and ROC AUC as the score.
Each fold runs CRFE and scores every size on its test rows.
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold
from tagarnina.crfe import CRFECV
selector = CRFECV(
estimator=RandomForestClassifier(
max_depth=10, min_samples_leaf=0.2, random_state=0
),
constraints=constraints,
step=0.1,
cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=0),
scoring="roc_auc",
)
Fitting#
The duration and credit amount features have a correlation of about 0.62, so they cannot both be selected. The full set of features therefore violates the constraints. Its score exists, because the first round of CRFE is unconstrained, but it is not a valid choice. CRFECV excludes it from the comparison and warns.
import warnings
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
selector.fit(X_transformed, y)
for warning in caught:
print(f"{warning.category.__name__}: {warning.message}")
UserWarning: The constraints are infeasible for the full set of 61 features, which is excluded from the comparison.
Cross-validation results#
cv_results_ holds the mean and standard deviation
of the held-out score for each size that all folds reached. The size with the
best mean is selected, and ties go to fewer features.
cv_results = pd.DataFrame(selector.cv_results_)
print(
cv_results[["n_features", "mean_test_score", "std_test_score"]]
.round(3)
.to_string(index=False)
)
print(f"Selected: {selector.n_features_} features")
n_features mean_test_score std_test_score
51 0.659 0.043
41 0.651 0.033
31 0.628 0.040
20 0.627 0.038
1 0.581 0.034
Selected: 51 features
Selected features#
The final run uses all the data and stops at the selected size. As in the basic example, the selection keeps each categorical feature whole and does not hold both duration and credit amount.
selected_features = selector.get_feature_names_out()
for feature in categorical_features:
columns = [c for c in X_transformed.columns if c.startswith(f"{feature}_")]
n_selected = sum(c in selected_features for c in columns)
assert n_selected in (0, len(columns))
print(
"duration and credit_amount:",
[f for f in ("duration", "credit_amount") if f in selected_features],
)
duration and credit_amount: ['duration']
The full set is absent, and the best mean score is at 51 features. The score decreases as more features are removed. The plot shows it for each size, with the selected size marked.
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.errorbar(
cv_results["n_features"],
cv_results["mean_test_score"],
yerr=cv_results["std_test_score"],
marker="o",
capsize=3,
)
ax.axvline(
selector.n_features_,
color="gray",
linestyle="--",
label=f"selected: {selector.n_features_}",
)
ax.set_xlabel("Number of features")
ax.set_ylabel("ROC AUC (held-out)")
ax.set_title("CRFECV scores by number of features")
ax.legend()
fig.tight_layout()

Total running time of the script: (0 minutes 10.029 seconds)