feature-engineeringDesign and implement features from raw data for machine learning models. Use when the user says "engineer features", "feature engineering for X", "create features from this dataset", "transform these variables", "encode categoricals", "handle skew", "create interaction features", "lag features", "extract features from text or dates", "improve model performa…
Independent GitHub snapshot · Automated classification · Original documentation governs
Based on "Feature Engineering for Machine Learning" by Alice Zheng and Amanda Casari. The core principle: features are the interface between raw data and model performance. The right transformation of an existing variable outperforms adding more data or tuning hyperparameters in most real-world settings. Feature engineering is domain knowledge encoded into math - and it is the highest-leverage step in the ML pipeline.
Before transforming anything, catalog what you have and what problems each column has.
For each column, note:
Create a feature engineering plan as a table before writing any code:
| Column | Type | Problem | Planned Transform |
|-------------|-----------|------------------|---------------------------|
| revenue | numeric | Right-skewed | log1p transform |
| country | nominal | 150 unique vals | Frequency encoding |
| signup_date | datetime | Raw timestamp | Extract day-of-week, hour |
| description | text | Unstructured | TF-IDF, 500 features |
Skewed distributions: Apply log transform to right-skewed variables. Use np.log1p to handle zeros.
df["log_revenue"] = np.log1p(df["revenue"])
Scaling: Standardize for distance-based models (KNN, SVM, neural nets). Tree-based models (XGBoost, Random Forest) do not require scaling.
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
df["revenue_scaled"] = scaler.fit_transform(df[["revenue"]])
Binning: Convert continuous to ordinal when the relationship to the target is non-linear.
df["age_bin"] = pd.cut(df["age"], bins=[0, 18, 35, 55, 100], labels=["<18", "18-35", "35-55", "55+"])
Capping outliers: Cap at a percentile rather than removing rows.
p99 = df["session_time"].quantile(0.99)
df["session_time_capped"] = df["session_time"].clip(upper=p99)
Match the encoding method to the cardinality and the model type.
| Cardinality | Model Type | Method |
|---|---|---|
| Low (< 10) | Any | One-hot encoding |
| Medium (10-50) | Tree-based | Ordinal / Label encoding |
| High (> 50) | Any | Frequency or target encoding |
| High (> 50) | Neural net | Embedding layer |
# One-hot
df = pd.get_dummies(df, columns=["color"], drop_first=True)
# Frequency encoding
freq = df["country"].value_counts() / len(df)
df["country_freq"] = df["country"].map(freq)
# Target encoding (use cross-validation to avoid leakage)
from category_encoders import TargetEncoder
encoder = TargetEncoder(cols=["city"])
df["city_encoded"] = encoder.fit_transform(df["city"], df["target"])
Datetime columns: Raw timestamps carry no signal. Extract structured components.
df["signup_hour"] = df["signup_ts"].dt.hour
df["signup_dow"] = df["signup_ts"].dt.dayofweek # 0=Mon, 6=Sun
df["signup_is_weekend"] = df["signup_dow"].isin([5, 6]).astype(int)
df["days_since_signup"] = (pd.Timestamp.now() - df["signup_ts"]).dt.days
Text columns: Start with TF-IDF for structured text. Move to embeddings when semantic meaning matters.
from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer(max_features=200, ngram_range=(1, 2), stop_words="english")
text_features = tfidf.fit_transform(df["description"])
Zheng and Casari's insight: the most predictive features are often combinations of existing ones, not raw columns.
Interaction features: Multiply or divide two numeric columns when their ratio or product has domain meaning.
df["revenue_per_session"] = df["revenue"] / (df["session_count"] + 1)
df["cost_per_click"] = df["spend"] / (df["clicks"] + 1) # +1 avoids division by zero
Aggregate features (group-level statistics): Capture behavior relative to a group.
group_stats = df.groupby("user_id")["purchase_amount"].agg(["mean", "std", "count"]).reset_index()
group_stats.columns = ["user_id", "user_avg_purchase", "user_std_purchase", "user_purchase_count"]
df = df.merge(group_stats, on="user_id", how="left")
Lag features for time series:
df = df.sort_values(["user_id", "date"])
df["revenue_lag_7d"] = df.groupby("user_id")["revenue"].shift(7)
df["revenue_rolling_30d"] = df.groupby("user_id")["revenue"].transform(lambda x: x.rolling(30).mean())
Before training, run these checks:
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
importance = pd.Series(model.feature_importances_, index=X_train.columns).sort_values(ascending=False)
print(importance.head(20))
Document the final feature set with a name, source column(s), transform applied, and rationale.
1. One-hot encoding high-cardinality categoricals Bad: One-hot encoding a "city" column with 500 unique values - creates 500 sparse binary columns. Good: Use frequency encoding or target encoding. Reserve one-hot for columns with fewer than 10 unique values.
2. Using the full dataset to fit encoders Bad: Fitting a TF-IDF or TargetEncoder on the entire dataset before splitting into train/test. Good: Fit only on training data. Apply (transform) to test data. This prevents leakage from test labels.
3. Adding features without checking importance Bad: Engineering 50 new features and feeding all of them to the model. Good: Run a quick feature importance check. Remove zero-importance features before final training.
4. Ignoring domain knowledge Bad: Letting the model figure out that revenue/sessions is meaningful by including both raw columns. Good: Create the ratio feature explicitly. Domain-derived features outperform raw features in most tabular settings.