The housing market is a fascinating one. Prices fluctuate, mortgage rates move equity around, and buying or selling a house takes real thought and planning. But before any of that, a more basic question came to mind: can I actually predict the price of a house before it's bought or sold? And what really drives that price?
Is it the house itself: number of rooms, number of bathrooms, a big garden, whether there's a garage and how large it is? Or is it the neighbourhood: good schools, proximity to shopping centres, closeness to hospitals?
As a data scientist, what better way to find out than by building a machine learning model to understand the key drivers of home sale prices?
The Data
I used the
Ames Housing dataset
from Kaggle. The training dataset had 1,460 houses and 80 columns.
On initial assessment, there was a lot of missing data. For example,
PoolQC and MiscFeature had 1,453 and 1,406
rows missing respectively out of 1,460. That's a lot, so both
columns were dropped outright, since the data present wasn't enough
to work with.
For the remaining missing values, treatment was done based on data type: numeric columns were filled with the column mean, and categorical columns were filled with the mode.
Finding the Features That Matter
With a clean dataset, the next question was: which of these 78
features actually move the needle on price? A correlation matrix
against SalePrice narrowed the field considerably. Only
a handful of features cleared a 0.5 correlation threshold:
| Feature | What it means |
|---|---|
| OverallQual | Overall material and finish quality |
| GrLivArea | Above ground living area, square feet |
| GarageCars | Size of garage in car capacity |
| GarageArea | Size of garage, square feet |
| TotalBsmtSF | Total square feet of basement area |
| 1stFlrSF | First floor square feet |
| YearBuilt | Original construction date |
| YearRemodAdd | Remodel date |
| TotRmsAbvGrd | Total rooms above grade |
| FullBath | Full bathrooms |
Overall quality rating came out on top by a wide margin, followed by
above ground living area. This points toward intuition: buyers pay
for well built, spacious homes. I ran a Pearson correlation test on
each of these top features to statistically confirm the correlation,
and they returned p-values so small (some below 1e-150) that the
relationships were unambiguously significant, not just noise.
Multicollinearity was present too. GarageCars and
GarageArea, for instance, were highly correlated with
each other. That's something to keep in mind before leaning too
heavily on a linear model's individual coefficients.
Model Building
I built a number of models, starting simple and working up:
-
Simple linear regression, one predictor: using
just
OverallQual, which had the strongest correlation to SalePrice - Multiple linear regression, six predictors: using the top correlated factors to SalePrice
- Multiple linear regression, all predictors
- Pipeline with scaling and polynomial features
- Ridge regression with GridSearchCV
Model 5 was the best performer, with an R² on test data of 0.78 and RMSE of about $38,553. Here's how all five compared:
| Model | R² | RMSE |
|---|---|---|
| Simple linear regression (OverallQual) | 0.65 | $48,890 |
| Multiple linear regression (6 predictors) | 0.69 | $45,613 |
| Multiple linear regression (all predictors) | -923,760 | $79.2M |
| Pipeline (scaled + polynomial) | 0.61 | $51,151 |
| Ridge regression (GridSearchCV) | 0.78 | $38,553 |
Looking at the third row, throwing every predictor into a plain linear regression made the model underperform badly. It is a clear reminder that as more features are added, you need something in place to stop the model from over-relying on them, which is exactly what the regularization penalty (GridSearchCV) in Ridge regression does.
Final Predictions
With a winning model in hand, the last step was applying it to the
actual Kaggle test set: 1,459 houses which had no
SalePrice. A slight issue came up here. Because of the
one-hot encoding applied to categorical columns, the train and test
sets didn't end up with quite the same columns. Some categorical
values, like a RoofMatl of Membran, simply
never appeared in the test set. To fix this, I added those missing
columns back into the test data as all-zero columns, so the model
always saw the same feature shape it was trained on. From there, I
was able to predict SalePrice for each house
Id
and package it into a downloadable CSV.
In Conclusion
More features aren't automatically better, as proven by Model 3. However, more features plus the right constraints, in this case regularization via Ridge regression, can be. This is because Ridge regression does not throw every feature at the wall and hope for the best. It adds a penalty that shrinks the influence of less useful or redundant predictors instead of letting them run wild. That penalty keeps the model grounded enough to generalise to houses it hasn't seen before, the difference between a model that memorises and one that actually learns.
The full notebook is available on GitHub.