I have always wondered if it was possible to know who needs mental health support before things reach a crisis point. Caseloads are large and the signs someone is deteriorating are not always obvious until it's too late. This project explores whether a predictive model, built entirely on data services already collect, could help flag patients at higher risk of crisis referral within 60 days.
I worked with a synthetic, pseudonymised dataset of 2,040 records across 28 fields: demographics, clinical scores, service use patterns and a binary outcome. Below is the pipeline I went through, and I have highlighted where the model works well and where it doesn't.
Data Quality
The first thing I found, before any real analysis, was a field called Post_Index_Crisis_Flag. It turned out to be a near-perfect predictor of the outcome, with 99% agreement, because it encodes information from after the point I was trying to predict from. I excluded it immediately. Target leakage is silent, and the only way to catch it is to understand your data before you touch it.
Beyond that, the dataset had a fairly typical set of real-world problems:
- 40 duplicate Patient IDs: exact duplicates, dropped by keeping the first occurrence, leaving 2,000 records
- Inconsistent Gender coding: "M", "male", "Male", "F", "female", "Female" all present; standardised to Male / Female
- Invalid Age values: including a minimum of -4; nullified outside the range 0–110
- Out-of-range IMD Decile values: 57 records with values of 0 or 11 (valid range is 1–10)
- 170 negative Days Since Last Contact: a date recording error; set to missing
- Implausible BMI values: outliers below the 1st and above the 99th percentile replaced with the median
Figure 1: Missing data and outlier treatment across key fields.
I handled missing data differently depending on the field. Ethnicity, Referral Source, and Primary Diagnosis got an 'Unknown' category rather than an imputed value, since guessing felt like it could introduce bias. Numeric fields used median imputation, and Smoking Status and Care Coordinator used mode imputation. Discharge Date became a binary flag instead, since a blank value there is meaningful clinical information, not just a gap.
What the Data Showed
Once the dataset was clean, two things stood out. The outcome was imbalanced: about 74% of patients had no crisis referral within 60 days, 26% did. That's enough to make accuracy alone a misleading metric, since predicting "no crisis" for everyone would score 74% and be useless.
No single feature strongly predicted the outcome on its own. Correlation analysis confirmed it: crisis is shaped by the interaction of severity, engagement and history, not any one factor alone. Patients who went on to crisis referral did have higher average Previous Admissions (0.86 vs 0.45), more A&E visits in the prior 12 months (1.52 vs 0.89), and longer gaps since last contact (214 vs 189 days).
Figure 2: Outcome distribution and mean feature differences between crisis and non-crisis groups.
Feature Engineering
Raw fields do not always translate cleanly into model inputs, so I derived several new features, each with a clear clinical rationale:
- PHQ-9 and GAD-7 severity bands: bucketed using validated clinical thresholds (Minimal / Mild / Moderate / Severe) rather than fed in as continuous values. This reduces noise and aligns with how clinicians actually interpret these scores.
- Age bands: non-linear age effects are better captured categorically than as a continuous variable
- IMD category: Low / Medium / High deprivation groupings from the raw decile
- Crisis Contact Flag: binary indicator of any prior crisis contact in the last 12 months
- DNA Count Flag: flags patients with a high Did Not Attend rate (above the 40th percentile) as a proxy for disengagement
- Is Open Case: extracted from Discharge Date; a missing date means the case is still active
I then dropped the raw columns superseded by these engineered features, to avoid redundancy.
Model Results
Because of the class imbalance, I used Average Precision (PR-AUC) rather than accuracy as my primary metric, since it focuses on how well the model identifies true crisis cases. I trained two models, Random Forest and Logistic Regression, inside full pipelines with 5-fold stratified cross-validated grid search.
Logistic Regression won on every metric that mattered:
| Metric | Logistic Regression | Random Forest |
|---|---|---|
| PR-AUC | 0.441 | 0.377 |
| ROC-AUC | 0.694 | 0.638 |
| Recall (default threshold) | 0.612 | 0.019 |
Figure 3: ROC curves, metric comparison, and confusion matrix at threshold 0.35.
An ROC-AUC of 0.694 felt like a realistic result. This means that there is a ~70% chance the crisis patient receives a higher risk score when compared with a non-crisis patient. Mental health crisis prediction in the literature typically sits between 0.65 and 0.78, and anything much higher usually means leakage, which is exactly what that Post_Index_Crisis_Flag showed earlier. A Recall of 0.612 means that the model correctly identifies 60% of patients who actually had a crisis.
What Mattered Most
The strongest predictors, from the Logistic Regression coefficients, were all features already collected in routine clinical practice, which matters if this were ever adopted:
Figure 4: Logistic Regression coefficients: red bars increase crisis risk, teal bars decrease it.
Features on the right (red) increase predicted crisis risk; features on the left (teal) lower it. A few things stood out:
- DNA count and rate: disengagement from services is the strongest signal
- Previous admissions and A&E visits: historical crisis patterns predict future ones
- Schizophrenia and Bipolar diagnoses: associated with deterioration independently of other factors
- No care coordinator allocated: This is the most actionable finding in the model
- GAD-7 and PHQ-9 severity: higher severity bands increase risk
Choosing a Threshold
The model produces a probability score, and then it's a judgement call where to draw the line. The default of 0.5 isn't appropriate here. Missing a crisis case (a false negative) costs far more than flagging someone who turns out to be stable (a false positive), so a lower threshold that catches more crises at the cost of more false flags felt like the right trade-off.
Figure 5: Sensitivity, specificity and precision across decision thresholds.
| Threshold | Sensitivity | Specificity | Precision |
|---|---|---|---|
| 0.25 | 100.0% | 0.7% | 25.9% |
| 0.30 | 100.0% | 6.4% | 27.0% |
| 0.35 ✓ | 95.1% | 20.9% | 29.4% |
| 0.40 | 86.4% | 35.0% | 31.6% |
| 0.50 | 61.2% | 68.0% | 39.9% |
I settled on 0.35, which catches 95.1% of actual crisis cases. The trade-off is lower specificity, but that is the right call for a screening tool. It is not fixed though and I would want to revisit it with clinical stakeholders based on team capacity.
Risk Stratification
Applying the model across the cohort, I grouped predicted probabilities into four risk tiers. There is no Low risk group, reflecting the nature of an active caseload where baseline risk is elevated across the board. That is a finding worth flagging to commissioners on its own.
Figure 6: Predicted risk tier sizes and observed crisis rate per tier.
| Tier | Patients | % of Cohort | Observed Crisis Rate | Suggested Response |
|---|---|---|---|---|
| Low | 0 | 0% | N/A | Routine schedule |
| Medium | 618 | 30.9% | ~13% | Enhanced monitoring |
| High | 1,022 | 51.1% | ~20% | Active review; care plan check |
| Very High | 360 | 18.0% | ~40% | Proactive outreach; same-week contact |
The Very High tier of 360 patients is where proactive outreach would have the greatest impact: same-week clinical review, checking whether a crisis plan and care coordinator are in place.
What This Model Is and Isn't
This is a screening tool. It surfaces patterns across a large caseload that would be hard to spot manually, and it helps clinical teams direct attention, it doesn't replace their judgement.
It is not diagnostic and should not trigger automated actions. A high risk score means "look at this person this week," not "this person will crisis-refer." There are things it can't see: what was said in an appointment last Tuesday, or whether a key relationship has broken down. Clinical context will always carry information the data doesn't.
Recommendations
- Prioritise care coordinator allocation for the Very High tier, the most actionable and modifiable finding
- Build Days Since Last Contact and DNA rate into caseload dashboards, regardless of whether a model is used
- Improve Referral Source recording, currently 25.6% unknown
- Revisit the threshold with clinical stakeholders, since 0.35 is a starting point, not a fixed rule
- Validate on prospective real-world data with a proper temporal train/test split
The full code for this project is available on GitHub..