SkillsSkill 3 of 5
Catch the model or data error before customers do
Split a calm average by segment, find the input that changed, and fix the decisions already made.
Worked example
Approvals are within tolerance. Do you sign off?
An average can stay inside tolerance while one group of customers is being wrongly declined.
The question
Fernmoor Bank, a fictional regional bank, approves personal loans with a scorecard. Each month a credit risk analyst signs off the monitoring report. The alert fires if the total approval rate moves more than 6 points.
October approval rate 55.0% vs 60.4% in September. Change within tolerance. No action needed.
| Sept apps | Sept approved | Sept rate | Oct apps | Oct approved | Oct rate | |
|---|---|---|---|---|---|---|
| Salaried | 800 | 480 | 60.0% | 800 | 488 | 61.0% |
| Self-employed | 200 | 124 | 62.0% | 200 | 62 | 31.0% |
| Total | 1000 | 604 | 60.4% | 1000 | 550 | 55.0% |
Invented data. The total fell 5.4 points, under the 6-point alert. Self-employed applicants are 20% of volume in both months.
- Scorecard
- No change.
- Income verification
- The vendor moved to a new data feed.
- Operations
- Some self-employed files show verified income of $0.
“The monthly report says October approvals are within tolerance. Do you sign it off, and if not, what do you do?”
Your turn
Say your answer out loud, or jot notes below. Both work. Or skip to the worked answer.
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Split the average
What would you do here? Open the move you’d make.
Split by segment: salaried rose to 61%, self-employed halved from 62% to 31%.
The strong move. The total fell 5.4 points, under the 6-point alert, because self-employed is a fifth of volume. The average hid it.
Sign off: the 5.4-point drop is inside the 6-point tolerance.
Tempting, but it misses something. The alert watches the total. Self-employed approvals fell 31 points; the tolerance was never built to see that.
Put it down to seasonality; approvals often dip in the fall.
Tempting, but it misses something. Salaried approvals rose a point the same month. A seasonal dip wouldn’t hit one segment and spare the other.
See how a strong candidate says it
“The total fell 5.4 points, under the 6-point alert. But self-employed approvals halved, from 62% to 31%, while salaried rose a point. A 31-point drop in a fifth of volume, net of the salaried gain, is the 5.4.”
Why it works: The aggregate passed the check because the broken segment is small.
Step 2: Find the input that changed
What would you do here? Open the move you’d make.
Conclude self-employed applicants got riskier this month.
Tempting, but it misses something. Nothing in the data says so. The income feed changed on October 1 and some self-employed files show $0 verified income.
Ask for the scorecard to be retrained on October’s data.
Tempting, but it misses something. The scorecard didn’t change; an input did. Retraining on $0 incomes would teach the model the error.
Trace the inputs: the income feed changed the day the drop began, and some self-employed files show $0 verified income.
The strong move. A plausible output from a bad input is still wrong. The fix starts where the data changed.
See how a strong candidate says it
“The scorecard didn’t change. The income feed did, the day the drop began, and some self-employed files show $0 verified income. That looks like a data error, not weaker applicants.”
Why it works: Plausible output from a bad input is still wrong.
Step 3: Fix forward and look back
What would you do here? Open the move you’d make.
Fix the feed and move on; new applications will be right.
Tempting, but it misses something. It leaves the 138 self-employed applicants already declined in October. At September’s rate, about 62 of them would have been approved.
Send self-employed files to manual income review until the feed is fixed, then re-decision October’s 138 self-employed declines.
The strong move. It stops new harm and repairs old: at 62%, about 62 of the 138 would have been approved.
Pause all personal lending until the vendor confirms the fix.
Tempting, but it misses something. Salaried decisions, 800 a month, look healthy at 61%. Stopping them adds harm without removing any.
See how a strong candidate says it
“I’d send self-employed applications to manual income review until the feed is fixed, then re-decision October’s 138 self-employed declines. At September’s rate, about 62 more of them would have been approved.”
Why it works: Stopping new harm is half the job. The customers already declined are the other half.
Step 4: Close the gap in monitoring
What would you do here? Open the move you’d make.
Add segment-level alerts, check the reasons in the 138 decline letters and ask compliance to test fair-lending impact.
The strong move. The 6-point total alert missed a 31-point segment drop. The control that failed is what needs fixing.
Tighten the total alert from 6 points to 5.
Tempting, but it misses something. It would have caught this month by 0.4 of a point, but a smaller segment breaking would still slip under it.
Write it up as a vendor incident and let the vendor own it.
Tempting, but it misses something. The vendor owns the feed; the bank owns its decisions. 138 customers got decline letters from Fernmoor.
See how a strong candidate says it
“I’d add segment-level alerts, check the reasons in those 138 decline letters, and ask compliance to test whether the error fell harder on any group protected under fair-lending law.”
Why it works: It fixes the control that missed the problem, not just this month.
The answer, in short
- Decision
- Don’t sign off. Treat it as a data incident: manual income review for self-employed applicants now, re-decision October’s 138 self-employed declines, and add segment alerts.
- Because
- Self-employed approvals fell from 62% to 31% the month the income feed changed. The 5.4-point total stayed under the alert only because the segment is 20% of volume.
- What would change it
- If source documents showed October’s self-employed applicants really had lower income, the declines would stand and the work becomes a monitoring fix.
Common traps
- Trusting the aggregate because it passed the alert.
- Blaming the applicants before checking the inputs.
- Fixing the feed but leaving past declines alone.
- Treating it as a technology ticket, not a customer issue.
Check your answer
- Did I break the result down by segment?
- Did I look for what changed in the inputs?
- Did I deal with customers already declined?
- Did I fix the monitoring that missed it?
Want feedback on how you’d answer? Practice “Catch the model or data error before customers do” with an AI interviewer on Socratify.
Last reviewed Nov 9, 2026