Explainable AI for B2B Credit Risk Assessment
Designing transparent, audit-ready AI credit decisioning UX that eliminated the "black box" problem in B2B lending, reducing decision appeals by 40% and achieving full EU AI Act compliance.
↑ The "Glass Box" strategy: every AI credit decision narrated, not just delivered.
The "black box" problem in B2B lending
Our B2B lending platform used a sophisticated ML model for credit risk scoring. The model performed well statistically, but it created a UX crisis. SMB applicants received decisions like "Application Declined" with no explanation. Loan officers couldn't explain those decisions to regulators or to the customer sitting across from them.
94% model accuracy. 22% user trust in the decision. 41% officer confidence explaining it. 35% of declined SMBs filed an appeal, every one citing an inability to understand the decision rationale.
"The model is right but I can't explain why to my customer. That's a problem.", Loan Officer
Designing for two users at once
I needed to bridge the explainability gap: not by simplifying the model, but by designing a UX layer that translated complex AI outputs into human-understandable, actionable narratives for two very different user types.
The SMB Applicant
"I got declined but no one can tell me why. What do I need to fix to get approved next time?"
The Loan Officer
"I need to explain this AI decision to the customer and the regulator. Right now, I can't."
The "Glass Box" strategy: instead of hiding the AI, we narrate it. Every credit decision becomes a story, what the AI saw, what it weighted, and what the applicant can do next. Transparency as a feature, not an afterthought.
Three compounding failures of an unexplained model
Decisions with no rationale
35% of declined SMBs filed appeals, every one citing the same root cause: an inability to understand why they'd been declined at all.
Real regulatory exposure
The EU AI Act and FCA guidelines require high-risk AI systems to provide human-readable decision explanations. This wasn't a nice-to-have. It was a compliance deadline.
Loan officer paralysis
Officers couldn't contextualize AI scores, so they overrode the system based on gut feel, quietly undermining the model's value and creating an unaccountable decision trail.
How might we turn an accurate but unexplainable AI decision into a narrative that an SMB owner can act on and a loan officer can defend, without changing the model itself?
If we surface the top factors behind every decision in plain English, pair each decline with a concrete improvement path, and require structured documentation for every human override, then appeals will drop, officer confidence will rise, and every decision will be audit-ready by default.
Finding the exact words applicants and officers needed
Before designing any UI, I needed to know precisely what "explanation" meant to each audience, not in the abstract, but in the specific words they used when they were confused or blocked.
Contextual Interviews · n=12
Interviews with loan officers to map their decision-making mental model and where the AI's output failed to match it.
Appeal Analysis · 200+ applications
Coded 200+ declined applications' appeal text to identify the top confusion triggers, ranked by frequency.
EU AI Act Compliance Audit
Worked with legal to define the minimum explainability threshold the design had to clear, not just the ideal.
Competitive Benchmarking
Benchmarked explainability UX patterns across Stripe, Experian, and Funding Circle to avoid reinventing solved problems.
Top 5 reasons SMBs appealed a decline
What the appeal data actually said
"No reason given" wasn't a UX complaint. It was the whole problem
68% of appeals cited this single issue. Every downstream fix, factor charts, roadmaps, audit trails, flowed from solving this one gap first.
Applicants wanted a path forward, not just a reason
54% asked "what do I improve" , almost as many as asked "why." An explanation without a next step was still a dead end.
Officers needed the same explanation the applicant got, plus an audit trail
Officers weren't asking for a different explanation, they needed the same one, framed in a way that also satisfied a regulator looking over their shoulder.
The rules we designed by, and why
1 · Explain the decision, don't just deliver it
Derived from 68% of appeals citing "no reason given" as the trigger.
→ Every decision ships with a Decision Summary Card and Top Factors Panel by default, not on request.
2 · Translate model language into plain English, always
Derived from benchmarking , competitors that showed raw ML scores scored worst on user trust.
→ Factor Contribution Chart uses plain-English labels, never ML jargon like "feature weight."
3 · Turn rejection into a growth path
Derived from 54% of appeals asking "what do I improve," almost as many as asked "why."
→ AI-generated Credit Health Roadmap with projected score impact per action.
4 · Make every override accountable
Derived from officers overriding the model on gut feel with zero documented reasoning.
→ Structured Override Audit Trail, no override ships without a documented, regulator-ready reason.
From confusion data to a compliance framework
01 Iteration 01 · Discovery Mapping the Officer's Mental Model
The 12 contextual interviews plus the 200+ appeal analysis ran in parallel , officer interviews explained the "why" behind the appeal patterns we were seeing in the data. This combination was more useful than either method alone: the data told us what was breaking, the interviews told us why it broke that way.
02 Iteration 02 · Concept & Benchmarking Prototyping the Factor Contribution Chart
Benchmarking Stripe, Experian, and Funding Circle showed a consistent pattern: horizontal bar charts with signed contributions (+/- points) outperformed pie charts or raw score displays for comprehension. I prototyped several factor-count variants before settling on showing the top 5, enough for a real answer, not so many it read as a data dump.
03 Iteration 03 · Compliance Validation Stress-Testing Against the EU AI Act
The Regulatory Constraints Ideation workshop with Legal and Data Science tested every design decision against a single question: would this survive a regulator's audit? That pressure-test is what produced the "Glass Box" framework, a documented standard for exactly how much explanation every AI-assisted decision must carry.
The choices that shaped the outcome
Decision 01 · Top-5 factor breakdown vs. a single headline reason
Tension: A single reason is simpler to design and less overwhelming to read.
Choice & trade-off: We showed the top 5 factors, not one. A single reason invited "but what about everything else?" questions that a fuller picture headed off entirely.
Decision 02 · Personalized AI-generated roadmap vs. generic improvement tips
Tension: Personalized, model-generated guidance requires more engineering than a static tips page.
Choice & trade-off: We built the personalized Credit Health Roadmap anyway, generic tips read as boilerplate compliance text, exactly the kind of "black box" language we were trying to eliminate.
Decision 03 · Structured override form vs. free-text override notes
Tension: A structured form takes longer for a busy officer to complete than typing a quick note.
Choice & trade-off: We required structure, free text wouldn't have satisfied audit requirements or given consistent data for future model retraining.
Explainability as a first-class feature
Every AI decision ships with its own decision summary, factor breakdown, and improvement path, no clicking through to "learn more" required.
AI Confidence: 94% · Reviewed: March 2025
Top Influencing Factors
↑ The Decision Summary Card, a signed, plain-English factor breakdown replacing a bare "Declined" verdict.
Decision Summary Card
Plain-English verdict with a confidence indicator, replacing raw technical score outputs.
Credit Health Roadmap
AI-generated improvement plan with a projected score change for each recommended action.
Request Human Review
One-click escalation with a pre-populated context brief, cutting officer prep time by 15 minutes per case.
Design doesn't happen in isolation
Legal: defining "explainable enough"
Legal turned EU AI Act language into a concrete design brief: exactly which factors, at what granularity, had to be disclosed for a "high-risk" credit decision.
Data Science: translating model internals honestly
Data Science validated that the plain-English factor labels genuinely matched what the model was weighting, a translated explanation that misrepresents the model is worse than no explanation at all.
Loan operations: validating the override workflow
Operations leadership stress-tested the structured override form against real edge cases to confirm it wouldn't slow officers down during high-volume periods.
How we arrived at the solution
Service Blueprinting
Mapped the end-to-end loan approval process, identifying the exact moments where "black box" AI decisions caused officers to pause or override the system.
Regulatory Constraints Ideation
Cross-functional session with Legal, Data Science, and Product to define how much AI transparency was legally required vs. technically feasible.
Illustration Placeholder
Prompt: A stylized illustration of a UX workshop for an AI credit-risk product, sticky notes and dashboard wireframes on a glass wall, in emerald green accent tones. Rendered as a clean illustration on a fully transparent background (no backdrop, scene, or color fill), so it displays cleanly on both light and dark page themes.
Transparency built trust: and revenue
"For the first time, I can sit across from an SMB client, show them exactly why the AI scored them the way it did, and walk them through what to do next. That's a game changer for client trust."
Senior Loan Officer, B2B Lending Platform
Honest reflections from the process
What Worked
Letting appeal data set the priority order
Ranking confusion triggers by frequency (68% down to 28%) meant we solved the highest-leverage problem first instead of guessing which explanation gap mattered most.
Treating compliance as a design constraint, not a legal afterthought
Running the Regulatory Constraints Ideation workshop before high-fidelity design meant we never designed something legal would later reject.
What I'd Do Differently
Interview SMB applicants directly, not just loan officers
Our research leaned heavily on officer interviews and appeal text. Direct moderated sessions with SMB owners reading their own Decision Summary Card would have sharpened the plain-English copy even further.
Instrument the Credit Health Roadmap's real-world accuracy
We shipped projected score changes per recommended action, but didn't build a feedback loop to confirm those projections held true once applicants actually acted on them, that's the next thing I'd instrument.