In finance, an LLM needs structured signals to be accurate
Judging credit risk with a model alone ranks worse than a classic model — feed the classifier's key features and probability into the prompt and the metrics catch up.
Key numbers
- Task: credit-card default prediction on the Default of Credit Card Clients dataset, comparing a fitted classifier, LLM prompting, and combinations of the two.
- Alone, a few-shot LLM has the highest recall (0.47) and F1 (0.50) of any single model but ranks worse than a random forest (AUC-ROC 0.72 versus 0.79).
- Telling the LLM to imitate the classifier changes nothing significant; pruning the prompt to the classifier's eight most important features raises recall by 0.071 and F1 by 0.032.
- Adding the classifier's predicted probability to the prompt lifts the LLM's AUC-ROC from 0.72 to 0.78 — matching the random forest — while keeping 0.118 higher recall.
Method and sample
arXiv paper “When Does a Classifier Help an LLM? Classifier-Guided Prompting and Hybrid Classifier-LLM Models for Credit-Default Prediction” (2026-08-30): classifier, LLM prompting and hybrid approaches compared on a public credit-default dataset, reporting recall, F1, AUC-ROC and AUC-PR with bootstrap confidence intervals.
What it means for brands
Financial conclusions need structured external signals behind them: give the model key fields and quantified outputs and the answer gets closer to professional judgement. The same applies to how AI describes a financial institution — it needs verifiable structured facts (ratios, licences, product terms), not just marketing copy.
How SourceSight handles it
For regulated projects SourceSight first checks whether AI can retrieve credential and product-term pages, then the content factory structures those facts; reports keep the 10 engines separate so one engine's verdict does not stand in for the rest.
Limits
The experiment uses one public dataset and a specific prompt design and does not represent real underwriting; compliance requires human review, and model output cannot be used as a decision on its own.
References
Run it on your own project
What the same prompts return across 10 engines is only knowable by running them.
Related research
Cross-category recommendations: +36% add-to-cart on item pages, +15% on cart pages
Recommending general merchandise to grocery shoppers — a milk frother with the milk — has long been hard; LLM retrieval plus real-time cart context moved production A/B metrics.
Industries · 2026-10-07Home services: 22% of homeowners already use AI to find a provider, and 80% of businesses don't know how to be found
The local services funnel is moving upstream: before a homeowner opens a website or calls, AI has already synthesised reviews, reputation and service descriptions into one answer.