I wrote this in 2017, about a year into my first data science job at an alternative business lender. The company was growing fast and lent to small businesses that banks usually turned away. I was the person building the risk models.

I came in expecting to apply the modelling standards I had learned in school. A lot of them did not survive contact with the data or the business. This post was my attempt to write down what I had learned so far, so I could keep getting better at the job.

1. Know your data and how it relates to the problem

Most of the lender’s business was loans to subprime businesses in Canada. Many of them lacked the collateral to get a bank loan, or wanted something faster and cheaper for a short-term need.

If you build a model to predict the risk of these businesses, you have to know exactly what kind of business you are lending to. It is easy, and tempting, to gather a pile of business characteristics and push them straight through a machine learning algorithm.

It is always better to choose, build and analyze those characteristics carefully. The relationships the model draws from them should make sense to someone who knows the business. Domain knowledge matters more than the algorithm.

2. Avoid models you cannot interpret

I learned this one the hard way. Reporting issues came up from several directions. One was the operations team, where the people handling loan applications struggled to explain model outcomes to business owners and their sales representatives.

Over time, people stopped trusting the scorecard. When a loan is rejected, the applicant and the sales rep deserve a fair reason. If you build a risk scorecard with a black-box method, the score is very hard to explain one characteristic at a time.

You also cannot easily tell a sales rep why a particular business scored the way it did when they ask for a specific reason for the decline.

3. Avoid models whose costs and benefits you do not understand

When I built my first scorecard, I did not realize how central it would become to the company. A model’s predictions turn into limits on product pricing and decide which products get offered to which customer segments.

That is why the characteristics you use to describe your customers need to be measurable and make business sense. One of them might turn out to mark the part of your customer base that makes the most money, or the part that loses the most.

4. Balance the model against what operations can support

One of the big promises of data science at the time was processing huge volumes of data in minutes. Applied work rarely looks like that. The real problems are operational: data that is not automated, not collected, or not organized.

Many of our customers were not comfortable with technology. The simplest way for them to apply was by email or on paper. That put a hard limit on what our data pipeline could do.

So I often had to trade predictive power against process automation. Sometimes a characteristic stayed out of the model because it could not be automated, was expensive to get, or simply could not be trusted.

I often saw weak validation statistics that more uncorrelated features would have fixed. But collecting those features was slow and costly. These limits ease over time as operations improve, the technology gets better, and you learn which data points are worth collecting.

5. Use industry standards as a benchmark, not a ceiling

Financial services is known for standard modelling practice and conservatism. I found it more useful to treat those practices as a benchmark and look for room to move through alternative underwriting.

That means knowing what data is unique to your company and would not normally be looked at by a large bank. In 2017, a business’s online presence was becoming hard to ignore. Bad reviews, a well-built website or good product photos could tip a financing decision either way.

This is where data science could add the most to underwriting. Text analysis of social media, location data and the judgment of experienced underwriters could tell you more than the handful of financial ratios a bank would normally be limited to.

Looking back from 2026

Most of this still holds. Domain knowledge first, interpretability for any model that affects a person, and honesty about what operations can support are as true now as they were then. If anything, they matter more.

Lesson two has become a rule rather than a preference. Credit decisions in most jurisdictions now come with stronger expectations around explainability, adverse action reasons and model risk management. A scorecard built with weight-of-evidence binning and logistic regression is still a sensible default. When a more complex model is used, it usually sits alongside SHAP-style explanations and a simpler challenger model, not in place of them.

Lesson four looks different today. Paper and email applications have mostly given way to online forms, bank account connections and open banking data. The operational bottleneck has moved from collecting data to governing it: consent, data quality and making sure a third-party feed means what you think it means.

Lesson five aged in both directions. Alternative data did become common in small business lending. But some of the signals I was excited about, like reviews and social media, turned out to be noisy and easy to game, and they raise fairness questions I did not think about enough at the time. I would now test any new signal for stability and for disparate impact before letting it near a decision.

The tooling has also changed. An LLM can now read a pile of bank statements or an application email and pull out structured fields in seconds, which removes part of the bottleneck I described. It does not remove the need to check that those fields are right, or to understand the business you are lending to.