Making the Sausage: Building and Using Models in the Real World (pt 1)

Someone somewhere almost certainly financed these…

For about 25 years, I helped finance companies, many of which are household names, use a sophisticated financial modeling tool. I got this gig because I’m good at explaining complicated things. To explain them, I needed to understand them. Thus, I spent years working with the guys who were building the model so I would understand it so I could explain it. I sometimes had direct input, but mostly I saw what was built after the fact. Either way, I had to understand it and how it worked in the real world.

What I propose to do here is walk through how a model works, how people use it, the gotchas, the assumptions, and what a model can and can’t do, illustrated by lightly scrubbed real-world experiences. As I go along, I’ll comment on the Imperial College model (as I understand it – not so much the mathematical details as how it was presented and used) versus how our model was used. This may end up a multi-part series.

Hope the regular readers find this interesting.

First, the users of the model: financial analysts and sales people, generally working for large sophisticated finance companies. The analysts did reporting and what-iffing; the sales people used the model to generate quotes for, generally, large financing deals. Examples: Company A wants a fleet of railcars, maybe worth $100M. They plan to use them over the next 30 years. As a salesperson from Large Finance Company B, I want to propose the following: my company will buy those railcars and lease them to Company A for a certain monthly payment. I will use the model to create a number of scenarios for Company A, showing different options. There will be some back-and-forth, during which time I will use the model to run other scenarios.

*Why* company B wants to purchase and then lease a fleet of railcars to Company A is beyond what I’m concerned with here – it’s some combination of cash flows, tax, and accounting considerations that make (or not) this arrangement appealing to both companies. What I want to do here is give a little taste of how complicated this all can get, and how those complications and uncertainties get addressed.

The first thing to note: there are probably something in the neighborhood of a trillion dollars worth of arrangements like this out there in the world at any time. Individual finance companies can have $100B or more invested in these sorts of deals. Screw up, and you can go out of business, sometimers in spectacular ways. People have careers riding on these deals working out. Therefore, every calculation, every assumption, every number MUST be demonstrated. Our model had hundreds of reports, validating every number and calculation. You could change one number or assumption, and watch that change flow through the deal in the reports. This is a critical distinction: A real-world model used by thousands of people with real money riding on it CANNOT be some sort of secret – giant finance companies had teams of people looking at it, making sure we got it right. Only after years of this scrutiny was the model accepted as an authority, and then only because everybody knew that not only they themselves had verified it, but their competitors had as well! Compare and contrast: Ferguson releases the conclusions of his model first, with only a light gloss on the mechanics. There were no teams of people, let alone teams of adversarial people, vetting his methods and output. He would have been jeered out of the room in my world. (1) Yet, way more than a trillion dollars was in play!

Our financial model had thousands of variables, but they fell into a much smaller number of logical groupings: dates, amounts, treatments, and settings. Anytime anything happens, it happens on a particular date. If the event involves money, it is a particular amount of money. But when money moves around, there are tax and accounting treatments to be spelled out (for every applicable country and state!). Finally, there’s a bunch of other stuff, such as expense allocations, yield definitions, costs of borrowing, constraints on calculations – stuff, and lots of it.

The way the model is used: Setup: an expert or a small group of experts from the finance company would sit down with me, or some other expert on the model, and we’d walk through the options. Sometimes, it was a simple process, we’d get the basic knocked off in an afternoon, and after a few iterations over the course of weeks or months, they’d be happy with how it worked for them. Other times, options were defined and redefined and tweaked over years and even decades – these were the largest companies, with the biggest teams of experts, and a budget for customizations.

Once the model was set up to incorporate the assumptions and preferences of a particular company, the analysts and sales people could use it to forecast, report, and put together proposals. No two companies made exactly the same assumptions, or defined key elements such as yields exactly the same way.

What this taught me: You must define terms! Every day terms critical to understanding what the model is telling you don’t mean the same thing to different groups or even individuals within a group. The million dollar question is: how much do you need to charge in order to make this deal work? You can measure this in cash, in simple rates, in more complex yields, in risk-adjusted returns over forecast money cost curves, and on and on. Each approach, each step of each approach, requires definition – what are you measuring? Against what?

Sometimes, even the experts don’t know. Sometimes, there are interesting interactions – change this, and the change will cascade through the numbers in sometimes non-obvious ways. Expectations will sometimes be wrong, where a change that looks like it should move the needle one way instead moves it the other way, because the interactions were not clearly understood.

Yet, at the end of the day, we could prove out every number. We could never have licensed copy 1 if we could not.

Even more than the usual skeptic, I turn a gimlet eye on any claims that a model has shown anything. Models show what you want them to show. You either spell out all the assumptions, demonstrate all the interactions, answer all questions, and stand up against your critics (we had ours) or you STFU.

Finally, the way you build a model like ours is through constant interactions with real-world experts and constant iterations. You get everybody’s input and feedback, and thus get everybody invested. There is no reason to trust a model in any field that has not gone through that process. Otherwise, the model is just your prejudices, your assumptions, and your blind spots. Plus math. Such a model merely tells us what you think and what you’d like to believe, nothing more.

  1. To be completely honest, there would probably be management types in the room who did not really understand the issues. Who knows how they would react, but the numbers guys would have gone nuts.

Author: Joseph Moore

Enough with the smarty-pants Dante quote. Just some opinionated blogger dude.

One thought on “Making the Sausage: Building and Using Models in the Real World (pt 1)”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s