Why performance analysts do not and perhaps should not have final say in player transfers

Earlier this week, Prozone released a much-shared and discussed video presentation by data scientist Omar Chaudhuri, among the current leading lights in the analytics field. It comes just in time for silly season in European football, which will likely be affected by individual performances in the World Cup.  

It’s an excellent illustration of the importance of determining the predictive value of certain metrics, whether goals scored, shot conversion rates, or Expected Goals, before using them to make a critical decision over which player to buy.

It’s also a great example of how clubs can gradually add layers of data to get a better, more accurate picture of several potential recruits before making a costly, long term decision.  

In hindsight of course, Chaudhuri’s approach to painting an intelligent picture of basic player data seems obvious. Yet much of the public discussion over good or bad players still comes down to how many goals they scored or how many of their shots were goals, two almost universally-referenced statistics which generally shift wildly from one season to the next.

For non-forwards or attacking midfield players, there is even less to go on. Our value judgments for these positions are almost entirely limited to subjective impressions. Players that appear to be a ‘sure thing’ at 20 are often washouts at 25.

One would hope football clubs don’t scout players along similar lines, but rather use an approach similar to Chaudhuri’s.

Yet does that mean the performance analyst (or technical scout) should have final say in which players the club recruits?

No.

And I think Chaudhuri might agree, based on the non-statistical factors he outlines at the end of his presentation, factors like cost, personality, brand etc.

In any case, I doubt there are many clubs that give their performance and statistical analysts keys to the kingdom when it comes to recruitment. Rather, that responsibility is often left to a manager or, as is more and more the case, a technical director. Someone who isn’t necessarily steeped in statistical science and who ideally relies on analysts or technical scouts for advice, but may end up selecting potential transfer targets that don’t perfectly reflect the research of their club analyst.

This isn’t necessarily a bad thing.

To understand why, imagine for a moment you are hired as technical director for a struggling Premier League club with a relatively limited budget. You are tasked with buying the best players available to make the most impact with comparatively modest means. It is understood that you and not the manager has final say in recruitment.

It’s important to remember in this scenario that this is your job. If you’re not good at it, you’ll get fired. You’ll lose a reference. You may have to change careers. You will face financial hardship.

The challenge suddenly becomes much more vivid. Target players are no longer abstract names on a page, but future employees you must convince to leave their current clubs.

Moreover, you have a board overseeing you who are reticent to splash cash on non-proven players. You have a manager who may be on board with the analytics, or who may be a brilliant tactician who thinks stats analysis is a pile of BS. You have many possible transfer targets at different clubs in different financial straits with different managers and sporting directors and chief executives and executive vice-chairmen and on and on and on, each with their own agendas. This is before we even discuss player agents and third party ownership and all the rest.

It’s a difficult mix of ego, competition, finance, contract law and pure luck. That’s before we’ve even spoken about whether these players will add any value to your club.

Most of these vagaries however should not be the performance analyst’s problem. They’re yours. It’s clear that predicting future performance is only one element of successful recruitment.

With this in mind, you might want to talk to the first team coach to get an idea of the kind of players they want (whether specifically or by position and type), then talk to the traditional scouts on who they’ve been following, and finally use your PAs as a kind of fail safe to weed out the obviously bad choices, before sitting down and making a realistic assessment of who’s gettable and who’s a lost cause and then presenting a case to the board.

While this doesn’t give the analysts much say, it at least uses them to help avoid making stupid decisions. As Phil Birnbaum wrote, the goal of an analytics staff should be to... “first, concentrate on eliminating bad decisions, not on making good decisions better.”

Nevertheless, if this is me we’re talking about, I’m going to want to ensure my club has one of the best stat/performance analysis teams in the league, if not all of Europe. I’d have them make a convincing case for which areas of the team need strengthening, and which undervalued players are available to strengthen the team, and do this work in close coordination with the manager. And I would use the traditional scouts as the fail safe.

I’d listen closely to the methodology my PA staff has used to come to these conclusions, and see if I can spot any glaring weaknesses. Finally, I’d ask my team to calculate the baseline risk of failure for a blind player transfer, compare it to the lower level of risk our predictive model has provided for our transfer targets, and calculate a value in pounds we’ve saved in knowing the difference (this would come in quite handy in making a case to a skeptical board).

Finally I’d make my own assessment--which clubs do they play for? Who are there agents?--before lodging any bids.  

Either scenario demonstrates why the complex business of player transfers today means that no matter how good a predictive model is at determining the best players at lowest cost for a club to buy, there will always be other factors to consider. Player recruitment is clearly a team effort.