Beyond the Scoreboard: What AI Can Learn From Sports Video

A squash scoreline is eleven numbers and a winner. It is the most compressed possible summary of forty minutes of decisions, and almost everything worth knowing is discarded in the compression.

I lost matches as a junior that I should have won, and the scoreline for those was identical to the scoreline for matches where I was simply outplayed. Same numbers, completely different problem. The information that would have distinguished them — that I was conceding the T after every drop, that my length collapsed in the third, that the opponent had worked out I would not go straight from deep on the backhand — was all present in the video. It just took a coach with time and patience to extract it.

That is the actual state of analysis in most racket sports. The information exists. Getting it out has historically required a person watching in real time, pausing, rewinding, and counting by hand.

The gap other sports closed with money

Football, basketball and tennis at the elite level solved this with capital: tracking installations, instrumented venues, dedicated analysts. The result is that a Premier League club knows things about a player's off-ball movement that a top-20 squash professional cannot find out about their own game.

Racket sports outside tennis never got that investment. And below the professional tier — where the overwhelming majority of competitive players are — it does not exist at all. Not because those players do not want analysis, but because the only mechanism for producing it was a human watching footage, and nobody has those hours.

This is the gap worth attacking, and it is widest exactly where the professional tools never reached.

What is actually recoverable from ordinary video

More than people expect, and the reason is that position over time is a very rich primitive.

Once you can recover where both players are on court throughout a match, in real-world coordinates rather than pixels, a large family of questions becomes answerable:

• Court coverage and distribution. Where a player spends the match, and how that changes across games.
• Recovery quality. Not just whether they return to the T, but how completely and how quickly, and how that degrades.
• Shot selection conditioned on position. The most useful single thing you can know about an opponent — not what they like to play, but what they play from where, under what time pressure.
• Rally construction. How points are built, and which patterns tend to precede a winner or an error.
• Pressure response. How all of the above changes between 3–3 and 9–9.

None of this requires a sensor on the player. It requires reading the video properly and calibrating the court so positions mean something.

Three questions video answers that a scoreboard cannot

Where do you concede the initiative? Matches turn on the moment control changes hands. That moment is invisible in the score — it usually happens two or three shots before the point ends. Positional data makes it visible, and it is often the same position every time.

Which patterns precede your errors? Unforced errors get counted, which makes them look like the problem. They are usually the symptom. The error at the end of a rally was set up by a position you took three shots earlier, and the pattern repeats all match.

What changes when it gets tight? Every player degrades under pressure, and every player degrades differently. Some stop attacking. Some attack too early. Some hold the T less well. Knowing your own failure mode is worth more than most technical work, and it is only observable across a body of matches.

Why "more data" is the wrong framing

The instinct when describing this is to talk about volume — thousands of shots, hours of footage. Volume is not the constraint.

The constraint is comparability. A single match analysed in isolation gives you observations you cannot calibrate. Was that a lot of errors from the backhand, or is that normal for you? Did she really recover badly in the fourth, or does she always look like that?

The value appears when the same measurements exist across many matches, and across many players, so that any individual observation has a reference frame. That is why the useful unit is not a match report but a persistent representation of the player — a Digital Twin that accumulates. One match tells you what happened. Forty tell you what is true.

The honest limits

Video does not see everything, and I would rather say so than oversell it.

It does not see intent. It sees that a player went short; it does not know whether that was a plan or a panic. It does not see physical state directly — it sees the consequences of fatigue, which is useful but is an inference. It does not see what a coach said between games.

And it is possible to over-read. Given enough measurements, patterns appear in noise. A system that surfaces every statistically detectable difference will bury the real ones under coincidences. Restraint about what to report is a design requirement, not a limitation — and one of the places where knowing the sport is what tells you which patterns are meaningful.

What this changes

The professionals will get better tools. That is fine and it is not the interesting part.

The interesting part is that a junior in a club with no analyst, filming on a phone, gets access to a kind of analysis that was previously available only to players with a coaching staff. That is the shift I care about, because I was that junior, and the difference between what I could see about my own game and what a well-resourced player could see about theirs was substantial and entirely structural.

The scoreline was never the information. It was just the only thing anyone could afford to record.