Teaching AI to Understand a Squash Match

If you have never tried to build computer vision for squash, the problem sounds like a solved one. Object detection has been commodity technology for years. Point a model at a video, get boxes around the players, done.

Then you look at actual squash footage.

Two players share a court roughly six and a half metres wide. They occlude each other constantly — that is not a failure case, it is the sport. The ball is small, black, and moves fast enough to smear across a frame. The back wall is glass, so the crowd is in your background. The side walls are white, so the ball disappears against them at the wrong exposure. And the camera is usually a phone on a tripod behind the glass, at whatever height the person filming happened to choose.

That last constraint is the one that shapes everything. We could make this much easier by requiring calibrated multi-camera rigs and instrumented courts. We would also have built a product that works in about forty venues worldwide. The interesting problem — the one worth solving — is reading ordinary footage of an ordinary match.

Here is how that problem decomposes.

Layer one: detection

Find the players. Find the ball.

Player detection is the tractable half. Off-the-shelf detectors handle "there is a person here" well, and the hard part is not detection but identity — keeping player A labelled as player A through every occlusion, every time they cross, every moment when one is entirely behind the other. Squash is adversarial toward tracking-by-detection in a way that most team sports are not, because the two objects you are tracking spend the match deliberately occupying the same space.

Ball tracking is a genuinely different problem and deserves to be treated as one. The ball is small, low-contrast, frequently motion-blurred, and often not visible at all for stretches of a rally. Approaches built for players do not transfer. What works is exploiting the thing that makes the ball different from everything else in frame: its trajectory is physically constrained. A ball follows a path. A false positive does not.

Layer two: geometry

A detection in pixel coordinates is close to worthless. "The player is at (840, 512)" means nothing that generalises, because it depends entirely on where the camera was standing.

Court calibration is what converts pixels into metres — recovering the mapping between the image and the real court, using the one thing every squash court reliably provides: known geometry. The court markings are standardised. The short line is 5.44 metres from the front wall. The service boxes are 1.6 metre squares. That known structure is enough to solve for the camera.

Once you have that, everything downstream becomes portable. "Recovered to within a metre of the T" means the same thing in every venue, from every camera angle. Without it, no positional claim you make survives a change of filming position, and you have built a system that works on your test footage and nothing else.

I would argue calibration is the highest-leverage component in the entire stack, and it gets the least attention because it is not glamorous.

Layer three: time

This is where the real difficulty lives.

A detection is not an event. "Player at position X at time T" is not a fact anyone in squash cares about. The facts they care about are: that was a drop shot, she recovered late, that rally was won on the third crosscourt.

Getting there means recovering structure over time:

• Segmenting rallies. Where does play start and stop? Not trivial when the camera runs continuously through let calls, towel breaks, and arguments with the referee.
• Detecting shots. A shot is a moment of contact, inferred largely from the ball's trajectory changing direction.
• Attributing shots. Which player hit it? Obvious to a human, genuinely hard when both players are in the same region of frame at the moment of contact.
• Classifying shots. Drive, drop, boast, lob, kill — categories defined by trajectory and court geometry, not by appearance.

Each of these depends on the layer beneath it, and errors compound. A slightly wrong calibration produces slightly wrong positions, which produce misattributed shots, which produce a tactical pattern that does not exist. Compounding error is the defining characteristic of this kind of pipeline, and it is the reason evaluation has to be built at every layer rather than only at the end.

Layer four: meaning

Only at this point does anything resemble insight.

You have rallies made of attributed, classified shots, with both players' positions in real-world coordinates throughout. Now you can ask the questions a coach asks. Where does this player concede the T? What do they play from deep on the backhand under time pressure? Which patterns precede their errors?

This is also where language models earn their place — not in the vision stack, where they contribute nothing useful, but in turning a structured description of a match into something a player can read and act on. The division of labour I have found holds up: vision and geometry produce the facts; the language layer explains them. Letting a language model anywhere near the extraction of facts from pixels is a way of generating confident, fluent nonsense.

Evaluating any of this

Ground truth in sports video is expensive. Someone has to sit down and label a match: every shot, every type, every attribution. It is slow, and it is the only thing that tells you whether the system works.

Two failure modes are worth naming, because both look fine in the aggregate:

Metrics that hide their errors. A shot classifier at high overall accuracy can be systematically wrong about one shot type that happens to be tactically important. Aggregate accuracy will not tell you. Per-class evaluation will.

Plausibility masquerading as correctness. The most dangerous output of a sports analytics system is an insight that reads well and is untrue. A human reviewing it has no way to tell — that is precisely why they are using the tool. Which means the system has to be right by construction, verified against labelled footage, rather than trusted because its output sounds like something a coach would say.

What I would tell someone starting

Build the evaluation before the model. Every time I have skipped this I have paid for it later, and paid more.

Respect the layer beneath you. Most "the model is bad" problems in this stack turn out to be calibration problems or attribution problems one level down.

Domain knowledge is not decoration. Knowing what a boast is, knowing why recovery position matters more than recovery speed, knowing that what happens at 9–9 is different — that knowledge determines what you measure. I spent a decade on court before I wrote any of this code, and it is the single largest advantage I have in building it.