Player Tracking vs Ball Tracking in Sports Computer Vision

If you are building computer vision for a racket sport, you need two things out of the video: where the players are, and where the ball is. It is tempting to treat these as one problem with two targets — point a detector at the frame, track what comes back.

That approach produces a system that tracks players adequately and loses the ball constantly. The two problems have almost nothing in common beyond both involving a camera.

Player tracking is an identity problem

Detecting that there is a person in a frame is close to solved. Modern detectors do it reliably at high frame rates on ordinary footage.

The difficulty is not detection. It is keeping the same person labelled as the same person over time.

In squash, two players share a court about six and a half metres wide. They occlude each other constantly. That is not an edge case — the entire sport is built around one player occupying the space the other wants. A tracker that assumes targets stay visually separable is wrong roughly every rally.

The specific failure is identity switching: player A disappears behind player B for eight frames, and when the detections resume the tracker assigns the labels the other way round. Every downstream conclusion from that moment on is attributed to the wrong athlete, and nothing about the output looks broken. The positions are plausible. The shots are plausible. They are simply assigned to the wrong person.

What helps:

• Motion continuity. People do not teleport. A track that requires a two-metre jump between consecutive frames is probably a switch.
• Appearance over time, held loosely — kit colour helps until both players wear white.
• Court-side priors. In squash, sustained knowledge of who was where, combined with the physical constraint that both players cannot occupy the same position, resolves a lot of ambiguity that frame-local reasoning cannot.
• Correcting backwards. Identity is easier to resolve in hindsight. Once the occlusion ends and both players are clearly separated again, you can often work out which assignment was consistent and repair the intervening frames. Offline analysis has this luxury; live systems do not.

The evaluation metric that matters here is not detection accuracy. It is identity switches per match. A tracker at 99% detection accuracy with twelve identity switches is worse, for analysis purposes, than one at 95% with none.

Ball tracking is a signal problem

The ball is a different beast entirely.

It is small — often a handful of pixels. It is fast enough to motion-blur into a streak, so in many frames there is no crisp object to detect at all. It is dark against dark, or lost against the white front wall depending on exposure. It disappears behind players. In footage shot through a glass back wall, it competes with reflections and a crowd.

Run a general-purpose detector on this and you get two problems at once: it misses the ball frequently, and it produces false positives on anything small and roughly round.

The thing that rescues ball tracking is a property players do not have: the ball obeys physics. Its trajectory is constrained. It travels in predictable arcs, changes direction only on contact with a wall, the floor, or a racket, and cannot accelerate spontaneously.

That constraint is the actual signal. It means you should not think about "detecting the ball in frame N" but about finding the trajectory that best explains a whole sequence of frames. Weak detections that fall on a physically coherent path are probably the ball. Strong detections that would require the ball to reverse mid-air are probably a shoelace.

Practically, that means:

• Reasoning over a temporal window rather than frame by frame.
• Tolerating gaps — you can interpolate through frames where the ball genuinely is not visible, provided the path either side is consistent.
• Using trajectory discontinuities as signal, not noise: a direction change is a contact, and contacts are exactly the events you care about.

That last point matters more than it sounds. For player tracking, you want to smooth away discontinuities. For ball tracking, the discontinuities are the data — every shot in the match is a kink in that trajectory.

Why the distinction has consequences

They fail differently, so they need different evaluation. Players: identity switches. Ball: trajectory continuity and contact detection, not per-frame recall. A ball tracker with 70% per-frame recall but perfect contact detection is excellent. One with 90% recall that misses contacts is not.

They need different temporal windows. Player tracking is largely causal and local. Ball tracking benefits enormously from looking forward as well as back.

Their errors compound differently. A player identity switch corrupts everything after it until something resets. A missed ball detection corrupts one shot. Both matter, but the first is the one that quietly destroys a match report.

One depends on the other. Attributing a shot to a player requires knowing both where the ball was contacted and who was there. So ball-tracking error and player-identity error multiply at exactly the point where the analysis becomes useful.

The layer that makes both usable

Neither tracker produces anything meaningful in pixel coordinates. "The ball was at (840, 512)" depends entirely on where the camera was standing.

Court calibration — solving for the camera using the court's known, standardised geometry — is what turns both outputs into real-world positions that mean the same thing across venues and camera angles. It is the least glamorous component in the stack and the highest-leverage one. Without it you have a system that works on your test footage and nowhere else.

Only once both trackers feed a calibrated coordinate space can you start asking the questions that actually matter about a match — and eventually build a persistent model of how a player competes.

More on the full stack in teaching AI to understand a squash match, and on the systems side in how I approach building these.