Researchers track whole body movement with just a smartwatch | Cornell Chronicle

Open original

Cornell researchers built EchoMotion, a system that uses a standard smartwatch's speaker and microphone to send out inaudible sound waves and read echoes off the body, estimating full-body pose without cameras. It is aimed at fitness, rehab, and eventually AI assistants that log what you do during the day.

  • Using ultrasonic echoes instead of cameras protects privacy and produces less data to process.
  • After a 12-minute calibration, average error for predicting each of 12 joint positions dropped to about 2.4 inches on an unmodified Galaxy Watch 6.
  • Power draw is around 12 milliwatts, and a 30-minute session used only 5% to 6% of the watch battery.

Cornell's SciFi Lab built EchoMotion, a system that tracks a person's full-body pose using only a smartwatch's built-in speaker and microphone. The watch emits ultrasonic sound that fills a sphere around the wearer, and the echoes bouncing off limbs are read by a small deep learning model that maps them to positions of 12 joints, from head to ankles. The key claim is that this works without cameras, which matters because existing whole-body tracking usually needs extra hardware like phones, earbuds, or VR headsets, or else camera setups that raise privacy concerns. Acoustic sensing sidesteps both problems and produces far less data to process. The numbers are the most concrete part of the story. In the prototype, sensing drew about 12 milliwatts, roughly 40 times less than a wearable camera, and a 30-minute session on a Galaxy Watch 6 consumed only 5 to 6 percent of the battery. Accuracy was measured against camera-captured poses. The team first trained on 10 volunteers doing 18 exercises using a 3D-printed watch with two speaker-microphone pairs. Then they moved to unmodified commercial watches: with 15 new users, average joint-location error was under four inches, and a single 12-minute calibration session cut that to about 2.4 inches. The same model, without retraining, also ran on a Galaxy Watch 7 and a Google Pixel Watch 3. In real-world tests, furniture and background noise didn't hurt performance, but a narrow hallway caused slight degradation because echoes from other people interfered. The authors are candid that the work is early. The sample sizes are small, dozens of users rather than thousands, and crowded settings like a subway remain untested; the team expects more data to help. The near-term application they're building is fitness coaching, where an agent would count reps and correct form on something like a Bulgarian split squat, and they are also pairing the approach with smartglasses to track handled objects. The broader pitch, from co-author Saif Mahmud, is that this supplies the body-tracking layer for future wearables that log a user's day automatically. That's a plausible direction, but the paper itself only demonstrates pose estimation in controlled conditions, and the gap between that and a shipping product is not small.

Ask about this article