@daphnesolves - Training Electric Fish RL
@daphnesolves on X A developer ports the weakly electric fish multi-agent RL simulation from Python to C, using PufferLib, and gets training to about one million steps per second. The result is a strong fish policy trained in roughly 10 seconds, against 6 to 9 hours in the original setup.
- The C port started at about 40,000 steps per second, and switching most doubles to floats added roughly 30% more speed.
- Profiling showed electric-field calculations dominated runtime, so restricting sensor calculations to elements within each organ's short sensory range cut the work sharply.
- Final throughput reached 1.85 million simulation steps per second, against about 4,300 for the reference Python environment, and a 527-run sweep finished in 12 hours.
The author, who has spent three years on self-play RL for autonomous driving, uses a paper on multi-agent RL for weakly electric fish as a jumping-off point. Their argument is that fast simulation, a clear reward, and an accurate enough model of the environment are what make RL work, and biology is a harder place to apply it because the underlying biophysics is expensive to compute and experimental data is slow to collect. The original paper, by Satpreet, Sonja and co-authors, trained fish policies that reproduced group behaviors like foraging and biting, none of which were scripted. Those behaviors emerged from assumptions about biophysics, the training setup, and the reward function. The original Python codebase needed five million steps per run and six to nine hours of wall-clock time, which is what motivated the speedup effort.
The engineering work, done with Joseph, is the meat of the piece. They ported the environment and PPO loop from Python to C with the help of a code agent, checked the translation against the original, and got an initial speed of about 40,000 steps per second. Two changes did most of the work. First, switching most variables from double to float precision (and using sinf instead of sin) cut memory traffic and gave a 30 percent speedup without materially changing results. Second, profiling showed nearly all time went to measuring the electric field, since each of the roughly four fish per arena carries 72 sensory organs that each sum contributions from every emitter, plus arena walls that reflect charges. The insight was that Mormyromasts and ampullary receptors only matter within about 4 to 10 cm and knollenorgans within about 100 cm, so the code now only computes contributions from elements inside each organ's range. Thresholds are configurable.
Combined with smaller optimizations, the result was 1.85 million simulation steps per second and about 966,000 training steps per second on one machine with an RTX 4090 and 16 CPU threads, against roughly 4,300 and 740 for the reference Python. That brings a strong fish policy into roughly 10 seconds of training. The core simulation is about 1,200 lines of C and lives in the PufferLib 5.0 branch under ocean/wef. The experiments section then covers the setup they kept from the paper, with randomized arena sizes between 40 and 300 cm, uniform or patchy food, and per-fish body size, plus reward terms normalized to [-1, 1] and cooldowns after eating and biting.