MIT CSAIL's new vision system helps robots understand their bodies - The Robot Report

MIT CSAIL has developed a novel robotic control system called Neural Jacobian Fields (NJF) that enables robots to understand and control their own bodies using only visual data from a single camera, without relying on embedded sensors or pre-designed models. This approach allows robots to learn their own internal models by observing the effects of random movements, providing them with a form of bodily self-awareness. The system was successfully tested on diverse robotic platforms, including a soft pneumatic hand, a rigid Allegro hand, a 3D-printed arm, and a sensorless rotating platform, demonstrating its robustness across different morphologies. The key innovation of NJF lies in decoupling robot control from hardware constraints, thus enabling more flexible, affordable, and unconventional robot designs without the need for complex sensor arrays or reinforced structures. By leveraging a neural network that combines 3D geometry reconstruction with a Jacobian field predicting how robot parts move in response to commands, NJF builds on neural radiance fields (NeRF) to