MIT vision system teaches robots to understand their bodies

MIT researchers at CSAIL have developed a novel robotic control system called Neural Jacobian Fields (NJF) that enables robots to learn how their bodies move in response to motor commands purely through visual observation, without relying on embedded sensors or hand-coded models. Using a single camera and random exploratory movements, NJF allows robots—ranging from soft robotic hands to rigid arms and rotating platforms—to autonomously build an internal model of their 3D geometry and control sensitivities. This approach mimics how humans learn to control their limbs by observing and adapting to their own movements, shifting robotics from traditional programming toward teaching robots through experience. NJF’s key innovation lies in decoupling robot control from hardware constraints, enabling designers to create soft, deformable, or irregularly shaped robots without embedding sensors or modifying structures for easier modeling. By leveraging a neural network inspired by neural radiance fields (NeRF), NJF reconstructs the robot’s shape and its response to control inputs solely from visual data. This