For remotely operated underwater vehicles, cloudy and turbulent waters often pose significant challenges. When vehicles settle on the seafloor or dig through sandbeds, they can stir up clouds of sediment, making it difficult for onboard cameras to see through. Often, the only option is to wait until the marine dust settles before proceeding safely. However, a new underwater mapping technique developed by engineers at MIT and the Woods Hole Oceanographic Institution (WHOI) may allow vehicles to see through murky, low-visibility waters. This method fuses visual images from optical cameras with acoustic data from sonar sensors, enabling a vehicle to quickly map the general shape of its surroundings using sonar, even in low-visibility waters. The vehicle can then approach specific shapes in the sonar-mapped environment, coming close enough for optical cameras to resolve detailed objects visually. This technique is akin to pairing a dolphin’s echolocation with a sea turtle’s close-range vision to navigate through murky water in real-time. The researchers tested the method in controlled tank experiments, manipulating the water's visibility. Even under the cloudiest conditions, the system was able to see through the sediment to map the tank's environment and visualize centimeter-scale details of objects. The team is further enhancing the technique, named Sonar-MASt3R, and envisions that this mapping method could safely guide underwater vehicles through murky environments for various applications, including scientific exploration, underwater construction, and deep-sea recovery. “We hope that this work enables us to conduct more operations in those challenging, low-visibility environments and helps provide more coverage in areas that are difficult to operate in today,” says Amy Phung, a graduate student in MIT’s Department of Aeronautics and Astronautics, who led the work.
To see underwater, scientists have typically taken an either/or approach, using either optical cameras or sonar sensors for guidance. Optical cameras provide detailed imagery of a scene, but only in relatively clear and well-lit waters. In contrast, sonar sensors perform well in both clear and murky water; they emit acoustic waves and measure the time and angle at which they return, allowing them to determine the exact shape, distance, and depth of objects in the environment, although sonar maps lack visual detail. To leverage the strengths of both modes, scientists have sought to combine the two in a new approach known as “opti-acoustic fusion.” In prior works, research groups have merged sonar and optical data in mapping techniques mostly geared toward object recognition and reconstructing workplace environments. Most techniques require time to sync and process data, thus do not work in real-time, while only a few can create 3D maps of environments. None have been applied to high-resolution mapping underwater in murky conditions.
Phung, a student in the MIT-WHOI Joint Program, and her advisor Camilli aimed to develop an opti-acoustic fusion technique that would generate detailed 3D maps of underwater environments in real-time and low-visibility conditions. The team was motivated, in part, by challenges in safely recovering unexploded underwater mines. “There can be old explosives in areas that make it unsafe for ships to be in, and the ability to safely dispose of those is best done by robotics,” Camilli says. “But many of these explosives are set in surf zone environments where visibility adds to the challenge of doing this safely. That’s one of many applications our technique can be used for.”
The new method, Sonar-MASt3R, builds on an existing technique, MASt3R, developed by researchers in France. MASt3R is an image matching algorithm trained to take in visual images of the same scene and quickly estimate the relative depth of each pixel. This allows MASt3R to generate a 3D map of the environment in real-time based on a camera’s 2D images. “The downside is that there is no sense of scale,” Phung says. “It will say ‘this pixel is five units closer than this pixel,’ but it can’t say whether that’s 5 meters or 5 feet.” Fortunately, sonar provides absolute measurements of scale. The timing of sonar reflections can be directly translated into specific depths and distances of objects that the signals bounced off, as well as their shape and contour. In their new work, Phung and Camilli used sonar data to correct MASt3R’s scaling and generate precise 3D maps of underwater environments. Even in murky water, the sonar-corrected map enables a vehicle to know the precise location of objects, allowing it to safely move in for closer inspection, which can then be done using conventional optical cameras.
The team tested Sonar-MASt3R in experiments with a tank filled with water, sediment, and various objects such as a small boulder, a coffee mug, and a packing crate. Inside the tank, they set up a robotic arm, onto which they mounted an underwater camera and a sonar sensor. For each experimental run, they first conducted a sweep trajectory, where the robotic arm slowly swept from one side of the tank to the other to capture sonar and visual data. With this initial sweep, Sonar-MASt3R quickly creates a coarse sonar-based map of the shapes and contours of the tank and its objects. The coarse map is then used to record close-up camera images of the objects, which are used to improve the map resolution. A “keyframe” approach quickly compares each new image frame to the last keyframe. If a frame provides new information not contained in the last keyframe, the image is added as a new keyframe to the map. If it is similar, it is immediately discarded. This way, the approach can quickly fill in the map with relevant visual detail in real-time. The researchers tested their new approach underwater, examining eight different levels of turbidity created by stirring the tank’s sediment. Compared with other opti-acoustic fusion approaches, Sonar-MASt3R generated more accurate 3D maps and resolved smaller, centimeter-scale details, even in cloudier conditions. In the cloudiest condition, where the robotic arm’s cameras could not see through, its sonar sensors were able to generate a rough map of the tank’s hidden objects. This initial map enabled the arm to move safely through the murk and closer to specific objects, which its underwater camera could then visualize in more detail. “An analogy would be if you were to go into a china shop in the dark, and try to pick your way around to find a specific coffee mug without knocking things over,” Camilli offers. “This would allow you to do that.” The team plans to test the approach in natural underwater conditions, where they suspect the mapping task should be more straightforward. “In a tank, it’s like an echo chamber,” Camilli says. “It’s like trying to do this in a funhouse mirror setting where you get all these distortions and reverberations and ghost images that really complicate the processing. If you put it in the real world, it should be easier.” They believe Sonar-MASt3R could help scientists safely explore murky underwater regions. “The real value in this effort is so we can use this technology in mission scenarios that are untractable right now,” Phung says. “And there are plenty of untractable missions because we don’t have the observational or perception capabilities.” This research was partially supported by NASA and the National Science Foundation.
Blogger's Review: The technological breakthrough of Sonar-MASt3R significantly enhances underwater detection and operational capabilities, especially in low-visibility environments. By effectively combining sonar and optical data, this system demonstrates the immense potential of opti-acoustic fusion, paving the way for future underwater scientific explorations and practical applications. Looking forward to its performance in real-world scenarios!