Ground-Truth: Seeing What's There and Not What Isn't
The ultimate purpose of our 3D sonar products is very simple. We want to let users see dangers hidden under the water's surface. With our forward looking navigation sonars, this means seeing the obstacles that ship operators want to avoid. This objective seems straight forward. However, the marine world is a complicated environment with lots of objects that can reflect sonar signals. Sometimes localized environmental conditions can even prevent sonar signals from propogating effectively. We want to make sure that the output from our sonars reflect (pun intended) the reality of what is under the water. This means that somehow we need to get an idea of what is actually there to determine if the outputs of our sonars are correct. This information is called "Ground Truth" and is the bain (or bounty) of every scientist and engineer who works on sensing technologies. It is impractical to cover all the procedures and techniques of effectively collecting ground truth data for a marine field-test in a blog posting. However, we can highlight a few interesting aspects of this never ending quest that makes working at FarSounder so interesting.
Qualitative Visual Comparison
Breakwaters seen by FarSounder sonar and radar
As there is no definitive method for high accuracy ground truth data of a large marine environment, let's start with the simplest case: qualitative visual comparison. Qualitative visual comparison is using easy to access, visual methods to see if the sonar is detecting things that are expected. Though an obvious response might be, "If we could see under the water, then we wouldn't need a sonar to show us what's there," there are some targets that extend above the sea surface (where they can be seen) that are known to extend below the surface (where they can't be seen). Such examples include bridge pilings, piers, and breakwaters. Other sensors such as laser range finders and radars can be used to measure distances to the portion of the target extending above the sea surface.
This technique has obvious limitations, as the underwater location of a target may not correspond exactly to the the above water location (think sloping breakwater), and many navigation hazards never break the surface. However, if a sonar can't handle the simple cases, it probably can't handle the complicated cases.
Using a Calibrated Target
Spherical corner reflector about to be calibrated
One technique FarSounder uses frequently is testing sonar performance with a calibrated target. Over the years, we've built a variety of reflectors used in various field tests. The most effective target we've used that is easy to build is a spherical corner reflector. To aid in quantitative measurements, we first calibrate the target in a large acoustic test tank. This allows us to characterize the reflector's target strength as a function of aspect angle. We have a nice target which is about 18 inches in diameter and has an average target strength of slightly less than that of a scuba diver.
During tests, we deploy the negatively buoyant corner reflector by suspending it from a surface float. On the surface float we mount a small hand held gps in a water-tight housing. The surface float is then towed by a small boat with 10-20 meters of line. This method allows us to easily change the depth of the target by adjusting the length of line between the reflector and the surface float. The gps allows us to download the location track of the target after the test and compare it with what the sonar detects.
Less Control: Real World Targets
FarSounder's mobile lab with radar
Obviously, all types of targets cannot be simulated using a towed corner reflector. There are many real world targets that are of interest to various users that are outside of our control. They include marine life and (to a lesser extent) divers. As part of a test near Cape Canaveral, we used our sonar to test the viability of detecting manatees. During this test, we mounted our sonar on a small pontoon boat which we moored near the Port Canaveral lock gates and hoped that some manatees would cross our path. In this case, ground truth gathering was nearly impossible. A trained marine mammal observer was used to help us visually detect manatees. Detailed notes of visual observations were used to help us determine where and when manatees were within our field of view. This method is clearly tedious, and many hours of waiting resulted in just a few manatees that were observed to cross our field of view without any interference from the heavy boat traffic.
Fish are also quite difficult to control or predict. During some of our pier based testing we've seen many moving targets which we believe to be schools of bait fish. Sometimes we've been able to see the water "boiling" as the bait fish attempt to flee from predators. Often, flocks of seagulls follow the boiling water. If lined up just right, we can see the seagulls with our mobile lab's radar system. Occasionally, these flocks/schools pass through our sonar's field of view, and we have a great set of ground truth data. Based on our knowledge of the local fisheries, we can even take some guesses as to what types of fish we're detecting.
Divers Have a Mind of Their Own
It is quite obvious that testing with uncooperative real world targets can be quite difficult. It is so difficult, in fact, that some scientists have resorted to making measurements with frozen or previously frozen fish in order to have a "more cooperative" target. However, even working with cooperative targets can be quite difficult. For example, think about testing diver detection sonar algorithms with real divers in a real world port environment.
Diver entering water during winter test
Often real world environments are not convenient for diver testing. Underwater visibility can be quite poor, currents can be strong, and swim distances involved are quite long. Over the years, we've performed a number of field tests both off of boats as well as off of piers using real divers. In these cases, we had to deal with all of the afore mentioned issues.
One of the simplest ways to have the divers navigate underwater is using a diver held, underwater compass to dead reckon. This works well at times when there are low currents and a little bit of visibility. By watching the sea floor, the diver can estimate any cross track drifting and compensate. Simple visual queues like watching the direction of the sand ripples can be very effective in this situation. However, when currents are present and underwater visibility is less than 1 meter, it is very difficult to keep a diver on track. Therefore, we sometimes add a wreck reel to the diver's tools when dead reckoning.
Diver's GPS track overlay on sonar sector display
In these cases, the diver ties the wreck reel to an anchored object (like a piling). Then, while swimming along a compass heading, the diver keeps the wreck reel line taught. By watching the angle of the wreck reel line and the compass heading, the diver can estimate if their course is aligned with their heading and compensate if necessary. This method works reasonably well for swims that depart from the sonar, travel up to 200m in distance (using two wreck wheels), and then return to the sonar. However, keep in mind that the diver must swim out before returning to the sonar. Therefore, the swim distance is actually twice the maximum diver range. The divers must be very fit and efficient swimmers to do more than a couple of runs like this in strong currents.
For longer swims, we've tried simply dropping a diver from a support boat and free swimming towards the sonar. This works sometimes but only in low current conditions. Therefore, we've also tried deploying an underwater guide line to act as a transect for a free swimming diver to follow. If the line is able to be deployed, this works very well. However, deploying the line can be very time intensive. It is also important to retreive the line after the day of testing as a passing fisherman may easily hook the line and suddenly be the proud new owner of hundreds of feet of nice line (Editor's note: This generally only occurs the night before an important test).
In all of the above cases, the diver always carries (tows) a dive flag with a gps attached to the float. This allows us to record the approximate path of the diver for comparison with our sonar recordings. We are now starting to experiment with a diver held gps using a floating surface antenna. This would not only record the diver's location but provide the diver with a navigation display allowing the diver to follow waypoints. Navy divers handle these complex navigation problems with a variety of very expensive tools including diver held USBL systems and diver navigation consoles with integrated bottom tracking speed logs.
Impact of Local Environmental Conditions
Though the basic specifications of nearly all sonars generalize system performance for a variety of cases, a thorough specification for a sonar system should include specified performance for a given environment. Environmental characterization should include: bathymetry, sea floor sediment composition, and sound speed profile. During field tests, we try to quantify these environment characterizations by measuing bathymetry (or using the best available historical bathymetric data), learning about general sediment characteristics for the area, and measuring the sound speed profile using a CTD. With a good estimation of these characteristics, we can use DAMUS (our Sonar Performance Prediction tool) to compare the actual sonar performance to the predicted performance.
As we've developed new systems and advanced technologies, we've had to also come up with equally advanced ways to put them to the test. Real-world problems require real-world testing, but we continue to do our best to ground truth these results so that we can provide accurate performance measurements and the customer with the best real-time information.
