Qimera 2.2.0 - New Functionality

Introduction

In this release, we have focused on the final phases of switching over fully to Grid 5.  This process was initiated with version 2.0 and in version 2.2 we have collected all feedback about performance, stability, etc and have addressed these items.  Secondarily, we have added some very interesting new filters and also a new add-on that allows for height-matching between different survey lines when there is poor vertical control.  Read below for details.

New Filters

Reject Outliers (1D)

We have two new point cloud filters that reject statistical outliers from the point cloud, they both share the general name "Reject Outliers".  The first, which we name "Reject Outliers (1D), is a variant that focuses on general seafloor topography, which covers most applications.  We specifically highlight this as a one-dimension (1D) filter since it only examines point clouds in the cell in the vertical direction only.  In this single vertical direction, there is also only one average value that can be calculated.  It is a great filter for flat or gently sloping seafloors but does not handle strong variations in topography or complex objects or structures.  For this we have the "True 3D" variant, which is discussed below.  But first, let's discuss this filter.

How does it work?  The general theory is to establish a virtual grid cell size and then calculate the average of that virtual grid cell and then reject points that are outside of the specified distance from the average.  The example below shows this general type of configuration.  This particular filter works very well to remove noise that is normally distributed around a stable average surface.

The true power in this type of filter is to create a sequence of such filters, which is possible in the dialog to manage filter profiles, as shown below.  The first iteration uses a very large cell size and a large threshold; this removes extreme outliers.  Once this is done, a second iteration then repeats the operation for a smaller cell size and smaller threshold.  The sequencing is necessary in some cases when extreme outliers can bias the average when using a small cell size.

How is it that outliers can affect the filter performance?  If the threshold is set too small, then outliers will cause the filter to remove some of the good soundings because the average is biased towards outliers.  An example is shown below where a 1.5 m cell size shows the effect of a single outlier on the average calculation (on the left).  Compare this to the image on the right where a 3 m cell size is used.  In this case, the cell has more good points to counteract the effect of the single outlier and the average is affected less.  In this case, the calculated average is safer to use to remove outliers both large and small.

Point cloud with 1.5 m cell size. The single outlier has great effect on the average.	Point cloud with 3.0 m cell size. The single outlier has less effect on the average.

How does this work in practice?  Here is a sequence of images showing the effects of a single pass of this new filter and also the result of a stacked iterative sequence of the filter.  The grids are color coded by uncertainty and the cast shadows highlight the presence of many spikes as well as areas of general high uncertainty in the overlap between survey lines.

Raw Data	Reject Outliers (1D) - Single Pass	Reject Outliers (True 1D) - Three passes

Reject Outliers (True 3D)

This filter is the second that we have integrated from the open source PCL library, the first was the Cluster filter which was released with Qimera version 2.0.  This filter is different from the 1D version because it can handle true 3D objects, like quay walls, pilings, and objects with overhangs.  It also elegantly handles rough topography and strong variations in topography.  It is much more powerful, but at the expense of additional computation time.  An example configuration is shown below.  The primary parameter that the user should specify is the distance between points that are to be considered before marking a point an outlier relative to its neighbors in the 3D point cloud in which it belongs.

Why do we name this filter "True 3D"?  We do so because it handles the case of true 3D objects like you would get from laser scanning of buildings or port facilities, or mapping of seabed engineering structures where vertical objects and overhangs are the norm.  All other filters available in Qimera assume a seafloor or surface for which there is only one vertical value in any given cell on the surface.  There are many mapping applications where this is not true though, and this filter is designed to work for these kinds of situations.  The examples below show data from an R2Sonic Sonic 2024 operating in 700 kHz UHD mode over a submerged tank in an artificial reef site.  Data courtesy of R2Sonic.

Raw Point Cloud	Filtered with Reject Outliers (1D) Outliers are removed but so are portions of 3D structure.	Filtered with Reject Outliers (True 3D) Outliers are removed and 3D structure is preserved.

Filter by Angle

We have continued on our path of making every Blocking Filter available as a Filter in the Filter toolbar.  In this release, we introduce the ability to Filter by Angle.  To refresh your memory, Blocking Filters can be applied online in Qinsy or offline in Qimera, they can apply to the entire line and can be easily enabled/disabled.  Being able to apply the same type of filters from the Filter Toolbar gives the additional power to select particular spatial areas to apply the filters, for example an aggressive angular filter in an area of heavy overlap can be applied by drawing a lasso selection around the area and applying the filter.

This filter works for all raw sonar formats since Qimera is able to access the beam angles for each points with raw sonar file formats.  On the other hand, not all processed point file formats that Qimera supports preserve the beam angle.  At the moment, the only such point file format that we have built support for are .fau files.

Lonely Cells Filter

We have introduced a new Lonely Cells filter, it removes isolated small clumps of grid cells that don’t connect to the main surface, which usually happens at the edges of surveys. This operation, though it appears to operate on the Dynamic Surface, is actually only using the grid to identify the lonely cells.  Once they are identified, the underlying points are flagged and the matching subset of the grid is recomputed.  Thus, it is ultimately the points that are flagged and this prevents them from re-appearing in any new grids or in any subsequent regridding operations.  The points are flagged with the Filter flag, this allows you to undo this filter via the Revert All Filtering option in the Filter Toolbar.  The only parameter the user must choose when running this operation is the maximum size of a block of lonely cells cannot exceed in order to be filtered.

An example is shown below. You’ll find this new filter in the Dynamic Surface menu, it currently operates on the entire surface.  The typical use case is as a first pass operation to remove the lonely cells prior to further filtering or point cleaning operations.  If the filter removes too much data at the edge, it is simple to use the lasso selection mode and the Revert All Filtering option from the filter toolbar to reclaim data in these situations.

Raw gridded data.  Note the lonely cells on the left side of the grid.	Filtered gridded data, with lonely cells removed.

Automatic Height Matching

In this release Qimera now offers a new add-on that allows users to fix vertical offsets between neighboring survey lines using a semi-automated tool.  If you are interested to try this new add-on, please contact sales@qps.nl to arrange for a demo or an evaluation.  At the moment, this works only for raw sonar data formats.  We have plans to add the same functionality for processed point formats such that Qimera Clean users will be able to take advantage of this as well.  This is the same strategy we took when releasing the TU Delft Sound Speed Inversion add-on in 2018.

The tool uses the same principle as the Slice Editor in that the user chooses a rectangular area in which to solve for vertical offsets.  The point cloud is displayed in order to guarantee that there is a "human in the loop" to override any problematic solutions.  As with all plotting tools in Qimera, you can easily save screen captures of the plot for before/after comparisons and reporting purposes.

Point cloud before solving for vertical offsets.	Point cloud after solving for vertical offsets.

Along with the point cloud viewer, the user has a control dock from which they operate the tool.  The control dock lists the files loaded in the point cloud viewer and it provides the ability to lock one or more particular files to be used as a reference.  Once a reference is selected, the user clicks the auto-solve icon, which is used throughout Qimera, to solve the offsets.  The point cloud display then updates for visual confirmation to the user.  The offsets are displayed in the control dock after the algorithm is run, these can also be manually adjusted to make small tweaks in the event that the automatic matching did not work.  If you are happy with the results, you can click the Save icon and carry on with further processing, or you can move the rectangular selection to another area that requires further intervention.  In this second case, the lines for which a vertical offset is already determined will become locked as well and thus used as a reference in the remaining areas. 

This type of tool really requires good accounting on what has been applied so that you and the recipients of your data can understand exactly what has been done to the data to counteract problems.  To this end, we have followed the Qimera philosophy of allowing the user to easily enable and disable these offsets after the tool is run.  If you visit the Processing Settings after you run the tool, you will see there is a tickbox control to easily allow you to revert to a raw state, this can be done for one file or a selection of files.

Additionally in the Processing Setting dialog, you can review the offsets for all files at any time.  You can save the offsets to a text file to include in any reporting.

We purposefully took the semi-automated approach since the hydrographic surveying industry is generally risk averse and suspicious of black box solutions.  These simple and semi-automated workflows allow users to see the effect of the tool for themselves in order to build trust in the tool.  Combined with the good processing state management and easy reporting capability, this helps build trust with users of this tool and it also helps build trust with the recipients of the data that was processed with this tool.

Command line scripting

We have started down a roadmap to allow for command line scripting in Qimera.  We have focused on the Kongsberg .all format at first since the QPS .db format is well supported online with Qinsy and users can already create QPDs and Dynamic Surfaces while online.  With the power of Qinsy's online capabilities, there is less of a need for offline command line scripting at the moment.

Users will find an additional .exe file in the Qimera installation directory named qimera-command.exe.  Running this .exe file with the --help option will give the user guidance on how to setup a script for processing.  Scripts can be created and run in any user infrastructure, there is no need for interaction with a processing or scheduling server, you can run this natively on your own machine with only a little bit of scripting experience.  This was specifically built to allow incorporation into the NOAA Charlene package, which allows for automated pre-processing and processing for NOAA workflows.

At the moment, only basic functionality is supported: project creation, import of .all files, convert to QPD, creation of a new Dynamic Surface, append to an existing Dynamic Surface and application of a filter profile from the Filter Toolbar.  We will add more functionality in the near future, as well as additional file formats.