Welcome to AGT’s documentation!¶
Introduction¶
This plugin is a toolbox for processing electrical resistivity (Geoscan Research RM15/RM85), magnetic (Sensys MXPDA / Bartington Grad601) and electromagnetic induction (Geonics EM31, GEM2 de Geophex, EMP400 de GSSI) data with shapefiles creation.
Electrical data processing module (Geoscan Research RM15/RM85) - grid survey¶
This module enables basic processing of resistivity data collected with a regular grid. For the moment, only the Pole-Pole configuration is available.
Input file
The input file is an ascii file (.dat) that contains the data and header that gives all the information concerning the configuration of the resistance meter and the survey mode:
Column file containing the data
RM85 Name of the resistance meter
2 Number of grids
30 Grid length (m)
30 Grid width (m)
0.5 Probe spacing (m)
3 Number of channels
4 Number of probes
1 Line step (m)
Pole-Pole Probe configuration
10 Current intensity
1 Number of the first grid
0 Coordinates of grid’s first point (bottom-left point)
0
2 Number of the second grid
0 coordinates of grid’s first point (bottom-left point)
30
20.95 Resistivity measurement
25
20.5
8.55
8.9
5.26
.
.
.
The value of dummy log is 999.
Header has two different roles, so must be accurately completed because it is used:
for data processing (automatically read by the software)
for storing metadata of the survey
Processing
The basic processing consists of reorganizing the raw data in order to separate each channel and assign the right X,Y position of each measurement.. The measured resistance values are transformed into apparent resistivity, depending on the chosen probe configuration (for the moment only the pole-pole configuration is available). Data can be exported in shapefiles (.shp) or ascii files (.dat). Two sub-modules are subsequently proposed:
Median filtering
This function allows noise removal using a median filter algorithm. The user has to define the kernel size and the threshold defined as a percentage tolerance with the median value.
Georeferencing
This function allows grid georeferencing using geometric translations and rotations. The method uses two control points provided by the user.
Magnetic data processing module (Sensys MXPDA / Bartington Grad601) - grid survey¶
This module allows basic processing of magnetic survey data, collected with a Sensys MXPDA or Bartington Grad601 magnetometer.
Input file
The input file is an .dat file, generated by Sensys Magneto-Arch software or a .dat file generated by Bartington software. File configurations are as follows:
Sensys MXPDA
X Y Value
0 0 2.8
0 0.1 2.87
0 0.2 3.08
0 0.3 2.59
0 0.4 1.89
.
.
.
Bartington Grad601
Time = 09:16:55
Date = 06/10/2017
Grid Number = 1
Number of Sensors = 2
Grid Size = 40 x 40
Method of collection = ZigZag
Starting Direction = West
Data Range = 100 nT
Line Spacing = 1.00 m
Sampling = 4 samples / m
Sensor Spacing = 1.0 m
Mean = 0.50
Max = 21.40
Min = -4.01
0.125 0.5 0.87 Y, X, value
0.375 0.5 0.86
0.625 0.5 1.21
0.875 0.5 1.78
2.875 0.5 1.16
.
.
.
Processing
The basic processing consists of reorganizing the data in order to separate each profile (a profile is defined with a fixed X coordinate). Data can be exported in shapefiles (.shp) and ascii files (.dat).
Three processing modules are proposed:
Median removal
This function removes the median value of each profile, in order to eliminate the shift due to electronic components of the sensors and constant magnetic disturbances. It is possible to limit the number of points used to determine the median value by using a percentile. Using a percentile threshold excludes strong magnetic anomalies from the median value computing.
Trend removal
This function allows a first-, second- or third-order polynomial removal on each profile.
Georeferencing
This function allows grid georeferencing using geometric translations and rotations. The method uses two control points provided by the user.
Magnetic data processing module (Sensys MXPDA) - GNSS survey¶
This module allows basic processing of magnetic survey data, collected with a Sensys MXPDA magnetometer coupled with a GNSS.
Input file
The input file is an .asc file, generated by Sensys Magneto-Arch software. Its configuration is as follows:
X, Y, difference of the vertical component of the magnetic field, profile name, the number of the probe.
30694328.591 5432511.556 5.5 “20161010-110332_GZP.prm” 1
30694328.717 5432511.772 31.2 “20161010-110332_GZP.prm” 2
30694328.844 5432511.987 -21.6 “20161010-110332_GZP.prm” 3
30694328.971 5432512.203 -8.3 “20161010-110332_GZP.prm” 4
30694329.098 5432512.418 -12.3 “20161010-110332_GZP.prm” 5
…
Collected data are georeferenced in UTM coordinates. The two first digits of the X coordinate correspond to the UTM zone (UTM-30 in this example).
Processing
he basic processing consists of reorganizing the data in order to separate each profile (a profile is defined as a set of data collected with one probe along one line). Points are then georeferenced in the chosen mapping system. Data can be exported in shapefiles (.shp) and ascii files (.dat).
Four processing modules are proposed:
Decimation
This function reduces the number of collected points by keeping a fraction of them (1 / n, n is chosen by the user). The user can use raw data or median filtered data (computed with a n data point moving median).
Median removal
This function removes the median value of each profile, in order to eliminate the shift due to electronic components of the sensors and constant magnetic disturbances. It is possible to limit the number of points used to determine the median value by using a percentile. Using a percentile threshold excludes strong magnetic anomalies from the median value computing.
Trend removal
This function allows a first-, second- or third-order polynomial removal on each profile.
Stationary point removal
This function eliminates stationary data points collected while the magnetometer is immobile.
EMI data processing module (EM31 from Geonics)¶
The module transforms the electrical conductivity values provided by the EM31 (based on McNeil, 1980). This processing overpasses the boundaries of the linear approximation which is only valid in first approximation for an instrument held on the ground (Z=0) and respecting the low induction number condition (i.e. low electrical conductivity). Data is transformed into a .shp file and can be uploaded into GIS.
Input file
Input file is an ascii file format (.dat) as delivered by DAT31W (Geoncis software). It contains the X, and Y position of each measurement, as well as the quadrature (QV1 in mS / m), in-phase part of the electromagnetic signal (IV1 in ppt) and a time stamp:
/ EAST, NORTH, QV1, IV1, TIME /
642039.43420000 7097622.22880000 30.10 1.03 15: 32: 39.555
642039.43560000 7097622.22740000 30.25 1.03 15: 32: 39.904
642039.43548000 7097622.22934000 30.18 1.03 15: 32: 40.262
642039.43478000 7097622.23249000 30.13 1.01 15: 32: 40.614
642039.43402000 7097622.23591000 30.02 1.00 15: 32: 40.991
642039.50925000 7097622.14500000 29.95 1.00 15: 32: 41.353
642039.58235000 7097622.05660000 29.98 1.00 15: 32: 41.699
642039.67784000 7097621.93750000 30.18 1.02 15: 32: 42.071
The user has to specify the coordinate system used during the survey.
Processing
The processing module estimates the apparent electrical conductivity values based on the solution of the integrals and the Hankel transform (Thiesson et al., 2014). This solution takes into account the height of the device and the coils configuration. It can therefore be applied regardless of the type of soil studied even valid in salty soil contexts.
McNeill J.D., 1980 - Electromagnetic terrain conductivity measurement at low induction number, technical note TN6, Geonics Ltd, Toronto, 15p.
Thiesson J., Kessouri P., Schamper C., Tabbagh A., 2014 - Calibration of frequency-domain electromagnetic devices used in near-surface surveying. Near Surface Geophysics, 12, 481-491.
EMI multi-frequency data processing module (GEM2 from Geophex, EMP400 from GSSI)¶
This module enables different processing methods to be performed on the data collected with the GEM2 (Geophex) or the EMP400 (GSSI).
Merging of GNSS and GEM2/EMP400 datasets (with clock offset correction)
Positionning shift correction between the GNSS antenna and the GEM2/EMP400 device
Data decimation
Median filtering by profile
1D moving window filter (median or average method)
Electrical conductivity computation
Magnetic susceptibility computation with correction of the induction effect
Device calibration based on vertical resistivity sounding or mean resistivity
GEM2 file input
The input file is an ascii file (.csv) in export format proposed by the software EMExport of the Geophex series. The module allows three different acquisition protocols (without GNSS, with a GNSS connected to the GEM2 and with a GNSS not connected to the GEM2) and it contains a variable data series based on an acquisition with GNSS or acquisition without GNSS.
GEM2 survey with GNSS
Line,Sample,X,Y,Mark,Status,GPSStat,GPSalt,Time[ms],Time[hhmmss.sss],PowerLn,I_5025Hz,Q_5025Hz, I_10325Hz,Q_10325Hz,I_21275Hz,Q_21275Hz,I_43725Hz,Q_43725Hz,I_89925Hz,Q_89925Hz,QSum
0, 21, 509779.22, 5061220.33, 0,0,4, 575.22, 35967000.100,095927.0001, 0.3,-1.22403e+003, 6.32362e+000,-1.55238e+003,-1.78986e+002,-1.99524e+003, 1.17386e+003,-2.12842e+003, 4.58693e+003, 3.66881e+003, 3.00984e+003, 8.59797e+003
GEM2 survey without GNSS
Line,Sample,X,Y,Mark,Status,Time[ms],Time[hhmmss.sss],PowerLn,I_5025Hz,Q_5025Hz,I_10325Hz, Q_10325Hz,I_21275Hz,Q_21275Hz,I_43725Hz,Q_43725Hz,I_89925Hz,Q_89925Hz,QSum
0,0, 0.00, 0.00, 0, 0, 52275802.200, 143115.8022, 0.2, 1.47162e+003, -3.85928e+002,9.69091e+002,1.29074e+001,4.03300e+002,1.14524e+003,6.69531e+001,5.15901e+003, 6.22617e+003, 5.16867e+003, 1.10741e+004,0,1, 0.00,1.06,0,0, 52275842.200,143115.8422, 0.1, 1.67830e+003, 1.36072e+001, 1.14406e+003,-1.37615e+002, 5.27930e+002, 1.16965e+003,-2.98461e+001, 5.38998e+003, 5.90062e+003, 5.46728e+003, 1.19029e+004
EMP400 file input The input file is an ascii file (.EMI) in export format proposed by the EMP400. The module allows two survey modes (without GNSS and with separate GNSS) and it contains a variable data series based on the survey procedure. Note that the EMP400 has a GNSS but, due to the low precision, the plugin doesn’t consider these data.
EMP400 survey
Header of 36 lines
- Record#, XCoord, YCoord, Time, InPhase[15000], Quad[15000], Conductivity[15000], InPhase[8000], Quad[8000], Conductivity[8000], InPhase[5000], Quad[5000], Conductivity[5000], Remark, Mark, Lat, Long, Alt, Tilt, Errors
31, 0.500, 18.000,07:19:22.385,-21373 ,1129 ,25.706 ,-6046 ,684 ,29.208 ,-2588 ,438 ,29.882 ,,, 45.6537417, 3.1515333,376.0000000,,NO ERRORS
The plugin automatically detects the number of frequencies employed for the data collection and the value of each frequency. Then, it detects whether the acquisition was performed with a GNSS or not. These specifications are then used in the different processing methods applied to the dataset.
The user has to specify the reference coordinate system (RCS) used during the survey. By default it is the WGS84 UTM31 North system. This RCS can be changed in the tab Parameters and saved as the new reference system default.
The input GNSS file is an ascii file (.dat). It must contain basic information, X and Y position, altitude and time of acquisition (hh:mm:ss.ss). Care must be taken to specify the RCS used for the GNSS data acquisition.
Input GNSS file format
/X, Y, Z, Time/
1709059.979, 6946271.346, 232.25,12:11:05
1709059.975, 6946271.352, 232.22,12:11:06
1709009.729, 6946415.921, 232.15,12:42:52
Processing
Merging GNSS and GEM2/EMP400 data This function is dedicated to merging the GEM2/EMP400 input file and the GNSS input file based on the time of acquisition of each data. By default the GEM2/EMP400 input file is a file without GNSS data. Timing is based on the time stamp of both devices. We can specify the offset existing between the two clocks in order to optimize the data positioning. Data decimation
This function reduces the number of data by keeping a fraction of them (1/n, n is chosen by the user). Final data is based on the result of a median calculated with a moving window on the n decimated points.
Spatial GNSS/GEM2/EMP400 shift
In the case of a GEM2/EMP400-GNSS acquisition, for practical reasons, it is possible to have a spatial shift between the GNSS antenna and the center of the GEM2/EMP400. Shift correction is calculated in accordance with the direction of the surveyor and the device.
Median filtering by profile (In-phase/Out-of-phase)
This function allows the removal of the median value for each profile in order to reduce the effects due to the difference in ground clearance, or an inaccurate horizontal orientation. Median calculation by profile can be applied on in-phase and/or out-of-phase datasets.
Moving window filter
This function applies a 1D moving window filter along each profile. The user can select the method (median or average method), the in-phase or out-of-phase component and the window size.
Geophysical parameter calculation
Calibration
This processing enables a device calibration based on the up and down measurements collected at the location of a vertical electrical sounding or a known mean electrical resistivity value. In order to do the calibration, click on the button Calibration parameter. You will then have to fill in the EMI calibration file path. This file contains the up and down measurements (Thiesson et al. 2014) needed for the calibration. It must contain 6 measurement points, alternating between the up and down measurement, starting with a down measurement. As the GEM2/EMP400 is unable to acquire discrete points, each measurement corresponds to a static continuous profile to determine the value of each point (with an average calculation). The user has to specify the height of the device from the ground for both up and down measurements (when the GEM2/EMP400 is lying on the ground the respective height value is 0.02 m due to the frame of the GEM2/EMP400). It is then possible to choose for the reference value, an average resistivity value or a 5 layers model. If you only have a 3 or 4 layers, fill the last one(s) with the same value (Example: for a 3 layers model, rau3=rau4=rau5).
Electrical conductivity and magnetic susceptibility computation
The processing module estimates the apparent electrical conductivity values and the apparent magnetic susceptibility values based on the solution of the integrals and the Hankel transform (Thiesson et al. 2014). This solution takes into account the device’s height and the coils’ configuration. It can therefore be applied regardless of the type of soil and is even valid in high conductivity soil medium.
For the apparent magnetic susceptibility computation, the process enables the removal of the conductivity effect on the in-phase signal. Warning: This option implies a relatively long processing time. It is also necessary to firstly check that the validity of the previously calculated conductivity (necessary for this calculation) is not negative in which case the estimation of the susceptibility will be incorrect.
The processed points are saved in a shapefile with the user-defined RCS. It contains X and Y positions, in-phase and out-of-phase processed values and the computed conductivity and susceptibility values for each frequency. In addition, the shapefile contains the profile number, the time stamp, the GNSS quality and the altitude.
Thiesson J., Kessouri P., Schamper C., Tabbagh A. 2014 - Calibration of frequency-domain electromagnetic devices used in near-surface surveying. Near Surface Geophysics, 12, 481-491.
Interpolator Module¶
This module allows the interpolation of geophysical data loaded as a shapefile point format. It uses SAGA interpolators (http://www.saga-gis.org/). It is suitable for Electrical, Magnetic and Electromagnetic (EMI) data, each using a different interpolation method for which most of the interpolating parameters have already been prefilled for ease of use:
Interpolator for magnetic data: Inverse Weighted Distance function
Defaults parameters :
‘DW_BANDWIDTH’: 1
‘DW_IDW_OFFSET’: False
‘DW_IDW_POWER’: 2
‘DW_WEIGHTING’: 1
‘SEARCH_DIRECTION’: 0
‘SEARCH_POINTS_ALL’: 0
‘SEARCH_POINTS_MAX’: 20
‘SEARCH_POINTS_MIN’: -1
‘SEARCH_RANGE’: 0
‘TARGET_DEFINITION’: 0
‘TARGET_TEMPLATE’: None
‘TARGET_USER_FITS’: 0
Interpolator for electrical resistivity data: Multi-level B Spline function
Defaults parameters :
‘EPSILON’: 0.0001
‘LEVEL_MAX’: 11
‘METHOD’: 1
‘TARGET_USER_FITS’: 0
Interpolator for electromagnetic data: Cubic Spline Interpolation function
Defaults parameters :
‘K’: 10
‘NPMAX’: 11
‘NPMIN’: 1
‘NPPC’: 3
‘TARGET_USER_FITS’: 0
These parameters have been defined as optimal for these three geophysical methods. However, if the user wishes to modify them, we invite him to use directly the interpolation functions provided by SAGA Processing Toolbox.
The user-defined parameters for the interpolation are the selected shapefile and the field to interpolate. The user has to enter the output raster pixel size and the search radius (in meters). Please note that the paths must not contain any special characters, as these may cause errors when running SAGA tools.
Minimum configuration: QGIS versions greater than or equal to 3.10.5 and SAGA tools must be previously activated in QGIS
Median Filtering Module¶
This module allows performing a median filtering on a raster file. Its specificity, compared to a classical median filtering is the possibility to define a filtering threshold corresponding to the maximum value that the tested pixel must not exceed in relation to the kernel median value (window on which the median is evaluated). The resulting processing file must be saved in raster .tif format.
The user has to define the size of the window on which the median is evaluated (5 corresponds to a 5x5 pixels window) and the threshold corresponds to the deviation in per cent between the value and the median.
This tool is inspired by the functions from the Geophpy library (https://pypi.org/project/GeophPy/) and more generally by Wumap (Laboratoire UMR 7619 Metis formerly Sisyphe).
Future developments¶
RM15/RM85 Download module