# 🌊 Bathymetrix-AI V6.3: Complete User Manual & Outputs Guide

Welcome to the definitive guide for **Bathymetrix-AI V6.3**. This document serves as an all-in-one, easy-to-read reference for understanding the terminology, the exact outputs generated by each phase grouped by folders and files, and the editable parameters available in the user interface.

---

## 📖 1. Abbreviations & Terminology

*   **SDB (Satellite-Derived Bathymetry):** The science of calculating water depth using the color (spectral reflectance) of the seabed captured by satellite sensors.
*   **OSW (Optically Shallow Water):** The coastal zone where sunlight successfully hits the sea bottom and bounces back to the satellite. Bathymetry is **only** valid here.
*   **ROI (Region of Interest):** A custom polygon you provide to tell the algorithm: *"Sample the pixels inside this shape to calibrate your thresholds."*
*   **NIR (Near-Infrared):** A band of light (e.g., Band 8 in Sentinel-2) that water absorbs heavily. It's used to detect the edge of water, land, and deep ocean.
*   **NDWI (Normalized Difference Water Index):** A mathematical formula used to easily distinguish water from dry land.
*   **Residuals (Prediction Error):** The exact difference (in meters) between what the ML model guessed and the true depth measured by the ICESat-2 laser.

---

## 📁 2. Pipeline Outputs by Directory Structure

All files and directories generated by SDB MasterFlow are organized inside your specified `Output Folder`. The directory tree is structured as follows:

```
[Output Folder] (Main Project Folder)
│
├── SDB_Validation_Dashboard.html
├── SDB_Full_Log.txt
├── 5_FINAL_SUMMARY.txt
├── 5_Stratified_Error_Analysis.csv
├── 5_Validation_Raw_Data.csv
├── 5_Plot_Scatter_Comparison.png
├── 5_Plot_Residuals.png
├── 5_Plot_Error_Histogram.png
│
├── 📁 Phase_01_Preprocessing/
│   ├── 01_Land_Water_Mask.tif
│   ├── 02_Sunglint_Corrected.tif
│   ├── 03_Feature_Stack.tif
│   ├── 04_DeepWater_Threshold_Stats.csv
│   ├── 05_ShallowWater_Pixel_Mask.tif
│   ├── 06_Final_OSW_Mask.tif
│   └── 07_OSW_Boundary_Polygon.gpkg
│
├── 📁 Phase_02_Filtering/
│   ├── 2_Cleaned_Training_Data.shp
│   ├── 2_Outliers_Rejected.shp
│   ├── 2_Plot_1_Trend.png
│   ├── 2_Plot_2_Variance.png
│   └── 2_Plot_3_Envelope.png
│
├── 📁 Phase_03_Initial_Modeling/
│   ├── 3_Actual_Model_Input_Points.shp
│   ├── 3_Feature_Analysis_Report.txt
│   ├── 3_Feature_Correlation_Plot.png
│   ├── 3_All_Algorithms_Benchmark.csv
│   ├── Phase3_Depth_Cleaned.tif
│   └── 📁 [Algorithm_Name]/
│       └── Validation_Scatter_Plot.png
│
└── 📁 Phase_04_Adaptive_Refinement/
    ├── 4_Feature_Analysis_Report.txt
    ├── 4_Feature_Correlation_Plot.png
    ├── 4-Residual_Error_Map.tif
    ├── 4_All_Algorithms_Benchmark.csv
    └── Final_Depth_Cleaned.tif
```

### 🏠 Main Project Folder (Root Directory)
This directory contains the final validation outputs, logs, and the web dashboard.
*   **`SDB_Validation_Dashboard.html`** (Interactive Web Dashboard):
    *   A comprehensive, premium-themed HTML dashboard visualizing project metrics, AutoML leaderboards, filtering plots, and scientific validation graphs.
    *   Features the project folder, processing status, and developer credentials (`Mohamed Aly Nasef`) along with the tool identifier (`SDB MasterFlow`) in the header.
*   **`SDB_Full_Log.txt`** (Processing Log):
    *   The complete running log of the pipeline containing timestamps, execution warnings, and information messages from all phases.
*   **`5_FINAL_SUMMARY.txt`** (Scientific Report):
    *   Text report comparing the overall accuracy statistics (RMSE, R2, MAE, wMAPE) between Phase 03 and Phase 04.
*   **`5_Stratified_Error_Analysis.csv`** (Spreadsheet):
    *   A breakdown of model accuracy metrics across specific depth intervals (e.g., 0-5m, 5-10m).
*   **`5_Validation_Raw_Data.csv`** (Spreadsheet):
    *   Spreadsheet dumping all validation points containing their actual coordinates, observed depth, and predicted SDB depth.
*   **`5_Plot_Scatter_Comparison.png`** (Image):
    *   A 1:1 scatter comparison plot of predicted vs. observed depths to evaluate overall correlation.
*   **`5_Plot_Residuals.png`** (Image):
    *   A residuals plot showing prediction error (residuals) against observed depth to inspect any systematic models bias.
*   **`5_Plot_Error_Histogram.png`** (Image):
    *   A density histogram displaying error distribution (centered on 0 indicates an unbiased prediction).

---

### 📁 Phase_01_Preprocessing/
Contains the results of water masking, sunglint correction, and optically shallow water extraction.
*   **`01_Land_Water_Mask.tif`** (Binary Raster: 0/1):
    *   **What it is:** The boundaries of water. `1` = Water, `0` = Land/Clouds.
    *   **Why it matters:** Acts as a cookie-cutter to ensure processing is only done on water, saving time and memory.
*   **`02_Sunglint_Corrected.tif`** (Float32 Raster):
    *   **What it is:** Original satellite image but wave-induced sun reflections have been mathematically subtracted.
    *   **Why it matters:** Removes surface glare to reveal the true seabed color underneath.
*   **`03_Feature_Stack.tif`** (Float32 Raster):
    *   **What it is:** The ML dataset containing mathematically calculated "Log-Ratios" (e.g., Blue/Green).
    *   **Why it matters:** Linearizes light decay in water. The ML model learns depth exclusively from this file.
*   **`04_DeepWater_Threshold_Stats.csv`** (Spreadsheet):
    *   **What it is:** The mathematical limits showing the exact Mean and Standard Deviation calculated by the algorithm.
    *   **Why it matters:** Separates Shallow from Deep water thresholds dynamically.
*   **`05_ShallowWater_Pixel_Mask.tif`** (Binary Raster: 0/1):
    *   **What it is:** The raw filter result of pixels that passed the Deep Water thresholding (`1`=Shallow, `0`=Deep).
    *   **Why it matters:** Shows the raw statistical cutoffs before any spatial smoothing is applied (often looks noisy).
*   **`06_Final_OSW_Mask.tif`** (Binary Raster: 0/1):
    *   **What it is:** The clean footprint of the Optically Shallow Water mask.
    *   **Why it matters:** It is polished by removing speckle noise and filling internal gaps, representing the true valid zone.
*   **`07_OSW_Boundary_Polygon.gpkg`** (Vector Polygon):
    *   **What it is:** The final clipper outlining the OSW mask.
    *   **Why it matters:** Automatically clips the final ML depth maps so they don't predict fake depths in the deep ocean.
    *   **Coordinate System:** Automatically inherits and embeds the Spatial Reference System (SRS/CRS) of the source satellite image to ensure seamless alignment and prevent GDAL cutline warnings during subsequent clipping phases.

---

### 📁 Phase_02_Filtering/
Contains the results of robust point filtering and outlier rejection.
*   **`2_Cleaned_Training_Data.shp`** (Vector Points):
    *   **What it is:** Ground truth data containing clean ICESat-2 depths.
    *   **Why it matters:** The RANSAC/Variance filter aggressively deletes noise (waves, birds, etc.), ensuring the ML model only trains on perfect data.
*   **`2_Outliers_Rejected.shp`** (Vector Points):
    *   **What it is:** The points that the algorithm flagged as noise or outliers and removed.
*   **`2_Plot_1_Trend.png`**, **`2_Plot_2_Variance.png`**, **`2_Plot_3_Envelope.png`** (Images):
    *   **What they are:** Diagnostic plots showing the statistical fit and uncertainty envelopes used during filtering.

---

### 📁 Phase_03_Initial_Modeling/
Contains Auto-ML benchmarking, band selection reports, and initial SDB depth maps.
*   **`3_Actual_Model_Input_Points.shp`** (Vector Points):
    *   **What it is:** The final points list used directly for model training after collision handling.
*   **`3_Feature_Analysis_Report.txt`** & **`3_Feature_Correlation_Plot.png`**:
    *   **What they are:** Report and plot showing how much each input band correlates with or contributes to the ML model's predictions, with a visual threshold for selected bands.
*   **`3_All_Algorithms_Benchmark.csv`** (Spreadsheet):
    *   **What it is:** CSV benchmark leaderboard of all tested Auto-ML models with their R², RMSE, and wMAPE metrics.
*   **`Phase3_Depth_Cleaned.tif`** (Float32 Raster):
    *   **What it is:** The Base SDB Map generated by the winning ML algorithm, cleaned and clipped to the valid shallow water zone.
*   **📁 [Algorithm_Name]/`Validation_Scatter_Plot.png`** (Image):
    *   **What it is:** Local validation scatter plot generated for each benchmarked algorithm.

---

### 📁 Phase_04_Adaptive_Refinement/
Contains the results of residual error correction and final refined SDB maps.
*   **`4_Feature_Analysis_Report.txt`** & **`4_Feature_Correlation_Plot.png`**:
    *   **What they are:** Identical to Phase 03, but specific to the residual retraining step.
*   **`4-Residual_Error_Map.tif`** (Float32 Raster):
    *   **What it is:** The Error Map showing exactly where Phase 03 was wrong and by how much.
    *   **Why it matters:** Uses K-NN/Kriging to interpolate errors across the entire area to catch geographic biases.
*   **`4_All_Algorithms_Benchmark.csv`** (Spreadsheet):
    *   **What it is:** Benchmark leaderboard of Auto-ML models after adaptive refinement.
*   **`Final_Depth_Cleaned.tif`** (Float32 Raster):
    *   **What it is:** The Final SDB Map.
    *   **Why it matters:** The ultimate, highest-accuracy map where the Error Surface has been mathematically applied to fix local biases.

---

## 🎛️ 3. User Interface (UI) Parameters & Defaults

This section explains every editable parameter in the plugin so you can tune it perfectly for your region.

> [!TIP]
> **Advanced Parameter Grouping:** All advanced parameters in the tool UI are now categorized with clear prefixes to indicate their scope:
> *   `[Phase 03]` - For base Auto-ML model parameters (e.g., base model spatial CV, ensemble size).
> *   `[Phase 04]` - For residual model parameters (e.g., residual spatial CV, KNN neighbors, P4 ensemble).
> *   `[Phase 03 & 04]` - For parameters shared by both modeling phases (e.g., training split ratio).
> *   `[General ML]` - For model hyperparameter search settings (e.g., RF/GB/XGB hyperparameter bounds).
> *   `[General]` - For general processing settings (e.g., output bit depth format, random seed state).


### ⚙️ [0] General Settings
*   **Processing Threads** `[Default: 4]`
    *   **What it does:** Dictates how many CPU cores to use. Higher numbers mean faster processing.

### 📡 [1] Phase 01: Advanced Pre-processing
*   **Coastal/Blue/Green/Red/NIR Bands** `[Defaults: 1, 2, 3, 4, 8]`
    *   **What it does:** Maps the specific band numbers of your input image (defaults match Sentinel-2).
*   **Apply Sunglint Correction** `[Default: True]`
    *   **What it does:** Removes surface reflections (wave glare) using Hedley's algorithm.
*   **Sunglint Deep Water %** `[Default: 1.0]`
    *   **What it does:** Uses the darkest 1% of pixels as the baseline to subtract glint.
*   **Enable Automated Masking** `[Default: False]`
    *   **What it does:** Turns on the automated water masking algorithm. Includes **Advanced Water Masking** using 3-Indices (NDWI, MNDWI, NWI) to accurately find water boundaries.
*   **Apply Deep Water Filter** `[Default: True]`
    *   **What it does:** Turns on the **Deep Water Filter** (customized for ML algorithms) to mathematically separate shallow from deep water and build the OSW mask.
*   **NIR Percentile for Deep Water** `[Default: 10.0]`
    *   **What it does:** Uses the darkest 10% of NIR pixels to calculate the statistical deep water thresholds.
*   **OSW Mask Median Filter Size** `[Default: 3]`
    *   **What it does:** Sets the size of the smoothing brush used to clean speckles from the OSW mask.

### 📏 [2] Phase 02: Robust Filtering
*   **Maximum Depth Threshold** `[Default: -30.0]`
    *   **What it does:** Deletes ICESat-2 points deeper than 30m, as optical SDB rarely penetrates deeper than this.
*   **Enable Data Filtering** `[Default: True]`
    *   **What it does:** Turns on the noise-removal algorithm (RANSAC/Variance Fit) for the point cloud.
*   **Filtering Strategy** `[Default: 2 - LS Variance]`
    *   **What it does:** Selects the mathematical approach used to find the true seabed points.
*   **Threshold / Sigma Multiplier** `[Default: 3.0]`
    *   **What it does:** Controls how aggressive the filter is. A higher number keeps more points but allows more noise.
*   **RANSAC Trials** `[Default: 100]`
    *   **What it does:** Sets the number of iterations the algorithm uses to find the best line of fit.

### 🤖 [3] Phase 03: Global Auto-ML & Feature Analysis
*   **Feature Correlation Method** `[Default: Automatic-RANSAC]`
    *   **What it does:** Chooses the method to evaluate band strength: Disabled (no filtering), Pearson (Linear), Spearman (Rank), Automatic-RANSAC (dynamic statistics), or Automatic-Random Forest (AutoML Feature Importance).
*   **Feature Correlation Threshold** `[Default: 0.2 (Selected from Dropdown)]`
    *   **What it does:** Drops weak bands whose score falls below this threshold. (Ignored if an Automatic method is selected).
*   **Algorithms to Benchmark** `[Default: All]`
    *   **What it does:** Chooses which AI models to pit against each other (Random Forest, SVR, Extra Trees, etc.).
*   **Optimization Iterations** `[Default: 10]`
    *   **What it does:** Determines how many times the AI tries different internal hyperparameters to get the best score.
*   **Output Median Filter Size** `[Default: 5]`
    *   **What it does:** Applies a final spatial smoothing kernel to the output depth map to remove "salt and pepper" pixel noise.
*   **[Phase 03] Enable Spatial Block Cross-Validation** `[Default: False]`
    *   **What it does:** Independently controls whether spatial block cross-validation is performed for the base global models.


### 🎯 [4] Phase 04: Adaptive Refinement
*   **Enable Adaptive Refinement** `[Default: False]`
    *   **What it does:** Turns on the K-NN spatial error correction to fix local geographic biases (e.g., dark seagrass mimicking deep water).
*   **[Phase 04] Enable Spatial Block Cross-Validation** `[Default: False]`
    *   **What it does:** Independently controls whether spatial block cross-validation is performed for the residual adaptive refinement.


### 📉 [5] Phase 05: Validation & Reporting
*   **Enable Validation** `[Default: False]`
    *   **What it does:** Triggers the creation of the Phase 05 reports, statistics, and graphs.
*   **Remove Positive Depths** `[Default: True]`
    *   **What it does:** Cleans the final map by deleting any fake elevations (like boats or piers) above 0m sea level.
*   **Apply Slope Filter** `[Default: True]`
    *   **What it does:** Deletes physically impossible sharp spikes or drops in the seabed caused by model glitches.
*   **Slope Filter Threshold** `[Default: 35.0]`
    *   **What it does:** Sets the maximum allowable slope (in degrees). Anything steeper is deleted.
