# 🌊 Bathymetrix-AI V7.0: Complete User Manual & Outputs Guide

Welcome to the definitive guide for **Bathymetrix-AI V7.0**. This document serves as an all-in-one, easy-to-read reference for understanding the terminology, the exact outputs generated by each phase grouped by folders and files, and the editable parameters available in the user interface.

---

## 📖 1. Abbreviations & Terminology

*   **SDB (Satellite-Derived Bathymetry):** The science of calculating water depth using the color (spectral reflectance) of the seabed captured by satellite sensors.
*   **OSW (Optically Shallow Water):** The coastal zone where sunlight successfully hits the sea bottom and bounces back to the satellite. Bathymetry is **only** valid here.
*   **ROI (Region of Interest):** A custom polygon you provide to tell the algorithm: *"Sample the pixels inside this shape to calibrate your thresholds."*
*   **NIR (Near-Infrared):** A band of light that water absorbs heavily. It's used to detect the edge of water, land, and deep ocean.
*   **NDWI (Normalized Difference Water Index):** A mathematical formula used to easily distinguish water from dry land.
*   **Residuals (Prediction Error):** The exact difference (in meters) between what the ML model guessed and the true depth measured by the ICESat-2 laser.
*   **MSI (Morphological Stability Index):** A normalized index indicating how physically stable or volatile the seabed is over multiple years.

---

## 📁 2. Pipeline Outputs by Directory Structure

All files and directories generated by Bathymetrix-AI are organized inside your specified `Output Folder`. The directory tree is structured as follows:

```
[Output Folder] (Main Project Folder)
│
├── SDB_Validation_Dashboard.html
├── SDB_Full_Log.txt
├── 5_FINAL_SUMMARY.txt
├── 5_Stratified_Error_Analysis.csv
├── 5_Validation_Raw_Data.csv
├── 5_Plot_Scatter_Comparison.png
├── 5_Plot_Residuals.png
├── 5_Plot_Error_Histogram.png
│
├── 📁 Phase_01_Preprocessing/
│   ├── 01_Land_Water_Mask.tif
│   ├── 02_Sunglint_Corrected.tif
│   ├── 03_Feature_Stack.tif
│   ├── 04_DeepWater_Threshold_Stats.csv
│   ├── 05_ShallowWater_Pixel_Mask.tif
│   ├── 06_Final_OSW_Mask.tif
│   └── 07_OSW_Boundary_Polygon.gpkg
│
├── 📁 Phase_02_Filtering/
│   ├── 2_Cleaned_Training_Data.shp
│   ├── 2_Outliers_Rejected.shp
│   ├── 2_Plot_1_Trend.png
│   ├── 2_Plot_2_Variance.png
│   └── 2_Plot_3_Envelope.png
│
├── 📁 Phase_03_Initial_Modeling/
│   ├── 3_Actual_Model_Input_Points.shp
│   ├── 3_Feature_Analysis_Report.txt
│   ├── 3_Feature_Correlation_Plot.png
│   ├── 3_All_Algorithms_Benchmark.csv
│   ├── Phase3_Depth_Cleaned.tif
│   └── 📁 [Algorithm_Name]/
│       └── Validation_Scatter_Plot.png
│
├── 📁 Phase_04_Adaptive_Refinement/
│   ├── 4_Feature_Analysis_Report.txt
│   ├── 4_Feature_Correlation_Plot.png
│   ├── 4-Residual_Error_Map.tif
│   ├── 4_All_Algorithms_Benchmark.csv
│   └── Final_Depth_Cleaned.tif
│
└── 📁 Coastal_Dynamics_Analysis/ (Standalone Tool)
    ├── Temporal_Analytics_Report.html
    ├── Sediment_Mass_Balance.csv
    ├── Benthic_Classification_Stats.csv
    ├── SDB_[YYYY].tif
    ├── Volumetric_Erosion_Accretion_[First]_[Last].tif
    ├── Morphological_Stability_Index_[First]_[Last].tif
    └── Shoreline_Change_Polygons_[First]_[Last].shp
```

### 🏠 Main Project Folder (Root Directory)
This directory contains the final validation outputs, logs, and the web dashboard.
*   **`SDB_Validation_Dashboard.html`** (Interactive Web Dashboard):
    *   A comprehensive, premium-themed HTML dashboard visualizing project metrics, AutoML leaderboards, filtering plots, and scientific validation graphs.
    *   Features the project folder, processing status, and developer credentials (`Mohamed Aly Nasef`) along with the tool identifier (`SDB MasterFlow`) in the header.
*   **`SDB_Full_Log.txt`** (Processing Log):
    *   The complete running log of the pipeline containing timestamps, execution warnings, and information messages from all phases.
*   **`5_FINAL_SUMMARY.txt`** (Scientific Report):
    *   Text report comparing the overall accuracy statistics (RMSE, R2, MAE, wMAPE) between Phase 03 and Phase 04.
*   **`5_Stratified_Error_Analysis.csv`** (Spreadsheet):
    *   A breakdown of model accuracy metrics across specific depth intervals (e.g., 0-5m, 5-10m).
*   **`5_Validation_Raw_Data.csv`** (Spreadsheet):
    *   Spreadsheet dumping all validation points containing their actual coordinates, observed depth, and predicted SDB depth.
*   **`5_Plot_Scatter_Comparison.png`** (Image):
    *   A 1:1 scatter comparison plot of predicted vs. observed depths to evaluate overall correlation.
*   **`5_Plot_Residuals.png`** (Image):
    *   A residuals plot showing prediction error (residuals) against observed depth to inspect any systematic models bias.
*   **`5_Plot_Error_Histogram.png`** (Image):
    *   A density histogram displaying error distribution (centered on 0 indicates an unbiased prediction).

---

### 📁 Phase_01_Preprocessing/
Contains the results of water masking, sunglint correction, and optically shallow water extraction.
*   **`01_Land_Water_Mask.tif`** (Binary Raster: 0/1):
    *   **What it is:** The boundaries of water. `1` = Water, `0` = Land/Clouds.
    *   **Why it matters:** Acts as a cookie-cutter to ensure processing is only done on water, saving time and memory.
*   **`02_Sunglint_Corrected.tif`** (Float32 Raster):
    *   **What it is:** Original satellite image but wave-induced sun reflections have been mathematically subtracted.
    *   **Why it matters:** Removes surface glare to reveal the true seabed color underneath.
*   **`03_Feature_Stack.tif`** (Float32 Raster):
    *   **What it is:** The ML dataset containing mathematically calculated Log-Ratios and Indices (e.g., NDWI, Blue/Green, Green/NIR).
    *   **Why it matters:** Linearizes light decay in water. The ML model learns depth exclusively from this file.
*   **`04_DeepWater_Threshold_Stats.csv`** (Spreadsheet):
    *   **What it is:** The mathematical limits showing the exact Mean and Standard Deviation calculated by the algorithm.
    *   **Why it matters:** Separates Shallow from Deep water thresholds dynamically.
*   **`05_ShallowWater_Pixel_Mask.tif`** (Binary Raster: 0/1):
    *   **What it is:** The raw filter result of pixels that passed the Deep Water thresholding (`1`=Shallow, `0`=Deep).
    *   **Why it matters:** Shows the raw statistical cutoffs before any spatial smoothing is applied (often looks noisy).
*   **`06_Final_OSW_Mask.tif`** (Binary Raster: 0/1):
    *   **What it is:** The clean footprint of the Optically Shallow Water mask.
    *   **Why it matters:** It is polished by removing speckle noise and filling internal gaps using advanced Elbow Point Detection, representing the true valid zone.
*   **`07_OSW_Boundary_Polygon.gpkg`** (Vector Polygon):
    *   **What it is:** The final clipper outlining the OSW mask.
    *   **Why it matters:** Automatically clips the final ML depth maps so they don't predict fake depths in the deep ocean.
    *   **Coordinate System:** Automatically inherits and embeds the Spatial Reference System (SRS/CRS) of the source satellite image to ensure seamless alignment and prevent GDAL cutline warnings during subsequent clipping phases.

---

### 📁 Phase_02_Filtering/
Contains the results of robust point filtering and outlier rejection.
*   **`2_Cleaned_Training_Data.shp`** (Vector Points):
    *   **What it is:** Ground truth data containing clean ICESat-2 depths.
    *   **Why it matters:** The RANSAC/Variance filter aggressively deletes noise (waves, birds, etc.), ensuring the ML model only trains on perfect data.
*   **`2_Outliers_Rejected.shp`** (Vector Points):
    *   **What it is:** The points that the algorithm flagged as noise or outliers and removed.
*   **`2_Plot_1_Trend.png`**, **`2_Plot_2_Variance.png`**, **`2_Plot_3_Envelope.png`** (Images):
    *   **What they are:** Diagnostic plots showing the statistical fit and uncertainty envelopes used during filtering. These plots use dynamic robust scaling to prevent extreme outliers from squishing the main data trends.

---

### 📁 Phase_03_Initial_Modeling/
Contains Auto-ML benchmarking, band selection reports, and initial SDB depth maps.
*   **`3_Actual_Model_Input_Points.shp`** (Vector Points):
    *   **What it is:** The final points list used directly for model training after collision handling. Features safe coordinate reprojection to prevent out-of-bounds errors.
*   **`3_Feature_Analysis_Report.txt`** & **`3_Feature_Correlation_Plot.png`**:
    *   **What they are:** Report and plot showing how much each input band correlates with or contributes to the ML model's predictions, with a visual threshold for selected bands.
*   **`3_All_Algorithms_Benchmark.csv`** (Spreadsheet):
    *   **What it is:** CSV benchmark leaderboard of all tested Auto-ML models with their R², RMSE, and wMAPE metrics.
*   **`Phase3_Depth_Cleaned.tif`** (Float32 Raster):
    *   **What it is:** The Base SDB Map generated by the winning ML algorithm, cleaned and clipped to the valid shallow water zone. Uses chunk-based ML prediction for memory efficiency and supports Uncertainty-Weighted Pixel Fusion for ensemble blending.
*   **📁 [Algorithm_Name]/`Validation_Scatter_Plot.png`** (Image):
    *   **What it is:** Local validation scatter plot generated for each benchmarked algorithm.

---

### 📁 Phase_04_Adaptive_Refinement/
Contains the results of residual error correction and final refined SDB maps.
*   **`4_Feature_Analysis_Report.txt`** & **`4_Feature_Correlation_Plot.png`**:
    *   **What they are:** Identical to Phase 03, but specific to the residual retraining step.
*   **`4-Residual_Error_Map.tif`** (Float32 Raster):
    *   **What it is:** The Error Map showing exactly where Phase 03 was wrong and by how much.
    *   **Why it matters:** Uses K-NN/Kriging to interpolate errors across the entire area to catch geographic biases.
*   **`4_All_Algorithms_Benchmark.csv`** (Spreadsheet):
    *   **What it is:** Benchmark leaderboard of Auto-ML models after adaptive refinement.
*   **`4-Refined_Uncertainty.tif`** (Float32 Raster):
    *   **What it is:** Empirical residual prediction uncertainty map at 95% confidence ($1.96 \sigma$).
    *   **Why it matters:** Evaluates spatial confidence for every pixel across the refined SDB map.
*   **`Final_Depth_Cleaned.tif`** (Float32 Raster):
    *   **What it is:** The Final SDB Map.
    *   **Why it matters:** The ultimate, highest-accuracy map where zero-mean centered spatial residual corrections and LOO robust Huber/smoothed IDW error surfaces have been applied to fix local biases. Incorporates Uncertainty-Weighted Pixel Fusion for robust model stacking.

---

### 🌊 📁 Aggregated_Results/ (SpatioSpectral Masterflow & Module 07)
Contains the merged results of multiple imageries processed with the same training dataset.
*   **`Aggregated_Depth_[Method].tif`** (Float32 Raster):
    *   **What it is:** The final selected depth map using statistical fusion (e.g., Weighted Median based on R2/RMSE).
    *   **Why it matters:** Harnesses the strengths of multiple satellite images, delivering the highest quality SDB output.
*   **`Aggregated_Intersection_Mask.tif`** (Binary Raster):
    *   **What it is:** The common valid area across all fused imageries.

---

### 🌊 📁 Coastal Dynamics Analysis (Standalone Module 06)
Contains the results of the volume calculations and shoreline migration tracking.
*   **`Temporal_Analytics_Report.html`** (Interactive Report):
    *   **What it is:** A comprehensive HTML report detailing volumetric changes, dredging volumes, target sub-region ROI metrics, uncertainty mode settings, and morphological stability with a color-coded spreadsheet.
    *   **Quality Control (QC) Value:** The Target ROI metrics within this report allow users to validate the model's real-world accuracy by confirming that calculated erosion/accretion quantities align with expected physical changes (e.g., known dredging volumes).
*   **`Quantile_Uncertainty_[Year].tif`** (Float32 Raster - generated when Quantile Regression mode is active):
    *   **What it is:** Spatially-adaptive uncertainty raster map ($\sigma$) estimated using Quantile Regression ($q_{\text{lower}}$ and $q_{\text{upper}}$) fitted on ground-truth ICESat-2 data.
*   **`Full_Process_Log.txt`** (Text File):
    *   **What it is:** Real-time log file recording all execution details, warnings, and processing steps throughout the Temporal Intelligence pipeline run.
*   **`Volumetric_Erosion_Accretion_[First]_[Last].tif`** (Float32 Raster):
    *   **What it is:** Measures the Magnitude and Direction of physical sand volume change (in cubic meters).
    *   **Methodology:** Uses a robust **Linear Regression (Time-Series Trend)** across all provided years. This mathematically filters out deep-water random noise and calculates the true rate of sand movement, identifying dredging (Erosion) and shoaling (Accretion). 
    *   **Mathematical Equation (Long-term Trend):**
        *   $Z_{diff} = Z_{arr} - \bar{Z}$
        *   $T_{diff} = T - \bar{T}$
        *   $\text{Slope (m/yr)} = \frac{\sum (T_{diff} \times Z_{diff})}{\sum (T_{diff}^2)}$
        *   $\Delta Z = \text{Slope} \times (\text{Year}_{last} - \text{Year}_{first})$
    *   **Auto-Detection:** Features **Auto-Detection** for Depth (positive) vs Elevation (negative) inputs via median analysis `np.nanmean(z1[valid]) > 0`, ensuring flawless sand tracking regardless of data format.
*   **`Morphological_Stability_Index_[First]_[Last].tif`** (Float32 Raster):
    *   **What it is:** Measures the Volatility and Consistency of the seabed, regardless of whether it is eroding or accreting.
    *   **Methodology:** A normalized index from 0 to 1 (0 = Highly Unstable, 1 = Highly Stable).
    *   **Mathematical Equation:**
        *   $MSI = 1.0 - \frac{\sigma_{true}}{|\mu| + 0.5}$
        *   Where $\sigma_{true} = \sqrt{\max(0, \sigma_{obs}^2 - \bar{\sigma}_u^2)}$ ($\bar{\sigma}_u$ is the mean uncertainty of the inputs).
*   **`StatCD_Uncertainty_Thresholding`** (Applied during Volume Calculation):
    *   The algorithm enforces a **Statistical Level of Detection (StatCD)** before calculating volumes. Any pixel where the magnitude of change $|\Delta Z|$ is less than the combined uncertainty $U_{comb}$ is forced to 0 (No Change).
    *   **Mathematical Equations (By Mode):**
        *   **Classical Mode (Z-score 1.96 $\sigma$):** $U_{comb} = 1.96 \times \sqrt{\sigma_1^2 + \sigma_2^2}$
        *   **MAD Mode (Median Absolute Deviation):** $U_{comb} = 1.4826 \times \text{MAD}$
        *   **Quantile Mode:** $U_{comb} = \frac{q_{upper} - q_{lower}}{1.349}$
*   **`Shoreline_Change_Polygons_[First]_[Last].shp`** (Vector Polygons):
    *   **What it is:** The horizontal migration of the coastline (Water-Land boundary) over time. Red polygons highlight coastal retreat (Erosion), while green polygons show beach advancement (Accretion).

---

## 🎛️ 3. User Interface (UI) Parameters & Defaults

This section explains editable parameters in the Temporal Intelligence and MasterFlow plugins.

### 🕒 Temporal Intelligence Settings
*   **Multi-Year Dataset Root Directory**
    *   **What it does:** The main folder containing subfolders for each year (e.g., `2019`, `2024`). Each subfolder must contain a Satellite Image and optional training/validation points.
*   **Global Training/Validation Points (Optional)**
    *   **What it does:** Instead of placing shapefiles in every yearly folder, you can provide one global shapefile containing a `Year` attribute field. The tool will automatically pair the correct points with the correct imagery.
*   **Target ROI for Volumetric Change (Polygon) (Optional)**
    *   **What it does:** Allows specifying a custom vector polygon mask to calculate accretion, erosion, and net sediment balance ($m^3$) specifically inside a targeted project zone in addition to the full study area. This is essential for **Model Quality Control**, enabling users to benchmark the pipeline's outputs against known ground-truth dredging or shoaling volumes in a specific area.
*   **Temporal Uncertainty Mode** `[Default: Classical Z-score (1.96σ)]`
    *   **What it does:** Selects between Classical Z-score (1.96σ homoscedastic Gaussian) and **Quantile Regression (Adaptive σ)** for spatially-adaptive uncertainty modeling.
*   **Quantile Regression Confidence Level** `[Default: 0.95]`
    *   **What it does:** Sets the confidence level (0.50–0.99) for Quantile Regression quantile bounds estimation.

### 📊 Sequential vs. Overall Trend Analysis Modes
*   **`Sequential` Mode (Year-to-Year Differences):**
    *   Calculates direct annual pixel subtraction ($\Delta Z = Z_{t2} - Z_{t1}$) between consecutive year pairs (e.g., 2019-2020, 2020-2021).
    *   **Use Case:** Captures short-term, annual, or seasonal morphodynamic changes, storm impacts, or immediate post-dredging volume changes.
*   **`Overall Trend` Mode (Multi-Year Linear Regression):**
    *   Computes pixel-wise robust linear regression across all dataset years (e.g., 2019 to 2024), multiplying the regression slope ($\text{m/year}$) by total elapsed time to derive overall volume changes.
    *   **Use Case:** Evaluates long-term morphological stability and sediment transport trends while filtering out single-year optical noise or temporal measurement anomalies.

> [!TIP]
> **Advanced Parameter Grouping:** All advanced parameters in the tool UI are now categorized with clear prefixes to indicate their scope:
> *   `[Phase 03]` - For base Auto-ML model parameters (e.g., base model spatial CV, ensemble size).
> *   `[Phase 04]` - For residual model parameters (e.g., residual spatial CV, KNN neighbors, P4 ensemble).
> *   `[Phase 03 & 04]` - For parameters shared by both modeling phases (e.g., training split ratio).
> *   `[General ML]` - For model hyperparameter search settings (e.g., RF/GB/XGB hyperparameter bounds).
> *   `[General]` - For general processing settings (e.g., output bit depth format, random seed state).


### ⚙️ [0] General Settings
*   **Processing Threads** `[Default: 4]`
    *   **What it does:** Dictates how many CPU cores to use. Higher numbers mean faster processing.

### 📡 [1] Phase 01: Advanced Pre-processing
*   **Coastal/Blue/Green/Red/NIR Bands** `[Defaults: 1, 2, 3, 4, 8]`
    *   **What it does:** Maps the specific band numbers of your input image.
*   **Apply Sunglint Correction** `[Default: True]`
    *   **What it does:** Removes surface reflections (wave glare) using Hedley's algorithm.
*   **Sunglint Deep Water %** `[Default: 1.0]`
    *   **What it does:** Uses the darkest 1% of pixels as the baseline to subtract glint.
*   **Enable Automated Masking** `[Default: False]`
    *   **What it does:** Turns on the automated water masking algorithm. Includes **Advanced Water Masking** using 3-Indices (NDWI, MNDWI, NWI) to accurately find water boundaries.
*   **Apply Deep Water Filter** `[Default: True]`
    *   **What it does:** Turns on the **Deep Water Filter** (customized for ML algorithms) to mathematically separate shallow from deep water and build the OSW mask.
*   **NIR Percentile for Deep Water** `[Default: 10.0]`
    *   **What it does:** Uses the darkest 10% of NIR pixels to calculate the statistical deep water thresholds. Can fall back automatically if manual OSW Polygons or ROIs fail.
*   **OSW Mask Median Filter Size** `[Default: 3]`
    *   **What it does:** Sets the size of the smoothing brush used to clean speckles from the OSW mask.

### 📏 [2] Phase 02: Robust Filtering
*   **Maximum Depth Threshold** `[Default: -30.0]`
    *   **What it does:** Deletes ICESat-2 points deeper than 30m, as optical SDB rarely penetrates deeper than this.
*   **Enable Data Filtering** `[Default: True]`
    *   **What it does:** Turns on the noise-removal algorithm (RANSAC/Variance Fit) for the point cloud.
*   **Filtering Strategy** `[Default: 2 - LS Variance]`
    *   **What it does:** Selects the mathematical approach used to find the true seabed points.
*   **Threshold / Sigma Multiplier** `[Default: 3.0]`
    *   **What it does:** Controls how aggressive the filter is. A higher number keeps more points but allows more noise.
*   **RANSAC Trials** `[Default: 100]`
    *   **What it does:** Sets the number of iterations the algorithm uses to find the best line of fit.

### 🤖 [3] Phase 03: Global Auto-ML & Feature Analysis
*   **Feature Correlation Method** `[Default: Automatic-RANSAC]`
    *   **What it does:** Chooses the method to evaluate band strength: Disabled (no filtering), Pearson (Linear), Spearman (Rank), Automatic-RANSAC (dynamic statistics), or Automatic-Random Forest (AutoML Feature Importance).
*   **Feature Correlation Threshold** `[Default: 0.2 (Selected from Dropdown)]`
    *   **What it does:** Drops weak bands whose score falls below this threshold. (Ignored if an Automatic method is selected).
*   **Algorithms to Benchmark** `[Default: All]`
    *   **What it does:** Chooses which AI models to pit against each other (Random Forest, SVR, Extra Trees, etc.).
*   **Optimization Iterations** `[Default: 10]`
    *   **What it does:** Determines how many times the AI tries different internal hyperparameters to get the best score.
*   **Output Median Filter Size** `[Default: 5]`
    *   **What it does:** Applies a final spatial smoothing kernel to the output depth map to remove "salt and pepper" pixel noise.
*   **[Phase 03] Enable Spatial Block Cross-Validation** `[Default: False]`
    *   **What it does:** Independently controls whether spatial block cross-validation is performed for the base global models.


### 🎯 [4] Phase 04: Adaptive Refinement
*   **Enable Adaptive Refinement** `[Default: False]`
    *   **What it does:** Turns on the K-NN spatial error correction to fix local geographic biases (e.g., dark seagrass mimicking deep water).
*   **[Phase 04] Enable Spatial Block Cross-Validation** `[Default: False]`
    *   **What it does:** Independently controls whether spatial block cross-validation is performed for the residual adaptive refinement.


### 📉 [5] Phase 05: Validation & Reporting
*   **Enable Validation** `[Default: False]`
    *   **What it does:** Triggers the creation of the Phase 05 reports, statistics, and graphs.
*   **Remove Positive Depths** `[Default: True]`
    *   **What it does:** Cleans the final map by deleting any fake elevations (like boats or piers) above 0m sea level.
*   **Apply Slope Filter** `[Default: True]`
    *   **What it does:** Deletes physically impossible sharp spikes or drops in the seabed caused by model glitches.
*   **Slope Filter Threshold** `[Default: 35.0]`
    *   **What it does:** Sets the maximum allowable slope (in degrees). Anything steeper is deleted.

### 🧩 [0.2] SpatioSpectral Aggregator Settings
*   **Aggregation Method** `[Default: Weighted Median (R2/RMSE)]`
    *   **What it does:** Determines how multiple depth maps from different dates are combined.
    *   **Select Best Scene (Highest R2 / Lowest RMSE):** Automatically scans the log files and selects the single highest-performing scene based on its training metrics.
    *   **Weighted Median / Weighted Mean:** Combines all scenes but mathematically gives more influence (higher weight) to scenes that had higher accuracy (R2) and lower error (RMSE) during training.
    *   **Median / Mean:** Simple, unweighted combination of all scenes.
