# EMDS Plugin Structure Documentation

## Overview

The EMDS QGIS Plugin is a comprehensive hierarchical workflow manager that integrates seamlessly with the existing engine bridge (`python-bridge/engine_bridge.py`). It provides a user-friendly interface for managing complex workflows while maintaining data integrity and layer visibility control.

## Architecture Diagram

```
┌─────────────────────────────────────────────────────────────────┐
│                        QGIS Application                          │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │              EMDSPlugin (emds_plugin.py)                  │  │
│  │  - Initializes plugin                                    │  │
│  │  - Creates dock widget                                  │  │
│  │  - Manages QGIS integration                             │  │
│  └───────────────────────────────────────────────────────────┘  │
│                            ▲                                      │
│                            │ signals/events                      │
│                            ▼                                      │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │         EMDSDockWidget (dock_widget.py)                   │  │
│  │  - Main UI container                                      │  │
│  │  - Manages user interactions                             │  │
│  │  - Handles tree view and context menus                   │  │
│  │  - Coordinates dialogs and data flow                     │  │
│  └───────────────────────────────────────────────────────────┘  │
│       ▲                 ▲                ▲                        │
│       │ tree updates    │ data updates   │ signals                │
│       ▼                 ▼                ▼                        │
│  ┌─────────────┐  ┌──────────────┐  ┌──────────────────┐        │
│  │ TreeModel   │  │ ProjectMgr   │  │ Dialogs          │        │
│  │ (tree_model)│  │ (models.py)  │  │ (dialogs.py)     │        │
│  │             │  │              │  │                  │        │
│  │ • Populate  │  │ • CRUD ops   │  │ • Create Project │        │
│  │ • Collapse  │  │ • Queries    │  │ • Create Assess. │        │
│  │ • Get paths │  │ • Hierarchy  │  │ • Create Proven. │        │
│  │             │  │              │  │ • Select Engine  │        │
│  │             │  │              │  │ • Config Step    │        │
│  └─────────────┘  └──────────────┘  └──────────────────┘        │
│                                                                   │
└─────────────────────────────────────────────────────────────────┘
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│              Engine Bridge (engine_bridge.py)                    │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  NetWeaverBridge    │    CDPBridge    │  Custom Engines   │  │
│  │  • process_data()   │ • process_data()│                   │  │
│  │  • validate_model() │ • validate_cfg()│                   │  │
│  │  • get_model_info() │ • get_info()    │                   │  │
│  └───────────────────────────────────────────────────────────┘  │
│                            ▼                                      │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │   .NET CLI Wrappers (subprocess)                          │  │
│  │   • NetWeaverCLI.exe  • CDPCLI.exe                        │  │
│  └───────────────────────────────────────────────────────────┘  │
│                            ▼                                      │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │   Real Engines (NetWeaver 2.0, CDP)                       │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
```

## File Organization

### Core Plugin Files

#### 1. **__init__.py**
- Plugin entry point
- Exports `classFactory()` function
- QGIS calls this to instantiate the plugin

```python
def classFactory(iface):
    return EMDSPlugin(iface)
```

#### 2. **metadata.txt**
- Plugin metadata (name, version, author)
- QGIS reads this to list the plugin
- Specifies minimum/maximum QGIS versions
- Defines category, tags, and icon

```ini
[general]
name=EMDS 2026-06
version=0.2.0
qgisMinimumVersion=4.0
author=EMDS Team
```

#### 3. **emds_plugin.py** (Main Plugin Class)
- **Purpose**: QGIS plugin lifecycle management
- **Key Methods**:
  - `initGui()`: Called when plugin loads → creates dock widget
  - `unload()`: Called when plugin unloads → cleanup
  - `toggle_dock()`: Show/hide dock widget
  - Signal handlers for item selection and visibility

**Responsibilities**:
- Initialize dock widget
- Register menu items and toolbar buttons
- Connect to QGIS signals
- Handle layer visibility in map canvas

### Data Layer

#### 4. **models.py** (Data Structures)
- Pure data models with no QGIS dependencies
- Hierarchical structure:
  - `Project`: Top-level container
  - `Assessment`: Spatial analysis (SpatialReference)
  - `Provenance`: Workflow grouping
  - `WorkflowStep`: Individual processing step (nested up to 6 levels)
  - `Results`: Output layer with metadata

**Key Features**:
- Dataclass-based models with `@dataclass` decorator
- Recursive methods for nested structures
- JSON serialization support
- Type hints for better IDE support

**ProjectManager**:
- Manages all projects
- Provides CRUD operations
- Recursive search for nested items
- JSON export functionality

### UI Layer

#### 5. **dock_widget.py** (Main UI Container)
- **Purpose**: Central hub for all UI interactions
- **Key Components**:
  - QTreeWidget for hierarchy display
  - Buttons for creating items
  - Context menu handling
  - Signal coordination

**Key Methods**:
- `_create_project()`: Workflow for new projects
- `_create_assessment()`: Workflow for new assessments
- `_create_provenance()`: Workflow for new provenances
- `_create_workflow_step()`: Two-step workflow (select engine → configure)
- `_delete_selected()`: Handle item deletion
- `_on_context_menu()`: Build and show context menu
- `_toggle_results_visibility()`: Control layer visibility

**Signals Emitted**:
- `project_created`: When project is created
- `item_selected`: When tree item is selected (includes path)
- `item_deleted`: When item is deleted
- `results_visibility_toggled`: When layer visibility changes

#### 6. **tree_model.py** (Tree View Logic)
- **Purpose**: Manages tree display and structure
- **Key Components**:
  - `TreeNodeData`: Data associated with each node
  - `EMDSTreeModel`: Static methods for tree operations

**Key Methods**:
- `create_item()`: Create tree widget item with metadata
- `populate_tree()`: Build entire tree from project manager
- `_add_workflow_step()`: Recursively add workflow steps
- `collapse_all_except_path()`: Smart collapsing
- `get_node_path()`: Extract IDs from selected item

**Smart Collapsing Logic**:
- When user selects an item, only that item's path is expanded
- All other branches are collapsed
- Reduces visual clutter for large hierarchies

#### 7. **dialogs.py** (User Input Dialogs)

**Dialogs**:
1. **CreateProjectDialog**
   - Project name and description
   - Input layer selection (checklist)
   - Returns: Project object

2. **CreateAssessmentDialog**
   - Assessment name and description
   - Returns: Assessment object

3. **CreateProvenanceDialog**
   - Provenance name and description
   - Returns: Provenance object

4. **SelectEngineDialog**
   - Radio buttons for engine types
   - NetWeaver, CDP, Template 1, Template 2
   - Returns: EngineType enum

5. **WorkflowStepConfigDialog**
   - Step name
   - Engine-specific parameters
   - Progress indicator
   - Returns: WorkflowStep object

### Integration Layer

#### 8. **example_integration.py** (Engine Bridge Integration)
- **Purpose**: Shows how to integrate with engine bridge
- **Key Components**:
  - `WorkflowExecutionThread`: Background thread for processing
  - `WorkflowExecutor`: Manager for workflow execution

**Key Methods**:
- `execute_step()`: Run a workflow step
- `_execute_netweaver()`: Process with NetWeaver
- `_execute_cdp()`: Process with CDP
- `load_layer_data()`: Load QGIS layer as DataFrame
- `add_results_layer()`: Add output as new QGIS layer

### Resource Files

#### 9. **resources.qrc**
- Qt resource file (XML format)
- References icon and other resources
- Can be compiled with `pyrcc5` if needed

#### 10. **resources/icon.svg**
- Plugin icon (32x32)
- SVG format for scalability
- Shown in plugin manager and toolbar

## Data Flow Examples

### Creating a Workflow Step

```
User clicks [+ Create Workflow Step]
    ↓
SelectEngineDialog opens (user chooses engine)
    ↓
WorkflowStepConfigDialog opens (user configures parameters)
    ↓
WorkflowStep object created
    ↓
Added to Provenance or parent WorkflowStep
    ↓
EMDSTreeModel.populate_tree() rebuilds tree
    ↓
workflow_step_created signal emitted
    ↓
Tree view updated with new item
```

### Executing a Workflow Step

```
User selects workflow step in tree
    ↓
item_selected signal emitted with path
    ↓
User clicks "Execute" (not yet in UI, for future)
    ↓
WorkflowExecutor.execute_step() called
    ↓
WorkflowExecutionThread spawned
    ↓
Thread calls appropriate engine bridge
    ↓
Results returned as DataFrame
    ↓
Results layer added to QGIS map
    ↓
Results object created and added to WorkflowStep
    ↓
Tree view updated with results
```

### Toggling Result Visibility

```
User clicks eye icon on results
    ↓
_toggle_results_visibility() called
    ↓
Results.visible toggled
    ↓
results_visibility_toggled signal emitted
    ↓
emds_plugin.py updates QGIS layer visibility
    ↓
Tree view item text updated
    ↓
Map canvas refreshed
```

## Integration Points with Engine Bridge

The plugin is designed to work with `python-bridge/engine_bridge.py`:

```python
# Example: Processing data through NetWeaver
from engine_bridge import NetWeaverBridge
import pandas as pd

bridge = NetWeaverBridge()
results_df = bridge.process_data(
    input_data=input_df,          # Pandas DataFrame
    model_file="model.nw2",       # From workflow step parameters
    timeout=300                   # From workflow step parameters
)

# Convert to GeoDataFrame and add to QGIS
gdf = gpd.GeoDataFrame(results_df)
layer = QgsVectorLayer(...)
```

## Future Enhancements

### Short-term (v0.2)
- [ ] Execute button and workflow execution UI
- [ ] Progress bars for long-running operations
- [ ] Error handling and recovery
- [ ] Execution history/logging
- [ ] Parameter validation before execution

### Medium-term (v0.3)
- [ ] Workflow templates and presets
- [ ] Batch execution of multiple steps
- [ ] Data lineage visualization
- [ ] Advanced parameter configuration UI
- [ ] Layer export in multiple formats

### Long-term (v1.0)
- [ ] Collaboration/sharing features
- [ ] Performance monitoring dashboard
- [ ] Integration with external data sources
- [ ] Advanced scheduling and automation
- [ ] Multi-user project management
- [ ] Database backend for large projects

## Testing

### Unit Testing
```bash
# Test data models
python3 -m pytest emds_qgis_plugin/test_models.py

# Test tree model
python3 -m pytest emds_qgis_plugin/test_tree_model.py
```

### Integration Testing
- Load plugin in QGIS
- Create project, assessments, provenances, workflow steps
- Test all CRUD operations
- Verify tree view updates correctly
- Test context menus
- Verify dialogs work as expected

### End-to-end Testing
- Execute workflow steps with sample data
- Verify results appear in QGIS map
- Test layer visibility toggling
- Verify results in tree view

## Troubleshooting Guide

### Plugin Not Loading
- Check `metadata.txt` syntax
- Verify `__init__.py` contains `classFactory()`
- Check QGIS log messages
- Ensure Python version compatibility

### Tree View Not Updating
- Verify `EMDSTreeModel.populate_tree()` is called
- Check that `ProjectManager` contains data
- Verify tree widget is properly initialized

### Engine Bridge Errors
- Ensure `python-bridge/engine_bridge.py` is in Python path
- Check engine subprocess is running
- Verify model files exist and are accessible
- Check system resources (disk space, memory)

### QGIS Layer Integration Issues
- Verify layer names match exactly
- Check `QgsProject.instance()` is available
- Ensure canvas is properly initialized
- Check layer geometry is valid

## Best Practices

1. **Data Validation**: Always validate user input in dialogs
2. **Error Handling**: Use try/except blocks around QGIS operations
3. **Signals/Slots**: Use Qt signals for loose coupling
4. **Threading**: Long operations should run in background threads
5. **Type Hints**: Always use type hints for better code clarity
6. **Documentation**: Document complex logic with docstrings
7. **Testing**: Write unit tests for models and algorithms
8. **Performance**: Monitor performance with large hierarchies

## Performance Considerations

- **Tree View**: Can handle ~1000 items efficiently
- **Nested Steps**: Max 6 levels to prevent visual clutter
- **Threading**: Use WorkflowExecutionThread for long operations
- **Memory**: Cache ProjectManager instead of rebuilding
- **UI Updates**: Batch tree view updates to avoid flicker

## Security Considerations

- **File Access**: Validate file paths before processing
- **User Input**: Sanitize all user input from dialogs
- **Engine Bridge**: Validate parameters passed to engines
- **Layer Files**: Check file permissions before writing results
- **Model Files**: Verify model file integrity before execution
