Compare commits
104 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| adf3b0f51c | |||
| 41ded0170c | |||
| 9b4191b2c3 | |||
| 53dbb43352 | |||
| ba03623110 | |||
| f24c4ed650 | |||
| 86501fd486 | |||
| 192eea6e0c | |||
| 43fb838c9b | |||
| 54483653f9 | |||
| e47803b705 | |||
| 439d0d9af6 | |||
| 2242b5ddfe | |||
| 9d0f42d55c | |||
| 1da7b5f6e7 | |||
| 006f27f7d9 | |||
| 07422cd0a7 | |||
| de30b80219 | |||
| 7d96ca9fad | |||
| 9b940ebd19 | |||
| 6d4da1b7da | |||
| 1e7f1616aa | |||
| 7a3ee3d5ba | |||
| 0e8b149718 | |||
| 2c27d0e1db | |||
| 9f18488752 | |||
| fab6404cca | |||
| c3626cc362 | |||
| d68ecfbc64 | |||
| d1272a6c13 | |||
| 33b3beb342 | |||
| f9384d9df6 | |||
| 156d707377 | |||
| dc1a2e3a0f | |||
| 5d6b8e6253 | |||
| 0cba083305 | |||
| a6be7a4788 | |||
| 2384f7f9b9 | |||
| cd5ef65d3d | |||
| 7bf9ca6201 | |||
| f587b42797 | |||
| 7ae464e172 | |||
| 980c9a20a2 | |||
| 448a38dede | |||
| f12e20b0f3 | |||
| 564d1f37e7 | |||
| 65bfb9f617 | |||
| 4f4ef6259b | |||
| 505263cec6 | |||
| 61dd686fb9 | |||
| c0f7a97a6f | |||
| 9575077045 | |||
| 34a1f7b9dc | |||
| d11aa11f99 | |||
| 0ca06d2507 | |||
| 6693f3a05f | |||
| de252d27b9 | |||
| db0e41a7d3 | |||
| ec460496d8 | |||
| 33e700529e | |||
| d644b7d40a | |||
| f635ba9c75 | |||
| 76b6e3373e | |||
| 0a13cab897 | |||
| d33ec5d225 | |||
| d31c2384df | |||
| c8db463204 | |||
| e8e50ef9bb | |||
| 0faed9309e | |||
| c980346d05 | |||
| 3e3d3f0c2b | |||
| 9eb7444d56 | |||
| 278d1763aa | |||
| f874264e13 | |||
| 5f6e47ece5 | |||
| 0e7cf24dbb | |||
| 2d080a2f51 | |||
| 6179bd51a7 | |||
| 57de4c366a | |||
| e18778e962 | |||
| e4469a0ebf | |||
| 6fedf4c7fc | |||
| 3f6dffa66e | |||
| 1b64fbde3d | |||
| 4f4c3b0d6e | |||
| bd20f0cae1 | |||
| 11c9a5f9d2 | |||
| aeae60146d | |||
| a82c81e8e3 | |||
| c56e9d7be2 | |||
| c8f800a1e5 | |||
| fc6b3424cf | |||
| 35e06c6734 | |||
| 783d233319 | |||
| 6509a2b91a | |||
| 52f8c40b86 | |||
| a86b5ba8f9 | |||
| 1b964545be | |||
| 97aa1708c2 | |||
| d12689f3b5 | |||
| f22e3ee95e | |||
| e890c5e810 | |||
| 078c30a592 | |||
| a847674004 |
@@ -113,13 +113,13 @@ jobs:
|
||||
NIX_HASH="sha256-$(python3 -c "import base64, binascii; print(base64.b64encode(binascii.unhexlify('$NEW_HASH')).decode())")"
|
||||
|
||||
# Update the NixOS configuration
|
||||
sed -i "s|version = \"v[^\"]*\"|version = \"$VERSION\"|" hosts/common/cm-dashboard.nix
|
||||
sed -i "s|sha256 = \"sha256-[^\"]*\"|sha256 = \"$NIX_HASH\"|" hosts/common/cm-dashboard.nix
|
||||
sed -i "s|version = \"v[^\"]*\"|version = \"$VERSION\"|" hosts/services/cm-dashboard.nix
|
||||
sed -i "s|sha256 = \"sha256-[^\"]*\"|sha256 = \"$NIX_HASH\"|" hosts/services/cm-dashboard.nix
|
||||
|
||||
# Commit and push changes
|
||||
git config user.name "Gitea Actions"
|
||||
git config user.email "actions@gitea.cmtec.se"
|
||||
git add hosts/common/cm-dashboard.nix
|
||||
git add hosts/services/cm-dashboard.nix
|
||||
git commit -m "Auto-update cm-dashboard to $VERSION
|
||||
|
||||
- Update version to $VERSION with automated release
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
# Agent Guide
|
||||
|
||||
Agents working in this repo must follow the instructions in `CLAUDE.md`.
|
||||
679
CLAUDE.md
679
CLAUDE.md
@@ -2,276 +2,154 @@
|
||||
|
||||
## Overview
|
||||
|
||||
A high-performance Rust-based TUI dashboard for monitoring CMTEC infrastructure. Built to replace Glance with a custom solution tailored for our specific monitoring needs and ZMQ-based metric collection.
|
||||
A high-performance Rust-based TUI dashboard for monitoring CMTEC infrastructure. Built with ZMQ-based metric collection and individual metrics architecture.
|
||||
|
||||
## Implementation Strategy
|
||||
## Current Features
|
||||
|
||||
### Current Implementation Status
|
||||
### Core Functionality
|
||||
- **Real-time Monitoring**: CPU, RAM, Storage, and Service status
|
||||
- **Service Management**: Start/stop services with user-stopped tracking
|
||||
- **Multi-host Support**: Monitor multiple servers from single dashboard
|
||||
- **NixOS Integration**: System rebuild via SSH + tmux popup
|
||||
- **Backup Monitoring**: Borgbackup status and scheduling
|
||||
|
||||
**System Panel Enhancement - COMPLETED** ✅
|
||||
### User-Stopped Service Tracking
|
||||
- Services stopped via dashboard are marked as "user-stopped"
|
||||
- User-stopped services report Status::OK instead of Warning
|
||||
- Prevents false alerts during intentional maintenance
|
||||
- Persistent storage survives agent restarts
|
||||
- Automatic flag clearing when services are restarted via dashboard
|
||||
|
||||
All system panel features successfully implemented:
|
||||
- ✅ **NixOS Collector**: Created collector for version and active users
|
||||
- ✅ **System Widget**: Unified widget combining NixOS, CPU, RAM, and Storage
|
||||
- ✅ **Build Display**: Shows NixOS build information without codename
|
||||
- ✅ **Active Users**: Displays currently logged in users
|
||||
- ✅ **Tmpfs Monitoring**: Added /tmp usage to RAM section
|
||||
- ✅ **Agent Deployment**: NixOS collector working in production
|
||||
|
||||
**Keyboard Navigation and Service Management - COMPLETED** ✅
|
||||
|
||||
All keyboard navigation and service selection features successfully implemented:
|
||||
- ✅ **Panel Navigation**: Shift+Tab cycles through visible panels only (System → Services → Backup)
|
||||
- ✅ **Service Selection**: Up/Down arrows navigate through parent services with visual cursor
|
||||
- ✅ **Focus Management**: Selection highlighting only visible when Services panel focused
|
||||
- ✅ **Status Preservation**: Service health colors maintained during selection (green/red icons)
|
||||
- ✅ **Smart Panel Switching**: Only cycles through panels with data (backup panel conditional)
|
||||
- ✅ **Scroll Support**: All panels support content scrolling with proper overflow indicators
|
||||
|
||||
**Current Status - October 27, 2025:**
|
||||
- All keyboard navigation features working correctly ✅
|
||||
- Service selection cursor implemented with focus-aware highlighting ✅
|
||||
- Panel scrolling fixed for System, Services, and Backup panels ✅
|
||||
- Build display working: "Build: 25.05.20251004.3bcc93c" ✅
|
||||
- Agent version display working: "Agent: v0.1.17" ✅
|
||||
- Cross-host version comparison implemented ✅
|
||||
- Automated binary release system working ✅
|
||||
- SMART data consolidated into disk collector ✅
|
||||
|
||||
**RESOLVED - Remote Rebuild Functionality:**
|
||||
- ✅ **System Rebuild**: Now uses simple SSH + tmux popup approach
|
||||
- ✅ **Process Isolation**: Rebuild runs independently via SSH, survives agent/dashboard restarts
|
||||
- ✅ **Configuration**: SSH user and rebuild alias configurable in dashboard config
|
||||
- ✅ **Service Control**: Works correctly for start/stop/restart of services
|
||||
|
||||
**Solution Implemented:**
|
||||
- Replaced complex SystemRebuild command infrastructure with direct tmux popup
|
||||
- Uses `tmux display-popup "ssh -tt {user}@{hostname} 'bash -ic {alias}'"`
|
||||
- Configurable SSH user and rebuild alias in dashboard config
|
||||
- Eliminates all agent crashes during rebuilds
|
||||
- Simple, reliable, and follows standard tmux interface patterns
|
||||
|
||||
**Current Layout:**
|
||||
```
|
||||
NixOS:
|
||||
Build: 25.05.20251004.3bcc93c
|
||||
Agent: v0.1.17 # Shows agent version from Cargo.toml
|
||||
Active users: cm, simon
|
||||
CPU:
|
||||
● Load: 0.02 0.31 0.86 • 3000MHz
|
||||
RAM:
|
||||
● Usage: 33% 2.6GB/7.6GB
|
||||
● /tmp: 0% 0B/2.0GB
|
||||
Storage:
|
||||
● root (Single):
|
||||
├─ ● nvme0n1 W: 1%
|
||||
└─ ● 18% 167.4GB/928.2GB
|
||||
### Custom Service Logs
|
||||
- Configure service-specific log file paths per host in dashboard config
|
||||
- Press `L` on any service to view custom log files via `tail -f`
|
||||
- Configuration format in dashboard config:
|
||||
```toml
|
||||
[service_logs]
|
||||
hostname1 = [
|
||||
{ service_name = "nginx", log_file_path = "/var/log/nginx/access.log" },
|
||||
{ service_name = "app", log_file_path = "/var/log/myapp/app.log" }
|
||||
]
|
||||
hostname2 = [
|
||||
{ service_name = "database", log_file_path = "/var/log/postgres/postgres.log" }
|
||||
]
|
||||
```
|
||||
|
||||
**System panel layout fully implemented with blue tree symbols ✅**
|
||||
**Tree symbols now use consistent blue theming across all panels ✅**
|
||||
**Overflow handling restored for all widgets ("... and X more") ✅**
|
||||
**Agent version display working correctly ✅**
|
||||
**Cross-host version comparison logging warnings ✅**
|
||||
**Backup panel visibility fixed - only shows when meaningful data exists ✅**
|
||||
**SSH-based rebuild system fully implemented and working ✅**
|
||||
### Service Management
|
||||
- **Direct Control**: Arrow keys (↑↓) or vim keys (j/k) navigate services
|
||||
- **Service Actions**:
|
||||
- `s` - Start service (sends UserStart command)
|
||||
- `S` - Stop service (sends UserStop command)
|
||||
- `J` - Show service logs (journalctl in tmux popup)
|
||||
- `L` - Show custom log files (tail -f custom paths in tmux popup)
|
||||
- `R` - Rebuild current host
|
||||
- **Visual Status**: Green ● (active), Yellow ◐ (inactive), Red ◯ (failed)
|
||||
- **Transitional Icons**: Blue arrows during operations
|
||||
|
||||
### Current Keyboard Navigation Implementation
|
||||
|
||||
**Navigation Controls:**
|
||||
- **Tab**: Switch between hosts (cmbox, srv01, srv02, steambox, etc.)
|
||||
- **Shift+Tab**: Cycle through visible panels (System → Services → Backup → System)
|
||||
- **Up/Down (System/Backup)**: Scroll through panel content
|
||||
- **Up/Down (Services)**: Move service selection cursor between parent services
|
||||
### Navigation
|
||||
- **Tab**: Switch between hosts
|
||||
- **↑↓ or j/k**: Select services
|
||||
- **s**: Start selected service (UserStart)
|
||||
- **S**: Stop selected service (UserStop)
|
||||
- **J**: Show service logs (journalctl)
|
||||
- **L**: Show custom log files
|
||||
- **R**: Rebuild current host
|
||||
- **B**: Run backup on current host
|
||||
- **q**: Quit dashboard
|
||||
|
||||
**Panel-Specific Features:**
|
||||
- **System Panel**: Scrollable content with CPU, RAM, Storage details
|
||||
- **Services Panel**: Service selection cursor for parent services only (docker, nginx, postgresql, etc.)
|
||||
- **Backup Panel**: Scrollable repository list with proper overflow handling
|
||||
## Core Architecture Principles
|
||||
|
||||
**Visual Feedback:**
|
||||
- **Focused Panel**: Blue border and title highlighting
|
||||
- **Service Selection**: Blue background with preserved status icon colors (green ● for active, red ● for failed)
|
||||
- **Focus-Aware Selection**: Selection highlighting only visible when Services panel focused
|
||||
- **Dynamic Statusbar**: Context-aware shortcuts based on focused panel
|
||||
### Structured Data Architecture (Planned Migration)
|
||||
Current system uses string-based metrics with complex parsing. Planning migration to structured JSON data to eliminate fragile string manipulation.
|
||||
|
||||
### Remote Command Execution - WORKING ✅
|
||||
**Current (String Metrics):**
|
||||
- Agent sends individual metrics with string names like `disk_nvme0n1_temperature`
|
||||
- Dashboard parses metric names with underscore counting and string splitting
|
||||
- Complex and error-prone metric filtering and extraction logic
|
||||
|
||||
**All Issues Resolved (as of 2025-10-24):**
|
||||
- ✅ **ZMQ Command Protocol**: Extended with ServiceControl and SystemRebuild variants
|
||||
- ✅ **Agent Handlers**: systemctl and nixos-rebuild execution with maintenance mode
|
||||
- ✅ **Dashboard Integration**: Keyboard shortcuts execute commands
|
||||
- ✅ **Service Control**: Fixed toggle logic - replaced with separate 's' (start) and 'S' (stop)
|
||||
- ✅ **System Rebuild**: Fixed permission issues and sandboxing problems
|
||||
- ✅ **Git Clone Approach**: Implemented for nixos-rebuild to avoid directory permissions
|
||||
- ✅ **Visual Feedback**: Directional arrows for service status (↑ starting, ↓ stopping, ↻ restarting)
|
||||
|
||||
### Terminal Popup for Real-time Output - IMPLEMENTED ✅
|
||||
|
||||
**Status (as of 2025-10-26):**
|
||||
- ✅ **Terminal Popup UI**: 80% screen coverage with terminal styling and color-coded output
|
||||
- ✅ **ZMQ Streaming Protocol**: CommandOutputMessage for real-time output transmission
|
||||
- ✅ **Keyboard Controls**: ESC/Q to close, ↑↓ to scroll, manual close (no auto-close)
|
||||
- ✅ **Real-time Display**: Live streaming of command output as it happens
|
||||
- ✅ **Version-based Agent Reporting**: Shows "Agent: v0.1.13" instead of nix store hash
|
||||
|
||||
**Current Implementation Issues:**
|
||||
- ❌ **Agent Process Crashes**: Agent dies during nixos-rebuild execution
|
||||
- ❌ **Inconsistent Output**: Different outputs each time 'R' is pressed
|
||||
- ❌ **Limited Output Visibility**: Not capturing all nixos-rebuild progress
|
||||
|
||||
**PLANNED SOLUTION - Systemd Service Approach:**
|
||||
|
||||
**Problem**: Direct nixos-rebuild execution in agent causes process crashes and inconsistent output.
|
||||
|
||||
**Solution**: Create dedicated systemd service for rebuild operations.
|
||||
|
||||
**Implementation Plan:**
|
||||
1. **NixOS Systemd Service**:
|
||||
```nix
|
||||
systemd.services.cm-rebuild = {
|
||||
description = "CM Dashboard NixOS Rebuild";
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
ExecStart = "${pkgs.nixos-rebuild}/bin/nixos-rebuild switch --flake . --option sandbox false";
|
||||
WorkingDirectory = "/var/lib/cm-dashboard/nixos-config";
|
||||
User = "root";
|
||||
StandardOutput = "journal";
|
||||
StandardError = "journal";
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
2. **Agent Modification**:
|
||||
- Replace direct nixos-rebuild execution with: `systemctl start cm-rebuild`
|
||||
- Stream output via: `journalctl -u cm-rebuild -f --no-pager`
|
||||
- Monitor service status for completion detection
|
||||
|
||||
3. **Benefits**:
|
||||
- **Process Isolation**: Service runs independently, won't crash agent
|
||||
- **Consistent Output**: Always same deterministic rebuild process
|
||||
- **Proper Logging**: systemd journal handles all output management
|
||||
- **Resource Management**: systemd manages cleanup and resource limits
|
||||
- **Status Tracking**: Can query service status (running/failed/success)
|
||||
|
||||
**Next Priority**: Implement systemd service approach for reliable rebuild operations.
|
||||
|
||||
**Keyboard Controls Status:**
|
||||
- **Services Panel**:
|
||||
- R (restart) ✅ Working
|
||||
- s (start) ✅ Working
|
||||
- S (stop) ✅ Working
|
||||
- **System Panel**: R (nixos-rebuild) ✅ Working with --option sandbox false
|
||||
- **Backup Panel**: B (trigger backup) ❓ Not implemented
|
||||
|
||||
**Visual Feedback Implementation - IN PROGRESS:**
|
||||
|
||||
Context-appropriate progress indicators for each panel:
|
||||
|
||||
**Services Panel** (Service status transitions):
|
||||
```
|
||||
● nginx active → ⏳ nginx restarting → ● nginx active
|
||||
● docker active → ⏳ docker stopping → ● docker inactive
|
||||
**Target (Structured Data):**
|
||||
```json
|
||||
{
|
||||
"hostname": "cmbox",
|
||||
"agent_version": "v0.1.130",
|
||||
"timestamp": 1763926877,
|
||||
"system": {
|
||||
"cpu": {
|
||||
"load_1min": 3.50,
|
||||
"load_5min": 3.57,
|
||||
"load_15min": 3.58,
|
||||
"frequency_mhz": 1500,
|
||||
"temperature_celsius": 45.2
|
||||
},
|
||||
"memory": {
|
||||
"usage_percent": 25.0,
|
||||
"total_gb": 23.3,
|
||||
"used_gb": 5.9,
|
||||
"swap_total_gb": 10.7,
|
||||
"swap_used_gb": 0.99,
|
||||
"tmpfs": [
|
||||
{"mount": "/tmp", "usage_percent": 15.0, "used_gb": 0.3, "total_gb": 2.0}
|
||||
]
|
||||
},
|
||||
"storage": {
|
||||
"drives": [
|
||||
{
|
||||
"name": "nvme0n1",
|
||||
"health": "PASSED",
|
||||
"temperature_celsius": 29.0,
|
||||
"wear_percent": 1.0,
|
||||
"filesystems": [
|
||||
{"mount": "/", "usage_percent": 24.0, "used_gb": 224.9, "total_gb": 928.2}
|
||||
]
|
||||
}
|
||||
],
|
||||
"pools": [
|
||||
{
|
||||
"name": "srv_media",
|
||||
"mount": "/srv/media",
|
||||
"type": "mergerfs",
|
||||
"health": "healthy",
|
||||
"usage_percent": 63.0,
|
||||
"used_gb": 2355.2,
|
||||
"total_gb": 3686.4,
|
||||
"data_drives": [
|
||||
{"name": "sdb", "temperature_celsius": 24.0}
|
||||
],
|
||||
"parity_drives": [
|
||||
{"name": "sdc", "temperature_celsius": 24.0}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"services": [
|
||||
{"name": "sshd", "status": "active", "memory_mb": 4.5, "disk_gb": 0.0}
|
||||
],
|
||||
"backup": {
|
||||
"status": "completed",
|
||||
"last_run": 1763920000,
|
||||
"next_scheduled": 1764006400,
|
||||
"total_size_gb": 150.5,
|
||||
"repository_health": "ok"
|
||||
}
|
||||
}
|
||||
```
|
||||
- Agent sends structured JSON over ZMQ
|
||||
- Dashboard accesses data directly: `data.system.storage.drives[0].temperature_celsius`
|
||||
- Type safety eliminates all parsing bugs
|
||||
|
||||
**System Panel** (Build progress in NixOS section):
|
||||
```
|
||||
NixOS:
|
||||
Build: 25.05.20251004.3bcc93c → Build: [████████████ ] 65%
|
||||
Active users: cm, simon Active users: cm, simon
|
||||
```
|
||||
|
||||
**Backup Panel** (OnGoing status with progress):
|
||||
```
|
||||
Latest backup: → Latest backup:
|
||||
● 2024-10-23 14:32:15 ● OnGoing
|
||||
└─ Duration: 1.3m └─ [██████ ] 60%
|
||||
```
|
||||
|
||||
**Critical Configuration Hash Fix - HIGH PRIORITY:**
|
||||
|
||||
**Problem:** Configuration hash currently shows git commit hash instead of actual deployed system hash.
|
||||
|
||||
**Current (incorrect):**
|
||||
- Shows git hash: `db11f82` (source repository commit)
|
||||
- Not accurate - doesn't reflect what's actually deployed
|
||||
|
||||
**Target (correct):**
|
||||
- Show nix store hash: `d8ivwiar` (first 8 chars from deployed system)
|
||||
- Source: `/nix/store/d8ivwiarhwhgqzskj6q2482r58z46qjf-nixos-system-cmbox-25.05.20251004.3bcc93c`
|
||||
- Pattern: Extract hash from `/nix/store/HASH-nixos-system-HOSTNAME-VERSION`
|
||||
|
||||
**Benefits:**
|
||||
1. **Deployment Verification:** Confirms rebuild actually succeeded
|
||||
2. **Accurate Status:** Shows what's truly running, not just source
|
||||
3. **Rebuild Completion Detection:** Hash change = rebuild completed
|
||||
4. **Rollback Tracking:** Each deployment has unique identifier
|
||||
|
||||
**Implementation Required:**
|
||||
1. Agent extracts nix store hash from `ls -la /run/current-system`
|
||||
2. Reports this as `system_config_hash` metric instead of git hash
|
||||
3. Dashboard displays first 8 characters: `Config: d8ivwiar`
|
||||
|
||||
**Next Session Priority Tasks:**
|
||||
|
||||
**Remaining Features:**
|
||||
1. **Fix Configuration Hash Display (CRITICAL)**:
|
||||
- Use nix store hash instead of git commit hash
|
||||
- Extract from `/run/current-system` -> `/nix/store/HASH-nixos-system-*`
|
||||
- Enables proper rebuild completion detection
|
||||
|
||||
2. **Command Response Protocol**:
|
||||
- Agent sends command completion/failure back to dashboard via ZMQ
|
||||
- Dashboard updates UI status from ⏳ to ● when commands complete
|
||||
- Clear success/failure status after timeout
|
||||
|
||||
3. **Backup Panel Features**:
|
||||
- Implement backup trigger functionality (B key)
|
||||
- Complete visual feedback for backup operations
|
||||
- Add backup progress indicators
|
||||
|
||||
**Enhancement Tasks:**
|
||||
- Add confirmation dialogs for destructive actions (stop/restart/rebuild)
|
||||
- Implement command history/logging
|
||||
- Add keyboard shortcuts help overlay
|
||||
|
||||
**Future Enhanced Navigation:**
|
||||
- Add Page Up/Down for faster scrolling through long service lists
|
||||
- Implement search/filter functionality for services
|
||||
- Add jump-to-service shortcuts (first letter navigation)
|
||||
|
||||
**Future Advanced Features:**
|
||||
- Service dependency visualization
|
||||
- Historical service status tracking
|
||||
- Real-time log viewing integration
|
||||
|
||||
## Core Architecture Principles - CRITICAL
|
||||
|
||||
### Individual Metrics Philosophy
|
||||
|
||||
**NEW ARCHITECTURE**: Agent collects individual metrics, dashboard composes widgets from those metrics.
|
||||
|
||||
### Maintenance Mode
|
||||
|
||||
**Purpose:**
|
||||
|
||||
- Suppress email notifications during planned maintenance or backups
|
||||
- Prevents false alerts when services are intentionally stopped
|
||||
|
||||
**Implementation:**
|
||||
|
||||
- Agent checks for `/tmp/cm-maintenance` file before sending notifications
|
||||
- File presence suppresses all email notifications while continuing monitoring
|
||||
- Dashboard continues to show real status, only notifications are blocked
|
||||
|
||||
**Usage:**
|
||||
|
||||
Usage:
|
||||
```bash
|
||||
# Enable maintenance mode
|
||||
touch /tmp/cm-maintenance
|
||||
|
||||
# Run maintenance tasks (backups, service restarts, etc.)
|
||||
# Run maintenance tasks
|
||||
systemctl stop service
|
||||
# ... maintenance work ...
|
||||
systemctl start service
|
||||
@@ -280,61 +158,208 @@ systemctl start service
|
||||
rm /tmp/cm-maintenance
|
||||
```
|
||||
|
||||
**NixOS Integration:**
|
||||
## Development and Deployment Architecture
|
||||
|
||||
- Borgbackup script automatically creates/removes maintenance file
|
||||
- Automatic cleanup via trap ensures maintenance mode doesn't stick
|
||||
- All cinfiguration are shall be done from nixos config
|
||||
### Development Path
|
||||
- **Location:** `~/projects/cm-dashboard`
|
||||
- **Purpose:** Development workflow only - for committing new code
|
||||
- **Access:** Only for developers to commit changes
|
||||
|
||||
**ARCHITECTURE ENFORCEMENT**:
|
||||
### Deployment Path
|
||||
- **Location:** `/var/lib/cm-dashboard/nixos-config`
|
||||
- **Purpose:** Production deployment only - agent clones/pulls from git
|
||||
- **Workflow:** git pull → `/var/lib/cm-dashboard/nixos-config` → nixos-rebuild
|
||||
|
||||
- **ZERO legacy code reuse** - Fresh implementation following ARCHITECT.md exactly
|
||||
- **Individual metrics only** - NO grouped metric structures
|
||||
- **Reference-only legacy** - Study old functionality, implement new architecture
|
||||
- **Clean slate mindset** - Build as if legacy codebase never existed
|
||||
### Git Flow
|
||||
```
|
||||
Development: ~/projects/cm-dashboard → git commit → git push
|
||||
Deployment: git pull → /var/lib/cm-dashboard/nixos-config → rebuild
|
||||
```
|
||||
|
||||
**Implementation Rules**:
|
||||
## Automated Binary Release System
|
||||
|
||||
1. **Individual Metrics**: Each metric is collected, transmitted, and stored individually
|
||||
2. **Agent Status Authority**: Agent calculates status for each metric using thresholds
|
||||
3. **Dashboard Composition**: Dashboard widgets subscribe to specific metrics by name
|
||||
4. **Status Aggregation**: Dashboard aggregates individual metric statuses for widget status
|
||||
**Testing & Building**:
|
||||
CM Dashboard uses automated binary releases instead of source builds.
|
||||
|
||||
- **Workspace builds**: `cargo build --workspace` for all testing
|
||||
- **Clean compilation**: Remove `target/` between architecture changes
|
||||
- **ZMQ testing**: Test agent-dashboard communication independently
|
||||
- **Widget testing**: Verify UI layout matches legacy appearance exactly
|
||||
### Creating New Releases
|
||||
```bash
|
||||
cd ~/projects/cm-dashboard
|
||||
git tag v0.1.X
|
||||
git push origin v0.1.X
|
||||
```
|
||||
|
||||
**NEVER in New Implementation**:
|
||||
This automatically:
|
||||
- Builds static binaries with `RUSTFLAGS="-C target-feature=+crt-static"`
|
||||
- Creates GitHub-style release with tarball
|
||||
- Uploads binaries via Gitea API
|
||||
|
||||
- Copy/paste ANY code from legacy backup
|
||||
- Calculate status in dashboard widgets
|
||||
- Hardcode metric names in widgets (use const arrays)
|
||||
### NixOS Configuration Updates
|
||||
Edit `~/projects/nixosbox/hosts/services/cm-dashboard.nix`:
|
||||
|
||||
# Important Communication Guidelines
|
||||
```nix
|
||||
version = "v0.1.X";
|
||||
src = pkgs.fetchurl {
|
||||
url = "https://gitea.cmtec.se/cm/cm-dashboard/releases/download/${version}/cm-dashboard-linux-x86_64.tar.gz";
|
||||
sha256 = "sha256-NEW_HASH_HERE";
|
||||
};
|
||||
```
|
||||
|
||||
NEVER write that you have "successfully implemented" something or generate extensive summary text without first verifying with the user that the implementation is correct. This wastes tokens. Keep responses concise.
|
||||
### Get Release Hash
|
||||
```bash
|
||||
cd ~/projects/nixosbox
|
||||
nix-build --no-out-link -E 'with import <nixpkgs> {}; fetchurl {
|
||||
url = "https://gitea.cmtec.se/cm/cm-dashboard/releases/download/v0.1.X/cm-dashboard-linux-x86_64.tar.gz";
|
||||
sha256 = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
|
||||
}' 2>&1 | grep "got:"
|
||||
```
|
||||
|
||||
NEVER implement code without first getting explicit user agreement on the approach. Always ask for confirmation before proceeding with implementation.
|
||||
### Building
|
||||
|
||||
**Testing & Building:**
|
||||
- **Workspace builds**: `nix-shell -p openssl pkg-config --run "cargo build --workspace"`
|
||||
- **Clean compilation**: Remove `target/` between major changes
|
||||
|
||||
## Enhanced Storage Pool Visualization
|
||||
|
||||
### Auto-Discovery Architecture
|
||||
|
||||
The dashboard uses automatic storage discovery to eliminate manual configuration complexity while providing intelligent storage pool grouping.
|
||||
|
||||
### Discovery Process
|
||||
|
||||
**At Agent Startup:**
|
||||
1. Parse `/proc/mounts` to identify all mounted filesystems
|
||||
2. Detect MergerFS pools by analyzing `fuse.mergerfs` mount sources
|
||||
3. Identify member disks and potential parity relationships via heuristics
|
||||
4. Store discovered storage topology for continuous monitoring
|
||||
5. Generate pool-aware metrics with hierarchical relationships
|
||||
|
||||
**Continuous Monitoring:**
|
||||
- Use stored discovery data for efficient metric collection
|
||||
- Monitor individual drives for SMART data, temperature, wear
|
||||
- Calculate pool-level health based on member drive status
|
||||
- Generate enhanced metrics for dashboard visualization
|
||||
|
||||
### Supported Storage Types
|
||||
|
||||
**Single Disks:**
|
||||
- ext4, xfs, btrfs mounted directly
|
||||
- Individual drive monitoring with SMART data
|
||||
- Traditional single-disk display for root, boot, etc.
|
||||
|
||||
**MergerFS Pools:**
|
||||
- Auto-detect from `/proc/mounts` fuse.mergerfs entries
|
||||
- Parse source paths to identify member disks (e.g., "/mnt/disk1:/mnt/disk2")
|
||||
- Heuristic parity disk detection (sequential device names, "parity" in path)
|
||||
- Pool health calculation (healthy/degraded/critical)
|
||||
- Hierarchical tree display with data/parity disk grouping
|
||||
|
||||
**Future Extensions Ready:**
|
||||
- RAID arrays via `/proc/mdstat` parsing
|
||||
- ZFS pools via `zpool status` integration
|
||||
- LVM logical volumes via `lvs` discovery
|
||||
|
||||
### Configuration
|
||||
|
||||
```toml
|
||||
[collectors.disk]
|
||||
enabled = true
|
||||
auto_discover = true # Default: true
|
||||
# Optional exclusions for special filesystems
|
||||
exclude_mount_points = ["/tmp", "/proc", "/sys", "/dev"]
|
||||
exclude_fs_types = ["tmpfs", "devtmpfs", "sysfs", "proc"]
|
||||
```
|
||||
|
||||
### Display Format
|
||||
|
||||
```
|
||||
Storage:
|
||||
● /srv/media (mergerfs (2+1)):
|
||||
├─ Pool Status: ● Healthy (3 drives)
|
||||
├─ Total: ● 63% 2355.2GB/3686.4GB
|
||||
├─ Data Disks:
|
||||
│ ├─ ● sdb T: 24°C
|
||||
│ └─ ● sdd T: 27°C
|
||||
└─ Parity: ● sdc T: 24°C
|
||||
● /:
|
||||
├─ ● nvme0n1 W: 13%
|
||||
└─ ● 7% 14.5GB/218.5GB
|
||||
```
|
||||
|
||||
### Implementation Benefits
|
||||
|
||||
- **Zero Configuration**: No manual pool definitions required
|
||||
- **Always Accurate**: Reflects actual system state automatically
|
||||
- **Scales Automatically**: Handles any number of pools without config changes
|
||||
- **Backwards Compatible**: Single disks continue working unchanged
|
||||
- **Future Ready**: Easy extension for additional storage technologies
|
||||
|
||||
### Current Status (v0.1.100)
|
||||
|
||||
**✅ Completed:**
|
||||
- Auto-discovery system implemented and deployed
|
||||
- `/proc/mounts` parsing with smart heuristics for parity detection
|
||||
- Storage topology stored at agent startup for efficient monitoring
|
||||
- Universal zero-configuration for all hosts (cmbox, steambox, simonbox, srv01, srv02, srv03)
|
||||
- Enhanced pool health calculation (healthy/degraded/critical)
|
||||
- Hierarchical tree visualization with data/parity disk separation
|
||||
|
||||
**🔄 In Progress - Complete Disk Collector Rewrite:**
|
||||
|
||||
The current disk collector has grown complex with mixed legacy/auto-discovery approaches. Planning complete rewrite with clean, simple workflow supporting both physical drives and mergerfs pools.
|
||||
|
||||
**New Clean Architecture:**
|
||||
|
||||
**Discovery Workflow:**
|
||||
1. **`lsblk`** to detect all mount points and backing devices
|
||||
2. **`df`** to get filesystem usage for each mount point
|
||||
3. **Group by physical drive** (nvme0n1, sda, etc.)
|
||||
4. **Parse `/proc/mounts`** for mergerfs pools
|
||||
5. **Generate unified metrics** for both storage types
|
||||
|
||||
**Physical Drive Display:**
|
||||
```
|
||||
● nvme0n1:
|
||||
├─ ● Drive: T: 35°C W: 1%
|
||||
├─ ● Total: 23% 218.0GB/928.2GB
|
||||
├─ ● /boot: 11% 0.1GB/1.0GB
|
||||
└─ ● /: 23% 214.9GB/928.2GB
|
||||
```
|
||||
|
||||
**MergerFS Pool Display:**
|
||||
```
|
||||
● /srv/media (mergerfs):
|
||||
├─ ● Pool: 63% 2355.2GB/3686.4GB
|
||||
├─ Data Disks:
|
||||
│ ├─ ● sdb T: 24°C
|
||||
│ └─ ● sdd T: 27°C
|
||||
└─ ● sdc T: 24°C (parity)
|
||||
```
|
||||
|
||||
**Implementation Benefits:**
|
||||
- **Pure auto-discovery**: No configuration needed
|
||||
- **Clean code paths**: Single workflow for all storage types
|
||||
- **Consistent display**: Status icons on every line, no redundant text
|
||||
- **Simple pipeline**: lsblk → df → group → metrics
|
||||
- **Support for both**: Physical drives and mergerfs pools
|
||||
|
||||
## Important Communication Guidelines
|
||||
|
||||
Keep responses concise and focused. Avoid extensive implementation summaries unless requested.
|
||||
|
||||
## Commit Message Guidelines
|
||||
|
||||
**NEVER mention:**
|
||||
|
||||
- Claude or any AI assistant names
|
||||
- Automation or AI-generated content
|
||||
- Any reference to automated code generation
|
||||
|
||||
**ALWAYS:**
|
||||
|
||||
- Focus purely on technical changes and their purpose
|
||||
- Use standard software development commit message format
|
||||
- Describe what was changed and why, not how it was created
|
||||
- Write from the perspective of a human developer
|
||||
|
||||
**Examples:**
|
||||
|
||||
- ❌ "Generated with Claude Code"
|
||||
- ❌ "AI-assisted implementation"
|
||||
- ❌ "Automated refactoring"
|
||||
@@ -342,83 +367,43 @@ NEVER implement code without first getting explicit user agreement on the approa
|
||||
- ✅ "Restructure storage widget with improved layout"
|
||||
- ✅ "Update CPU thresholds to production values"
|
||||
|
||||
## Development and Deployment Architecture
|
||||
## Planned Architecture Migration
|
||||
|
||||
**CRITICAL:** Development and deployment paths are completely separate:
|
||||
### Phase 1: Structured Data Types (Shared Crate)
|
||||
- Create Rust structs matching target JSON structure
|
||||
- Replace `Metric` enum with typed data structures
|
||||
- Add serde serialization/deserialization
|
||||
|
||||
### Development Path
|
||||
- **Location:** `~/projects/nixosbox`
|
||||
- **Purpose:** Development workflow only - for committing new cm-dashboard code
|
||||
- **Access:** Only for developers to commit changes
|
||||
- **Code Access:** Running cm-dashboard code shall NEVER access this path
|
||||
### Phase 2: Agent Refactor
|
||||
- Update collectors to return typed structs instead of `Vec<Metric>`
|
||||
- Remove string metric name generation
|
||||
- Send structured JSON over ZMQ
|
||||
|
||||
### Deployment Path
|
||||
- **Location:** `/var/lib/cm-dashboard/nixos-config`
|
||||
- **Purpose:** Production deployment only - agent clones/pulls from git
|
||||
- **Access:** Only cm-dashboard agent for deployment operations
|
||||
- **Workflow:** git pull → `/var/lib/cm-dashboard/nixos-config` → nixos-rebuild
|
||||
### Phase 3: Dashboard Refactor
|
||||
- Replace metric parsing logic with direct field access
|
||||
- Remove `extract_pool_name()`, `extract_drive_name()`, underscore counting
|
||||
- Widgets access `data.system.storage.drives[0].temperature_celsius`
|
||||
|
||||
### Git Flow
|
||||
```
|
||||
Development: ~/projects/nixosbox → git commit → git push
|
||||
Deployment: git pull → /var/lib/cm-dashboard/nixos-config → rebuild
|
||||
```
|
||||
### Phase 4: Migration & Cleanup
|
||||
- Support both formats during transition
|
||||
- Gradual rollout with backward compatibility
|
||||
- Remove legacy string metric system
|
||||
|
||||
## Automated Binary Release System
|
||||
## Implementation Rules
|
||||
|
||||
**IMPLEMENTED:** cm-dashboard now uses automated binary releases instead of source builds.
|
||||
1. **Agent Status Authority**: Agent calculates status for each metric using thresholds
|
||||
2. **Dashboard Composition**: Dashboard widgets subscribe to specific metrics by name
|
||||
3. **Status Aggregation**: Dashboard aggregates individual metric statuses for widget status
|
||||
|
||||
### Release Workflow
|
||||
**NEVER:**
|
||||
- Copy/paste ANY code from legacy implementations
|
||||
- Calculate status in dashboard widgets
|
||||
- Hardcode metric names in widgets (use const arrays)
|
||||
- Create files unless absolutely necessary for achieving goals
|
||||
- Create documentation files unless explicitly requested
|
||||
|
||||
1. **Automated Release Creation**
|
||||
- Gitea Actions workflow builds static binaries on tag push
|
||||
- Creates release with `cm-dashboard-linux-x86_64.tar.gz` tarball
|
||||
- No manual intervention required for binary generation
|
||||
|
||||
2. **Creating New Releases**
|
||||
```bash
|
||||
cd ~/projects/cm-dashboard
|
||||
git tag v0.1.X
|
||||
git push origin v0.1.X
|
||||
```
|
||||
|
||||
This automatically:
|
||||
- Builds static binaries with `RUSTFLAGS="-C target-feature=+crt-static"`
|
||||
- Creates GitHub-style release with tarball
|
||||
- Uploads binaries via Gitea API
|
||||
|
||||
3. **NixOS Configuration Updates**
|
||||
Edit `~/projects/nixosbox/hosts/common/cm-dashboard.nix`:
|
||||
|
||||
```nix
|
||||
version = "v0.1.X";
|
||||
src = pkgs.fetchurl {
|
||||
url = "https://gitea.cmtec.se/cm/cm-dashboard/releases/download/${version}/cm-dashboard-linux-x86_64.tar.gz";
|
||||
sha256 = "sha256-NEW_HASH_HERE";
|
||||
};
|
||||
```
|
||||
|
||||
4. **Get Release Hash**
|
||||
```bash
|
||||
cd ~/projects/nixosbox
|
||||
nix-build --no-out-link -E 'with import <nixpkgs> {}; fetchurl {
|
||||
url = "https://gitea.cmtec.se/cm/cm-dashboard/releases/download/v0.1.X/cm-dashboard-linux-x86_64.tar.gz";
|
||||
sha256 = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
|
||||
}' 2>&1 | grep "got:"
|
||||
```
|
||||
|
||||
5. **Commit and Deploy**
|
||||
```bash
|
||||
cd ~/projects/nixosbox
|
||||
git add hosts/common/cm-dashboard.nix
|
||||
git commit -m "Update cm-dashboard to v0.1.X with static binaries"
|
||||
git push
|
||||
```
|
||||
|
||||
### Benefits
|
||||
|
||||
- **No compilation overhead** on each host
|
||||
- **Consistent static binaries** across all hosts
|
||||
- **Faster deployments** - download vs compile
|
||||
- **No library dependency issues** - static linking
|
||||
- **Automated pipeline** - tag push triggers everything
|
||||
**ALWAYS:**
|
||||
- Prefer editing existing files to creating new ones
|
||||
- Follow existing code conventions and patterns
|
||||
- Use existing libraries and utilities
|
||||
- Follow security best practices
|
||||
237
Cargo.lock
generated
237
Cargo.lock
generated
@@ -17,9 +17,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aho-corasick"
|
||||
version = "1.1.3"
|
||||
version = "1.1.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8e60d3430d3a69478ad0993f19238d2df97c507009a52b3c10addcd7f6bcb916"
|
||||
checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301"
|
||||
dependencies = [
|
||||
"memchr",
|
||||
]
|
||||
@@ -71,22 +71,22 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "anstyle-query"
|
||||
version = "1.1.4"
|
||||
version = "1.1.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9e231f6134f61b71076a3eab506c379d4f36122f2af15a9ff04415ea4c3339e2"
|
||||
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
|
||||
dependencies = [
|
||||
"windows-sys 0.60.2",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "anstyle-wincon"
|
||||
version = "3.0.10"
|
||||
version = "3.0.11"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3e0633414522a32ffaac8ac6cc8f748e090c5717661fddeea04219e2344f5f2a"
|
||||
checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
|
||||
dependencies = [
|
||||
"anstyle",
|
||||
"once_cell_polyfill",
|
||||
"windows-sys 0.60.2",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -95,6 +95,15 @@ version = "1.0.100"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a23eb6b1614318a8071c9b2521f36b424b2c83db5eb3a0fead4a6c0809af6e61"
|
||||
|
||||
[[package]]
|
||||
name = "ar_archive_writer"
|
||||
version = "0.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f0c269894b6fe5e9d7ada0cf69b5bf847ff35bc25fc271f08e1d080fce80339a"
|
||||
dependencies = [
|
||||
"object",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "async-trait"
|
||||
version = "0.1.89"
|
||||
@@ -144,9 +153,9 @@ checksum = "46c5e41b57b8bba42a04676d81cb89e9ee8e859a1a66f80a5a72e1cb76b34d43"
|
||||
|
||||
[[package]]
|
||||
name = "bytes"
|
||||
version = "1.10.1"
|
||||
version = "1.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d71b6127be86fdcfddb610f7182ac57211d4b18a3e9c82eb2d17662f2227ad6a"
|
||||
checksum = "b35204fbdc0b3f4446b89fc1ac2cf84a8a68971995d0bf2e925ec7cd960f9cb3"
|
||||
|
||||
[[package]]
|
||||
name = "cassowary"
|
||||
@@ -156,9 +165,9 @@ checksum = "df8670b8c7b9dae1793364eafadf7239c40d669904660c5960d74cfd80b46a53"
|
||||
|
||||
[[package]]
|
||||
name = "cc"
|
||||
version = "1.2.41"
|
||||
version = "1.2.46"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ac9fe6cdbb24b6ade63616c0a0688e45bb56732262c158df3c0c4bea4ca47cb7"
|
||||
checksum = "b97463e1064cb1b1c1384ad0a0b9c8abd0988e2a91f52606c80ef14aadb63e36"
|
||||
dependencies = [
|
||||
"find-msvc-tools",
|
||||
"jobserver",
|
||||
@@ -230,9 +239,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "clap"
|
||||
version = "4.5.49"
|
||||
version = "4.5.52"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f4512b90fa68d3a9932cea5184017c5d200f5921df706d45e853537dea51508f"
|
||||
checksum = "aa8120877db0e5c011242f96806ce3c94e0737ab8108532a76a3300a01db2ab8"
|
||||
dependencies = [
|
||||
"clap_builder",
|
||||
"clap_derive",
|
||||
@@ -240,9 +249,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "clap_builder"
|
||||
version = "4.5.49"
|
||||
version = "4.5.52"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0025e98baa12e766c67ba13ff4695a887a1eba19569aad00a472546795bd6730"
|
||||
checksum = "02576b399397b659c26064fbc92a75fede9d18ffd5f80ca1cd74ddab167016e1"
|
||||
dependencies = [
|
||||
"anstream",
|
||||
"anstyle",
|
||||
@@ -270,7 +279,7 @@ checksum = "a1d728cc89cf3aee9ff92b05e62b19ee65a02b5702cff7d5a377e32c6ae29d8d"
|
||||
|
||||
[[package]]
|
||||
name = "cm-dashboard"
|
||||
version = "0.1.28"
|
||||
version = "0.1.130"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"chrono",
|
||||
@@ -286,12 +295,13 @@ dependencies = [
|
||||
"toml",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
"wake-on-lan",
|
||||
"zmq",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "cm-dashboard-agent"
|
||||
version = "0.1.28"
|
||||
version = "0.1.130"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -314,7 +324,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cm-dashboard-shared"
|
||||
version = "0.1.28"
|
||||
version = "0.1.130"
|
||||
dependencies = [
|
||||
"chrono",
|
||||
"serde",
|
||||
@@ -502,9 +512,9 @@ checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be"
|
||||
|
||||
[[package]]
|
||||
name = "find-msvc-tools"
|
||||
version = "0.1.4"
|
||||
version = "0.1.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "52051878f80a721bb68ebfbc930e07b65ba72f2da88968ea5c06fd6ca3d3a127"
|
||||
checksum = "3a3076410a55c90011c298b04d0cfa770b00fa04e1e3c97d3f6c9de105a03844"
|
||||
|
||||
[[package]]
|
||||
name = "fnv"
|
||||
@@ -767,9 +777,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "icu_collections"
|
||||
version = "2.0.0"
|
||||
version = "2.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "200072f5d0e3614556f94a9930d5dc3e0662a652823904c3a75dc3b0af7fee47"
|
||||
checksum = "4c6b649701667bbe825c3b7e6388cb521c23d88644678e83c0c4d0a621a34b43"
|
||||
dependencies = [
|
||||
"displaydoc",
|
||||
"potential_utf",
|
||||
@@ -780,9 +790,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "icu_locale_core"
|
||||
version = "2.0.0"
|
||||
version = "2.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0cde2700ccaed3872079a65fb1a78f6c0a36c91570f28755dda67bc8f7d9f00a"
|
||||
checksum = "edba7861004dd3714265b4db54a3c390e880ab658fec5f7db895fae2046b5bb6"
|
||||
dependencies = [
|
||||
"displaydoc",
|
||||
"litemap",
|
||||
@@ -793,11 +803,10 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "icu_normalizer"
|
||||
version = "2.0.0"
|
||||
version = "2.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "436880e8e18df4d7bbc06d58432329d6458cc84531f7ac5f024e93deadb37979"
|
||||
checksum = "5f6c8828b67bf8908d82127b2054ea1b4427ff0230ee9141c54251934ab1b599"
|
||||
dependencies = [
|
||||
"displaydoc",
|
||||
"icu_collections",
|
||||
"icu_normalizer_data",
|
||||
"icu_properties",
|
||||
@@ -808,42 +817,38 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "icu_normalizer_data"
|
||||
version = "2.0.0"
|
||||
version = "2.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "00210d6893afc98edb752b664b8890f0ef174c8adbb8d0be9710fa66fbbf72d3"
|
||||
checksum = "7aedcccd01fc5fe81e6b489c15b247b8b0690feb23304303a9e560f37efc560a"
|
||||
|
||||
[[package]]
|
||||
name = "icu_properties"
|
||||
version = "2.0.1"
|
||||
version = "2.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "016c619c1eeb94efb86809b015c58f479963de65bdb6253345c1a1276f22e32b"
|
||||
checksum = "e93fcd3157766c0c8da2f8cff6ce651a31f0810eaa1c51ec363ef790bbb5fb99"
|
||||
dependencies = [
|
||||
"displaydoc",
|
||||
"icu_collections",
|
||||
"icu_locale_core",
|
||||
"icu_properties_data",
|
||||
"icu_provider",
|
||||
"potential_utf",
|
||||
"zerotrie",
|
||||
"zerovec",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "icu_properties_data"
|
||||
version = "2.0.1"
|
||||
version = "2.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "298459143998310acd25ffe6810ed544932242d3f07083eee1084d83a71bd632"
|
||||
checksum = "02845b3647bb045f1100ecd6480ff52f34c35f82d9880e029d329c21d1054899"
|
||||
|
||||
[[package]]
|
||||
name = "icu_provider"
|
||||
version = "2.0.0"
|
||||
version = "2.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "03c80da27b5f4187909049ee2d72f276f0d9f99a42c306bd0131ecfe04d8e5af"
|
||||
checksum = "85962cf0ce02e1e0a629cc34e7ca3e373ce20dda4c4d7294bbd0bf1fdb59e614"
|
||||
dependencies = [
|
||||
"displaydoc",
|
||||
"icu_locale_core",
|
||||
"stable_deref_trait",
|
||||
"tinystr",
|
||||
"writeable",
|
||||
"yoke",
|
||||
"zerofrom",
|
||||
@@ -884,9 +889,12 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "indoc"
|
||||
version = "2.0.6"
|
||||
version = "2.0.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f4c7245a08504955605670dbf141fceab975f15ca21570696aebe9d2e71576bd"
|
||||
checksum = "79cf5c93f93228cf8efb3ba362535fb11199ac548a09ce117c9b1adc3030d706"
|
||||
dependencies = [
|
||||
"rustversion",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ipnet"
|
||||
@@ -896,9 +904,9 @@ checksum = "469fb0b9cefa57e3ef31275ee7cacb78f2fdca44e4765491884a2b119d4eb130"
|
||||
|
||||
[[package]]
|
||||
name = "is_terminal_polyfill"
|
||||
version = "1.70.1"
|
||||
version = "1.70.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7943c866cc5cd64cbc25b2e01621d07fa8eb2a1a23160ee81ce38704e97b8ecf"
|
||||
checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695"
|
||||
|
||||
[[package]]
|
||||
name = "itertools"
|
||||
@@ -927,9 +935,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "js-sys"
|
||||
version = "0.3.81"
|
||||
version = "0.3.82"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ec48937a97411dcb524a265206ccd4c90bb711fca92b2792c407f268825b9305"
|
||||
checksum = "b011eec8cc36da2aab2d5cff675ec18454fad408585853910a202391cf9f8e65"
|
||||
dependencies = [
|
||||
"once_cell",
|
||||
"wasm-bindgen",
|
||||
@@ -987,9 +995,9 @@ checksum = "df1d3c3b53da64cf5760482273a98e575c651a67eec7f77df96b5b642de8f039"
|
||||
|
||||
[[package]]
|
||||
name = "litemap"
|
||||
version = "0.8.0"
|
||||
version = "0.8.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "241eaef5fd12c88705a01fc1066c48c4b36e0dd4377dcdc7ec3942cea7a69956"
|
||||
checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77"
|
||||
|
||||
[[package]]
|
||||
name = "lock_api"
|
||||
@@ -1103,6 +1111,15 @@ dependencies = [
|
||||
"autocfg",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "object"
|
||||
version = "0.32.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a6a622008b6e321afc04970976f62ee297fdbaa6f95318ca343e3eebb9648441"
|
||||
dependencies = [
|
||||
"memchr",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "once_cell"
|
||||
version = "1.21.3"
|
||||
@@ -1111,15 +1128,15 @@ checksum = "42f5e15c9953c5e4ccceeb2e7382a716482c34515315f7b03532b8b4e8393d2d"
|
||||
|
||||
[[package]]
|
||||
name = "once_cell_polyfill"
|
||||
version = "1.70.1"
|
||||
version = "1.70.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a4895175b425cb1f87721b59f0f286c2092bd4af812243672510e1ac53e2e0ad"
|
||||
checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
|
||||
|
||||
[[package]]
|
||||
name = "openssl"
|
||||
version = "0.10.74"
|
||||
version = "0.10.75"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "24ad14dd45412269e1a30f52ad8f0664f0f4f4a89ee8fe28c3b3527021ebb654"
|
||||
checksum = "08838db121398ad17ab8531ce9de97b244589089e290a384c900cb9ff7434328"
|
||||
dependencies = [
|
||||
"bitflags 2.10.0",
|
||||
"cfg-if",
|
||||
@@ -1149,9 +1166,9 @@ checksum = "d05e27ee213611ffe7d6348b942e8f942b37114c00cc03cec254295a4a17852e"
|
||||
|
||||
[[package]]
|
||||
name = "openssl-sys"
|
||||
version = "0.9.110"
|
||||
version = "0.9.111"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0a9f0075ba3c21b09f8e8b2026584b1d18d49388648f2fbbf3c97ea8deced8e2"
|
||||
checksum = "82cab2d520aa75e3c58898289429321eb788c3106963d0dc886ec7a5f4adc321"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"libc",
|
||||
@@ -1261,36 +1278,37 @@ checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c"
|
||||
|
||||
[[package]]
|
||||
name = "potential_utf"
|
||||
version = "0.1.3"
|
||||
version = "0.1.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "84df19adbe5b5a0782edcab45899906947ab039ccf4573713735ee7de1e6b08a"
|
||||
checksum = "b73949432f5e2a09657003c25bca5e19a0e9c84f8058ca374f49e0ebe605af77"
|
||||
dependencies = [
|
||||
"zerovec",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "proc-macro2"
|
||||
version = "1.0.101"
|
||||
version = "1.0.103"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "89ae43fd86e4158d6db51ad8e2b80f313af9cc74f5c0e03ccb87de09998732de"
|
||||
checksum = "5ee95bc4ef87b8d5ba32e8b7714ccc834865276eab0aed5c9958d00ec45f49e8"
|
||||
dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "psm"
|
||||
version = "0.1.27"
|
||||
version = "0.1.28"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e66fcd288453b748497d8fb18bccc83a16b0518e3906d4b8df0a8d42d93dbb1c"
|
||||
checksum = "d11f2fedc3b7dafdc2851bc52f277377c5473d378859be234bc7ebb593144d01"
|
||||
dependencies = [
|
||||
"ar_archive_writer",
|
||||
"cc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quote"
|
||||
version = "1.0.41"
|
||||
version = "1.0.42"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ce25767e7b499d1b604768e7cde645d14cc8584231ea6b295e9c9eb22c02e1d1"
|
||||
checksum = "a338cc41d27e6cc6dce6cefc13a0729dfbb81c262b1f519331575dd80ef3067f"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
]
|
||||
@@ -1610,9 +1628,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "signal-hook-mio"
|
||||
version = "0.2.4"
|
||||
version = "0.2.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "34db1a06d485c9142248b7a054f034b349b212551f3dfd19c94d45a754a217cd"
|
||||
checksum = "b75a19a7a740b25bc7944bdee6172368f988763b744e3d4dfe753f6b4ece40cc"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"mio 0.8.11",
|
||||
@@ -1715,9 +1733,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "2.0.107"
|
||||
version = "2.0.110"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2a26dbd934e5451d21ef060c018dae56fc073894c5a7896f882928a76e6d081b"
|
||||
checksum = "a99801b5bd34ede4cf3fc688c5919368fea4e4814a4664359503e6015b280aea"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
@@ -1825,9 +1843,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "tinystr"
|
||||
version = "0.8.1"
|
||||
version = "0.8.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5d4f6d1145dcb577acf783d4e601bc1d76a13337bb54e6233add580b07344c8b"
|
||||
checksum = "42d3e9c45c09de15d06dd8acf5f4e0e399e85927b7f00711024eb7ae10fa4869"
|
||||
dependencies = [
|
||||
"displaydoc",
|
||||
"zerovec",
|
||||
@@ -1873,9 +1891,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "tokio-util"
|
||||
version = "0.7.16"
|
||||
version = "0.7.17"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "14307c986784f72ef81c89db7d9e28d6ac26d16213b109ea501696195e6e3ce5"
|
||||
checksum = "2efa149fe76073d6e8fd97ef4f4eca7b67f599660115591483572e406e165594"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"futures-core",
|
||||
@@ -2000,9 +2018,9 @@ checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
|
||||
|
||||
[[package]]
|
||||
name = "unicode-ident"
|
||||
version = "1.0.19"
|
||||
version = "1.0.22"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f63a545481291138910575129486daeaf8ac54aee4387fe7906919f7830c7d9d"
|
||||
checksum = "9312f7c4f6ff9069b165498234ce8be658059c6728633667c526e27dc2cf1df5"
|
||||
|
||||
[[package]]
|
||||
name = "unicode-segmentation"
|
||||
@@ -2054,9 +2072,9 @@ checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426"
|
||||
|
||||
[[package]]
|
||||
name = "version-compare"
|
||||
version = "0.2.0"
|
||||
version = "0.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "852e951cb7832cb45cb1169900d19760cfa39b82bc0ea9c0e5a14ae88411c98b"
|
||||
checksum = "03c2856837ef78f57382f06b2b8563a2f512f7185d732608fd9176cb3b8edf0e"
|
||||
|
||||
[[package]]
|
||||
name = "version_check"
|
||||
@@ -2064,6 +2082,12 @@ version = "0.9.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
|
||||
|
||||
[[package]]
|
||||
name = "wake-on-lan"
|
||||
version = "0.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1ccf60b60ad7e5b1b37372c5134cbcab4db0706c231d212e0c643a077462bc8f"
|
||||
|
||||
[[package]]
|
||||
name = "walkdir"
|
||||
version = "2.5.0"
|
||||
@@ -2100,9 +2124,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "wasm-bindgen"
|
||||
version = "0.2.104"
|
||||
version = "0.2.105"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c1da10c01ae9f1ae40cbfac0bac3b1e724b320abfcf52229f80b547c0d250e2d"
|
||||
checksum = "da95793dfc411fbbd93f5be7715b0578ec61fe87cb1a42b12eb625caa5c5ea60"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"once_cell",
|
||||
@@ -2111,25 +2135,11 @@ dependencies = [
|
||||
"wasm-bindgen-shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wasm-bindgen-backend"
|
||||
version = "0.2.104"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "671c9a5a66f49d8a47345ab942e2cb93c7d1d0339065d4f8139c486121b43b19"
|
||||
dependencies = [
|
||||
"bumpalo",
|
||||
"log",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn",
|
||||
"wasm-bindgen-shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wasm-bindgen-futures"
|
||||
version = "0.4.54"
|
||||
version = "0.4.55"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7e038d41e478cc73bae0ff9b36c60cff1c98b8f38f8d7e8061e79ee63608ac5c"
|
||||
checksum = "551f88106c6d5e7ccc7cd9a16f312dd3b5d36ea8b4954304657d5dfba115d4a0"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"js-sys",
|
||||
@@ -2140,9 +2150,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "wasm-bindgen-macro"
|
||||
version = "0.2.104"
|
||||
version = "0.2.105"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7ca60477e4c59f5f2986c50191cd972e3a50d8a95603bc9434501cf156a9a119"
|
||||
checksum = "04264334509e04a7bf8690f2384ef5265f05143a4bff3889ab7a3269adab59c2"
|
||||
dependencies = [
|
||||
"quote",
|
||||
"wasm-bindgen-macro-support",
|
||||
@@ -2150,31 +2160,31 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "wasm-bindgen-macro-support"
|
||||
version = "0.2.104"
|
||||
version = "0.2.105"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9f07d2f20d4da7b26400c9f4a0511e6e0345b040694e8a75bd41d578fa4421d7"
|
||||
checksum = "420bc339d9f322e562942d52e115d57e950d12d88983a14c79b86859ee6c7ebc"
|
||||
dependencies = [
|
||||
"bumpalo",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn",
|
||||
"wasm-bindgen-backend",
|
||||
"wasm-bindgen-shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wasm-bindgen-shared"
|
||||
version = "0.2.104"
|
||||
version = "0.2.105"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "bad67dc8b2a1a6e5448428adec4c3e84c43e561d8c9ee8a9e5aabeb193ec41d1"
|
||||
checksum = "76f218a38c84bcb33c25ec7059b07847d465ce0e0a76b995e134a45adcb6af76"
|
||||
dependencies = [
|
||||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "web-sys"
|
||||
version = "0.3.81"
|
||||
version = "0.3.82"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9367c417a924a74cae129e6a2ae3b47fabb1f8995595ab474029da749a8be120"
|
||||
checksum = "3a1f95c0d03a47f4ae1f7a64643a6bb97465d9b740f0fa8f90ea33915c99a9a1"
|
||||
dependencies = [
|
||||
"js-sys",
|
||||
"wasm-bindgen",
|
||||
@@ -2528,17 +2538,16 @@ checksum = "f17a85883d4e6d00e8a97c586de764dabcc06133f7f1d55dce5cdc070ad7fe59"
|
||||
|
||||
[[package]]
|
||||
name = "writeable"
|
||||
version = "0.6.1"
|
||||
version = "0.6.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ea2f10b9bb0928dfb1b42b65e1f9e36f7f54dbdf08457afefb38afcdec4fa2bb"
|
||||
checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9"
|
||||
|
||||
[[package]]
|
||||
name = "yoke"
|
||||
version = "0.8.0"
|
||||
version = "0.8.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5f41bb01b8226ef4bfd589436a297c53d118f65921786300e427be8d487695cc"
|
||||
checksum = "72d6e5c6afb84d73944e5cedb052c4680d5657337201555f9f2a16b7406d4954"
|
||||
dependencies = [
|
||||
"serde",
|
||||
"stable_deref_trait",
|
||||
"yoke-derive",
|
||||
"zerofrom",
|
||||
@@ -2546,9 +2555,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "yoke-derive"
|
||||
version = "0.8.0"
|
||||
version = "0.8.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "38da3c9736e16c5d3c8c597a9aaa5d1fa565d0532ae05e27c24aa62fb32c0ab6"
|
||||
checksum = "b659052874eb698efe5b9e8cf382204678a0086ebf46982b79d6ca3182927e5d"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
@@ -2609,9 +2618,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "zerotrie"
|
||||
version = "0.2.2"
|
||||
version = "0.2.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "36f0bbd478583f79edad978b407914f61b2972f5af6fa089686016be8f9af595"
|
||||
checksum = "2a59c17a5562d507e4b54960e8569ebee33bee890c70aa3fe7b97e85a9fd7851"
|
||||
dependencies = [
|
||||
"displaydoc",
|
||||
"yoke",
|
||||
@@ -2620,9 +2629,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "zerovec"
|
||||
version = "0.11.4"
|
||||
version = "0.11.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e7aa2bd55086f1ab526693ecbe444205da57e25f4489879da80635a46d90e73b"
|
||||
checksum = "6c28719294829477f525be0186d13efa9a3c602f7ec202ca9e353d310fb9a002"
|
||||
dependencies = [
|
||||
"yoke",
|
||||
"zerofrom",
|
||||
@@ -2631,9 +2640,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "zerovec-derive"
|
||||
version = "0.11.1"
|
||||
version = "0.11.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5b96237efa0c878c64bd89c436f661be4e46b2f3eff1ebb976f7ef2321d2f58f"
|
||||
checksum = "eadce39539ca5cb3985590102671f2567e659fca9666581ad3411d59207951f3"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
|
||||
518
README.md
518
README.md
@@ -1,88 +1,108 @@
|
||||
# CM Dashboard
|
||||
|
||||
A real-time infrastructure monitoring system with intelligent status aggregation and email notifications, built with Rust and ZMQ.
|
||||
A high-performance Rust-based TUI dashboard for monitoring CMTEC infrastructure. Built with ZMQ-based metric collection and individual metrics architecture.
|
||||
|
||||
## Current Implementation
|
||||
## Features
|
||||
|
||||
This is a complete rewrite implementing an **individual metrics architecture** where:
|
||||
### Core Monitoring
|
||||
- **Real-time metrics**: CPU, RAM, Storage, and Service status
|
||||
- **Multi-host support**: Monitor multiple servers from single dashboard
|
||||
- **Service management**: Start/stop services with intelligent status tracking
|
||||
- **NixOS integration**: System rebuild via SSH + tmux popup
|
||||
- **Backup monitoring**: Borgbackup status and scheduling
|
||||
- **Email notifications**: Intelligent batching prevents spam
|
||||
|
||||
- **Agent** collects individual metrics (e.g., `cpu_load_1min`, `memory_usage_percent`) and calculates status
|
||||
- **Dashboard** subscribes to specific metrics and composes widgets
|
||||
- **Status Aggregation** provides intelligent email notifications with batching
|
||||
- **Persistent Cache** prevents false notifications on restart
|
||||
### User-Stopped Service Tracking
|
||||
Services stopped via the dashboard are intelligently tracked to prevent false alerts:
|
||||
|
||||
## Dashboard Interface
|
||||
- **Smart status reporting**: User-stopped services show as Status::OK instead of Warning
|
||||
- **Persistent storage**: Tracking survives agent restarts via JSON storage
|
||||
- **Automatic management**: Flags cleared when services restarted via dashboard
|
||||
- **Maintenance friendly**: No false alerts during intentional service operations
|
||||
|
||||
## Architecture
|
||||
|
||||
### Individual Metrics Philosophy
|
||||
- **Agent**: Collects individual metrics, calculates status using thresholds
|
||||
- **Dashboard**: Subscribes to specific metrics, composes widgets from individual data
|
||||
- **ZMQ Communication**: Efficient real-time metric transmission
|
||||
- **Status Aggregation**: Host-level status calculated from all service metrics
|
||||
|
||||
### Components
|
||||
|
||||
```
|
||||
┌─────────────────┐ ZMQ ┌─────────────────┐
|
||||
│ │◄──────────►│ │
|
||||
│ Agent │ Metrics │ Dashboard │
|
||||
│ - Collectors │ │ - TUI │
|
||||
│ - Status │ │ - Widgets │
|
||||
│ - Tracking │ │ - Commands │
|
||||
│ │ │ │
|
||||
└─────────────────┘ └─────────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────┐ ┌─────────────────┐
|
||||
│ JSON Storage │ │ SSH + tmux │
|
||||
│ - User-stopped │ │ - Remote rebuild│
|
||||
│ - Cache │ │ - Process │
|
||||
│ - State │ │ isolation │
|
||||
└─────────────────┘ └─────────────────┘
|
||||
```
|
||||
|
||||
### Service Control Flow
|
||||
|
||||
1. **User Action**: Dashboard sends `UserStart`/`UserStop` commands
|
||||
2. **Agent Processing**:
|
||||
- Marks service as user-stopped (if stopping)
|
||||
- Executes `systemctl start/stop service`
|
||||
- Syncs state to global tracker
|
||||
3. **Status Calculation**:
|
||||
- Systemd collector checks user-stopped flag
|
||||
- Reports Status::OK for user-stopped inactive services
|
||||
- Normal Warning status for system failures
|
||||
|
||||
## Interface
|
||||
|
||||
```
|
||||
cm-dashboard • ● cmbox ● srv01 ● srv02 ● steambox
|
||||
┌system──────────────────────────────┐┌services─────────────────────────────────────────┐
|
||||
│CPU: ││Service: Status: RAM: Disk: │
|
||||
│● Load: 0.10 0.52 0.88 • 400.0 MHz ││● docker active 27M 496MB │
|
||||
│RAM: ││● docker-registry active 19M 496MB │
|
||||
│● Used: 30% 2.3GB/7.6GB ││● gitea active 579M 2.6GB │
|
||||
│● tmp: 0.0% 0B/2.0GB ││● gitea-runner-default active 11M 2.6GB │
|
||||
│Disk nvme0n1: ││● haasp-core active 9M 1MB │
|
||||
│● Health: PASSED ││● haasp-mqtt active 3M 1MB │
|
||||
│● Usage @root: 8.3% • 75.4/906.2 GB ││● haasp-webgrid active 10M 1MB │
|
||||
│● Usage @boot: 5.9% • 0.1/1.0 GB ││● immich-server active 240M 45.1GB │
|
||||
│ ││● mosquitto active 1M 1MB │
|
||||
│ ││● mysql active 38M 225MB │
|
||||
│ ││● nginx active 28M 24MB │
|
||||
│ ││ ├─ ● gitea.cmtec.se 51ms │
|
||||
│ ││ ├─ ● haasp.cmtec.se 43ms │
|
||||
│ ││ ├─ ● haasp.net 43ms │
|
||||
│ ││ ├─ ● pages.cmtec.se 45ms │
|
||||
└────────────────────────────────────┘│ ├─ ● photos.cmtec.se 41ms │
|
||||
┌backup──────────────────────────────┐│ ├─ ● unifi.cmtec.se 46ms │
|
||||
│Latest backup: ││ ├─ ● vault.cmtec.se 47ms │
|
||||
│● Status: OK ││ ├─ ● www.kryddorten.se 81ms │
|
||||
│Duration: 54s • Last: 4h ago ││ ├─ ● www.mariehall2.se 86ms │
|
||||
│Disk usage: 48.2GB/915.8GB ││● postgresql active 112M 357MB │
|
||||
│P/N: Samsung SSD 870 QVO 1TB ││● redis-immich active 8M 45.1GB │
|
||||
│S/N: S5RRNF0W800639Y ││● sshd active 2M 0 │
|
||||
│● gitea 2 archives 2.7GB ││● unifi active 594M 495MB │
|
||||
│● immich 2 archives 45.0GB ││● vaultwarden active 12M 1MB │
|
||||
│● kryddorten 2 archives 67.6MB ││ │
|
||||
│● mariehall2 2 archives 321.8MB ││ │
|
||||
│● nixosbox 2 archives 4.5MB ││ │
|
||||
│● unifi 2 archives 2.9MB ││ │
|
||||
│● vaultwarden 2 archives 305kB ││ │
|
||||
│NixOS: ││Service: Status: RAM: Disk: │
|
||||
│Build: 25.05.20251004.3bcc93c ││● docker active 27M 496MB │
|
||||
│Agent: v0.1.43 ││● gitea active 579M 2.6GB │
|
||||
│Active users: cm, simon ││● nginx active 28M 24MB │
|
||||
│CPU: ││ ├─ ● gitea.cmtec.se 51ms │
|
||||
│● Load: 0.10 0.52 0.88 • 3000MHz ││ ├─ ● photos.cmtec.se 41ms │
|
||||
│RAM: ││● postgresql active 112M 357MB │
|
||||
│● Usage: 33% 2.6GB/7.6GB ││● redis-immich user-stopped │
|
||||
│● /tmp: 0% 0B/2.0GB ││● sshd active 2M 0 │
|
||||
│Storage: ││● unifi active 594M 495MB │
|
||||
│● root (Single): ││ │
|
||||
│ ├─ ● nvme0n1 W: 1% ││ │
|
||||
│ └─ ● 18% 167.4GB/928.2GB ││ │
|
||||
└────────────────────────────────────┘└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Navigation**: `←→` switch hosts, `r` refresh, `q` quit
|
||||
### Navigation
|
||||
- **Tab**: Switch between hosts
|
||||
- **↑↓ or j/k**: Navigate services
|
||||
- **s**: Start selected service (UserStart)
|
||||
- **S**: Stop selected service (UserStop)
|
||||
- **J**: Show service logs (journalctl in tmux popup)
|
||||
- **L**: Show custom log files (tail -f custom paths in tmux popup)
|
||||
- **R**: Rebuild current host
|
||||
- **B**: Run backup on current host
|
||||
- **q**: Quit
|
||||
|
||||
## Features
|
||||
|
||||
- **Real-time monitoring** - Dashboard updates every 1-2 seconds
|
||||
- **Individual metric collection** - Granular data for flexible dashboard composition
|
||||
- **Intelligent status aggregation** - Host-level status calculated from all services
|
||||
- **Smart email notifications** - Batched, detailed alerts with service groupings
|
||||
- **Persistent state** - Prevents false notifications on restarts
|
||||
- **ZMQ communication** - Efficient agent-to-dashboard messaging
|
||||
- **Clean TUI** - Terminal-based dashboard with color-coded status indicators
|
||||
|
||||
## Architecture
|
||||
|
||||
### Core Components
|
||||
|
||||
- **Agent** (`cm-dashboard-agent`) - Collects metrics and sends via ZMQ
|
||||
- **Dashboard** (`cm-dashboard`) - Real-time TUI display consuming metrics
|
||||
- **Shared** (`cm-dashboard-shared`) - Common types and protocol
|
||||
- **Status Aggregation** - Intelligent batching and notification management
|
||||
- **Persistent Cache** - Maintains state across restarts
|
||||
|
||||
### Status Levels
|
||||
|
||||
- **🟢 Ok** - Service running normally
|
||||
- **🔵 Pending** - Service starting/stopping/reloading
|
||||
- **🟡 Warning** - Service issues (high load, memory, disk usage)
|
||||
- **🔴 Critical** - Service failed or critical thresholds exceeded
|
||||
- **❓ Unknown** - Service state cannot be determined
|
||||
### Status Indicators
|
||||
- **Green ●**: Active service
|
||||
- **Yellow ◐**: Inactive service (system issue)
|
||||
- **Red ◯**: Failed service
|
||||
- **Blue arrows**: Service transitioning (↑ starting, ↓ stopping, ↻ restarting)
|
||||
- **"user-stopped"**: Service stopped via dashboard (Status::OK)
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Build
|
||||
### Building
|
||||
|
||||
```bash
|
||||
# With Nix (recommended)
|
||||
@@ -93,21 +113,20 @@ sudo apt install libssl-dev pkg-config # Ubuntu/Debian
|
||||
cargo build --workspace
|
||||
```
|
||||
|
||||
### Run
|
||||
### Running
|
||||
|
||||
```bash
|
||||
# Start agent (requires configuration file)
|
||||
# Start agent (requires configuration)
|
||||
./target/debug/cm-dashboard-agent --config /etc/cm-dashboard/agent.toml
|
||||
|
||||
# Start dashboard
|
||||
./target/debug/cm-dashboard --config /path/to/dashboard.toml
|
||||
# Start dashboard (inside tmux session)
|
||||
tmux
|
||||
./target/debug/cm-dashboard --config /etc/cm-dashboard/dashboard.toml
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Agent Configuration (`agent.toml`)
|
||||
|
||||
The agent requires a comprehensive TOML configuration file:
|
||||
### Agent Configuration
|
||||
|
||||
```toml
|
||||
collection_interval_seconds = 2
|
||||
@@ -116,50 +135,27 @@ collection_interval_seconds = 2
|
||||
publisher_port = 6130
|
||||
command_port = 6131
|
||||
bind_address = "0.0.0.0"
|
||||
timeout_ms = 5000
|
||||
heartbeat_interval_ms = 30000
|
||||
transmission_interval_seconds = 2
|
||||
|
||||
[collectors.cpu]
|
||||
enabled = true
|
||||
interval_seconds = 2
|
||||
load_warning_threshold = 9.0
|
||||
load_warning_threshold = 5.0
|
||||
load_critical_threshold = 10.0
|
||||
temperature_warning_threshold = 100.0
|
||||
temperature_critical_threshold = 110.0
|
||||
|
||||
[collectors.memory]
|
||||
enabled = true
|
||||
interval_seconds = 2
|
||||
usage_warning_percent = 80.0
|
||||
usage_critical_percent = 95.0
|
||||
|
||||
[collectors.disk]
|
||||
enabled = true
|
||||
interval_seconds = 300
|
||||
usage_warning_percent = 80.0
|
||||
usage_critical_percent = 90.0
|
||||
|
||||
[[collectors.disk.filesystems]]
|
||||
name = "root"
|
||||
uuid = "4cade5ce-85a5-4a03-83c8-dfd1d3888d79"
|
||||
mount_point = "/"
|
||||
fs_type = "ext4"
|
||||
monitor = true
|
||||
|
||||
[collectors.systemd]
|
||||
enabled = true
|
||||
interval_seconds = 10
|
||||
memory_warning_mb = 1000.0
|
||||
memory_critical_mb = 2000.0
|
||||
service_name_filters = [
|
||||
"nginx*", "postgresql*", "redis*", "docker*", "sshd*",
|
||||
"gitea*", "immich*", "haasp*", "mosquitto*", "mysql*",
|
||||
"unifi*", "vaultwarden*"
|
||||
]
|
||||
excluded_services = [
|
||||
"nginx-config-reload", "sshd-keygen", "systemd-",
|
||||
"getty@", "user@", "dbus-", "NetworkManager-"
|
||||
]
|
||||
service_name_filters = ["nginx*", "postgresql*", "docker*", "sshd*"]
|
||||
excluded_services = ["nginx-config-reload", "systemd-", "getty@"]
|
||||
nginx_latency_critical_ms = 1000.0
|
||||
http_timeout_seconds = 10
|
||||
|
||||
[notifications]
|
||||
enabled = true
|
||||
@@ -167,251 +163,203 @@ smtp_host = "localhost"
|
||||
smtp_port = 25
|
||||
from_email = "{hostname}@example.com"
|
||||
to_email = "admin@example.com"
|
||||
rate_limit_minutes = 0
|
||||
trigger_on_warnings = true
|
||||
trigger_on_failures = true
|
||||
recovery_requires_all_ok = true
|
||||
suppress_individual_recoveries = true
|
||||
|
||||
[status_aggregation]
|
||||
enabled = true
|
||||
aggregation_method = "worst_case"
|
||||
notification_interval_seconds = 30
|
||||
|
||||
[cache]
|
||||
persist_path = "/var/lib/cm-dashboard/cache.json"
|
||||
aggregation_interval_seconds = 30
|
||||
```
|
||||
|
||||
### Dashboard Configuration (`dashboard.toml`)
|
||||
### Dashboard Configuration
|
||||
|
||||
```toml
|
||||
[zmq]
|
||||
hosts = [
|
||||
{ name = "server1", address = "192.168.1.100", port = 6130 },
|
||||
{ name = "server2", address = "192.168.1.101", port = 6130 }
|
||||
]
|
||||
connection_timeout_ms = 5000
|
||||
reconnect_interval_ms = 10000
|
||||
subscriber_ports = [6130]
|
||||
|
||||
[ui]
|
||||
refresh_interval_ms = 1000
|
||||
theme = "dark"
|
||||
[hosts]
|
||||
predefined_hosts = ["cmbox", "srv01", "srv02"]
|
||||
|
||||
[ssh]
|
||||
rebuild_user = "cm"
|
||||
rebuild_alias = "nixos-rebuild-cmtec"
|
||||
backup_alias = "cm-backup-run"
|
||||
```
|
||||
|
||||
## Collectors
|
||||
## Technical Implementation
|
||||
|
||||
The agent implements several specialized collectors:
|
||||
### Collectors
|
||||
|
||||
### CPU Collector (`cpu.rs`)
|
||||
#### Systemd Collector
|
||||
- **Service Discovery**: Uses `systemctl list-unit-files` + `list-units --all`
|
||||
- **Status Calculation**: Checks user-stopped flag before assigning Warning status
|
||||
- **Memory Tracking**: Per-service memory usage via `systemctl show`
|
||||
- **Sub-services**: Nginx site latency, Docker containers
|
||||
- **User-stopped Integration**: `UserStoppedServiceTracker::is_service_user_stopped()`
|
||||
|
||||
- Load average (1, 5, 15 minute)
|
||||
- CPU temperature monitoring
|
||||
- Real-time process monitoring (top CPU consumers)
|
||||
- Status calculation with configurable thresholds
|
||||
#### User-Stopped Service Tracker
|
||||
- **Storage**: `/var/lib/cm-dashboard/user-stopped-services.json`
|
||||
- **Thread Safety**: Global singleton with `Arc<Mutex<>>`
|
||||
- **Persistence**: Automatic save on state changes
|
||||
- **Global Access**: Static methods for collector integration
|
||||
|
||||
### Memory Collector (`memory.rs`)
|
||||
#### Other Collectors
|
||||
- **CPU**: Load average, temperature, frequency monitoring
|
||||
- **Memory**: RAM/swap usage, tmpfs monitoring
|
||||
- **Disk**: Filesystem usage, SMART health data
|
||||
- **NixOS**: Build version, active users, agent version
|
||||
- **Backup**: Borgbackup repository status and metrics
|
||||
|
||||
- RAM usage (total, used, available)
|
||||
- Swap monitoring
|
||||
- Real-time process monitoring (top RAM consumers)
|
||||
- Memory pressure detection
|
||||
### ZMQ Protocol
|
||||
|
||||
### Disk Collector (`disk.rs`)
|
||||
```rust
|
||||
// Metric Message
|
||||
#[derive(Serialize, Deserialize)]
|
||||
pub struct MetricMessage {
|
||||
pub hostname: String,
|
||||
pub timestamp: u64,
|
||||
pub metrics: Vec<Metric>,
|
||||
}
|
||||
|
||||
- Filesystem usage per mount point
|
||||
- SMART health monitoring
|
||||
- Temperature and wear tracking
|
||||
- Configurable filesystem monitoring
|
||||
// Service Commands
|
||||
pub enum AgentCommand {
|
||||
ServiceControl {
|
||||
service_name: String,
|
||||
action: ServiceAction,
|
||||
},
|
||||
SystemRebuild { /* SSH config */ },
|
||||
CollectNow,
|
||||
}
|
||||
|
||||
### Systemd Collector (`systemd.rs`)
|
||||
pub enum ServiceAction {
|
||||
Start, // System-initiated
|
||||
Stop, // System-initiated
|
||||
UserStart, // User via dashboard (clears user-stopped)
|
||||
UserStop, // User via dashboard (marks user-stopped)
|
||||
Status,
|
||||
}
|
||||
```
|
||||
|
||||
- Service status monitoring (`active`, `inactive`, `failed`)
|
||||
- Memory usage per service
|
||||
- Service filtering and exclusions
|
||||
- Handles transitional states (`Status::Pending`)
|
||||
### Maintenance Mode
|
||||
|
||||
### Backup Collector (`backup.rs`)
|
||||
Suppress notifications during planned maintenance:
|
||||
|
||||
- Reads TOML status files from backup systems
|
||||
- Archive age verification
|
||||
- Disk usage tracking
|
||||
- Repository health monitoring
|
||||
```bash
|
||||
# Enable maintenance mode
|
||||
touch /tmp/cm-maintenance
|
||||
|
||||
# Perform maintenance
|
||||
systemctl stop service
|
||||
# ... work ...
|
||||
systemctl start service
|
||||
|
||||
# Disable maintenance mode
|
||||
rm /tmp/cm-maintenance
|
||||
```
|
||||
|
||||
## Email Notifications
|
||||
|
||||
### Intelligent Batching
|
||||
- **Real-time dashboard**: Immediate status updates
|
||||
- **Batched emails**: Aggregated every 30 seconds
|
||||
- **Smart grouping**: Services organized by severity
|
||||
- **Recovery suppression**: Reduces notification spam
|
||||
|
||||
The system implements smart notification batching to prevent email spam:
|
||||
|
||||
- **Real-time dashboard updates** - Status changes appear immediately
|
||||
- **Batched email notifications** - Aggregated every 30 seconds
|
||||
- **Detailed groupings** - Services organized by severity
|
||||
|
||||
### Example Alert Email
|
||||
|
||||
### Example Alert
|
||||
```
|
||||
Subject: Status Alert: 2 critical, 1 warning, 15 started
|
||||
Subject: Status Alert: 1 critical, 2 warnings, 0 recoveries
|
||||
|
||||
Status Summary (30s duration)
|
||||
Host Status: Ok → Warning
|
||||
|
||||
🔴 CRITICAL ISSUES (2):
|
||||
postgresql: Ok → Critical
|
||||
nginx: Warning → Critical
|
||||
🔴 CRITICAL ISSUES (1):
|
||||
postgresql: Ok → Critical (memory usage 95%)
|
||||
|
||||
🟡 WARNINGS (1):
|
||||
redis: Ok → Warning (memory usage 85%)
|
||||
🟡 WARNINGS (2):
|
||||
nginx: Ok → Warning (high load 8.5)
|
||||
redis: user-stopped → Warning (restarted by system)
|
||||
|
||||
✅ RECOVERIES (0):
|
||||
|
||||
🟢 SERVICE STARTUPS (15):
|
||||
docker: Unknown → Ok
|
||||
sshd: Unknown → Ok
|
||||
...
|
||||
|
||||
--
|
||||
CM Dashboard Agent
|
||||
Generated at 2025-10-21 19:42:42 CET
|
||||
CM Dashboard Agent v0.1.43
|
||||
```
|
||||
|
||||
## Individual Metrics Architecture
|
||||
|
||||
The system follows a **metrics-first architecture**:
|
||||
|
||||
### Agent Side
|
||||
|
||||
```rust
|
||||
// Agent collects individual metrics
|
||||
vec![
|
||||
Metric::new("cpu_load_1min".to_string(), MetricValue::Float(2.5), Status::Ok),
|
||||
Metric::new("memory_usage_percent".to_string(), MetricValue::Float(78.5), Status::Warning),
|
||||
Metric::new("service_nginx_status".to_string(), MetricValue::String("active".to_string()), Status::Ok),
|
||||
]
|
||||
```
|
||||
|
||||
### Dashboard Side
|
||||
|
||||
```rust
|
||||
// Widgets subscribe to specific metrics
|
||||
impl Widget for CpuWidget {
|
||||
fn update_from_metrics(&mut self, metrics: &[&Metric]) {
|
||||
for metric in metrics {
|
||||
match metric.name.as_str() {
|
||||
"cpu_load_1min" => self.load_1min = metric.value.as_f32(),
|
||||
"cpu_load_5min" => self.load_5min = metric.value.as_f32(),
|
||||
"cpu_temperature_celsius" => self.temperature = metric.value.as_f32(),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Persistent Cache
|
||||
|
||||
The cache system prevents false notifications:
|
||||
|
||||
- **Automatic saving** - Saves when service status changes
|
||||
- **Persistent storage** - Maintains state across agent restarts
|
||||
- **Simple design** - No complex TTL or cleanup logic
|
||||
- **Status preservation** - Prevents duplicate notifications
|
||||
|
||||
## Development
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
cm-dashboard/
|
||||
├── agent/ # Metrics collection agent
|
||||
├── agent/ # Metrics collection agent
|
||||
│ ├── src/
|
||||
│ │ ├── collectors/ # CPU, memory, disk, systemd, backup
|
||||
│ │ ├── status/ # Status aggregation and notifications
|
||||
│ │ ├── cache/ # Persistent metric caching
|
||||
│ │ ├── config/ # TOML configuration loading
|
||||
│ │ └── notifications/ # Email notification system
|
||||
├── dashboard/ # TUI dashboard application
|
||||
│ │ ├── collectors/ # CPU, memory, disk, systemd, backup, nixos
|
||||
│ │ ├── service_tracker.rs # User-stopped service tracking
|
||||
│ │ ├── status/ # Status aggregation and notifications
|
||||
│ │ ├── config/ # TOML configuration loading
|
||||
│ │ └── communication/ # ZMQ message handling
|
||||
├── dashboard/ # TUI dashboard application
|
||||
│ ├── src/
|
||||
│ │ ├── ui/widgets/ # CPU, memory, services, backup widgets
|
||||
│ │ ├── metrics/ # Metric storage and filtering
|
||||
│ │ └── communication/ # ZMQ metric consumption
|
||||
├── shared/ # Shared types and utilities
|
||||
│ │ ├── ui/widgets/ # CPU, memory, services, backup, system
|
||||
│ │ ├── communication/ # ZMQ consumption and commands
|
||||
│ │ └── app.rs # Main application loop
|
||||
├── shared/ # Shared types and utilities
|
||||
│ └── src/
|
||||
│ ├── metrics.rs # Metric, Status, and Value types
|
||||
│ ├── protocol.rs # ZMQ message format
|
||||
│ └── cache.rs # Cache configuration
|
||||
└── README.md # This file
|
||||
│ ├── metrics.rs # Metric, Status, StatusTracker types
|
||||
│ ├── protocol.rs # ZMQ message format
|
||||
│ └── cache.rs # Cache configuration
|
||||
└── CLAUDE.md # Development guidelines and rules
|
||||
```
|
||||
|
||||
### Building
|
||||
|
||||
### Testing
|
||||
```bash
|
||||
# Debug build
|
||||
cargo build --workspace
|
||||
# Build and test
|
||||
nix-shell -p openssl pkg-config --run "cargo build --workspace"
|
||||
nix-shell -p openssl pkg-config --run "cargo test --workspace"
|
||||
|
||||
# Release build
|
||||
cargo build --workspace --release
|
||||
|
||||
# Run tests
|
||||
cargo test --workspace
|
||||
|
||||
# Check code formatting
|
||||
cargo fmt --all -- --check
|
||||
|
||||
# Run clippy linter
|
||||
# Code quality
|
||||
cargo fmt --all
|
||||
cargo clippy --workspace -- -D warnings
|
||||
```
|
||||
|
||||
### Dependencies
|
||||
## Deployment
|
||||
|
||||
- **tokio** - Async runtime
|
||||
- **zmq** - Message passing between agent and dashboard
|
||||
- **ratatui** - Terminal user interface
|
||||
- **serde** - Serialization for metrics and config
|
||||
- **anyhow/thiserror** - Error handling
|
||||
- **tracing** - Structured logging
|
||||
- **lettre** - SMTP email notifications
|
||||
- **clap** - Command-line argument parsing
|
||||
- **toml** - Configuration file parsing
|
||||
### Automated Binary Releases
|
||||
```bash
|
||||
# Create new release
|
||||
cd ~/projects/cm-dashboard
|
||||
git tag v0.1.X
|
||||
git push origin v0.1.X
|
||||
```
|
||||
|
||||
## NixOS Integration
|
||||
This triggers automated:
|
||||
- Static binary compilation with `RUSTFLAGS="-C target-feature=+crt-static"`
|
||||
- GitHub-style release creation
|
||||
- Tarball upload to Gitea
|
||||
|
||||
This project is designed for declarative deployment via NixOS:
|
||||
|
||||
### Configuration Generation
|
||||
|
||||
The NixOS module automatically generates the agent configuration:
|
||||
### NixOS Integration
|
||||
Update `~/projects/nixosbox/hosts/services/cm-dashboard.nix`:
|
||||
|
||||
```nix
|
||||
# hosts/common/cm-dashboard.nix
|
||||
services.cm-dashboard-agent = {
|
||||
enable = true;
|
||||
port = 6130;
|
||||
version = "v0.1.43";
|
||||
src = pkgs.fetchurl {
|
||||
url = "https://gitea.cmtec.se/cm/cm-dashboard/releases/download/${version}/cm-dashboard-linux-x86_64.tar.gz";
|
||||
sha256 = "sha256-HASH";
|
||||
};
|
||||
```
|
||||
|
||||
### Deployment
|
||||
|
||||
Get hash via:
|
||||
```bash
|
||||
# Update NixOS configuration
|
||||
git add hosts/common/cm-dashboard.nix
|
||||
git commit -m "Update cm-dashboard configuration"
|
||||
git push
|
||||
|
||||
# Rebuild system (user-performed)
|
||||
sudo nixos-rebuild switch --flake .
|
||||
cd ~/projects/nixosbox
|
||||
nix-build --no-out-link -E 'with import <nixpkgs> {}; fetchurl {
|
||||
url = "URL_HERE";
|
||||
sha256 = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
|
||||
}' 2>&1 | grep "got:"
|
||||
```
|
||||
|
||||
## Monitoring Intervals
|
||||
|
||||
- **CPU/Memory**: 2 seconds (real-time monitoring)
|
||||
- **Disk usage**: 300 seconds (5 minutes)
|
||||
- **Systemd services**: 10 seconds
|
||||
- **SMART health**: 600 seconds (10 minutes)
|
||||
- **Backup status**: 60 seconds (1 minute)
|
||||
- **Email notifications**: 30 seconds (batched)
|
||||
- **Dashboard updates**: 1 second (real-time display)
|
||||
- **Metrics Collection**: 2 seconds (CPU, memory, services)
|
||||
- **Metric Transmission**: 2 seconds (ZMQ publish)
|
||||
- **Dashboard Updates**: 1 second (UI refresh)
|
||||
- **Email Notifications**: 30 seconds (batched)
|
||||
- **Disk Monitoring**: 300 seconds (5 minutes)
|
||||
- **Service Discovery**: 300 seconds (5 minutes cache)
|
||||
|
||||
## License
|
||||
|
||||
MIT License - see LICENSE file for details
|
||||
|
||||
MIT License - see LICENSE file for details.
|
||||
63
TODO.md
63
TODO.md
@@ -1,63 +0,0 @@
|
||||
# TODO
|
||||
|
||||
## Systemd filtering (agent)
|
||||
|
||||
- remove user systemd collection
|
||||
- reduce number of systemctl call
|
||||
- Cahnge so only services in include list are detected
|
||||
- Filter on exact name
|
||||
- Add support for "\*" in filtering
|
||||
|
||||
## System panel (agent/dashboard)
|
||||
|
||||
use following layout:
|
||||
'''
|
||||
NixOS:
|
||||
Build: xxxxxx
|
||||
Agen: xxxxxx
|
||||
CPU:
|
||||
● Load: 0.02 0.31 0.86
|
||||
└─ Freq: 3000MHz
|
||||
RAM:
|
||||
● Usage: 33% 2.6GB/7.6GB
|
||||
└─ ● /tmp: 0% 0B/2.0GB
|
||||
Storage:
|
||||
● /:
|
||||
├─ ● nvme0n1 T: 40C • W: 4%
|
||||
└─ ● 8% 75.0GB/906.2GB
|
||||
'''
|
||||
|
||||
- Add support to show login/active users
|
||||
- Add support to show timestamp/version for latest nixos rebuild
|
||||
|
||||
## Backup panel (dashboard)
|
||||
|
||||
use following layout:
|
||||
'''
|
||||
Latest backup:
|
||||
● <timestamp>
|
||||
└─ Duration: 1.3m
|
||||
Disk:
|
||||
● Samsung SSD 870 QVO 1TB
|
||||
├─ S/N: S5RRNF0W800639Y
|
||||
└─ Usage: 50.5GB/915.8GB
|
||||
Repos:
|
||||
● gitea (4) 5.1GB
|
||||
● immich (4) 45.0GB
|
||||
● kryddorten (4) 67.8MB
|
||||
● mariehall2 (4) 322.7MB
|
||||
● nixosbox (4) 5.5MB
|
||||
● unifi (4) 5.7MB
|
||||
● vaultwarden (4) 508kB
|
||||
'''
|
||||
|
||||
## Keyboard navigation and scrolling (dashboard)
|
||||
|
||||
- Add keyboard navigation between panels "Shift-Tab"
|
||||
- Add lower statusbar with dynamic updated shortcuts when switchng between panels
|
||||
|
||||
## Remote execution (agent/dashboard)
|
||||
|
||||
- Add support for send command via dashboard to agent to do nixos rebuid
|
||||
- Add support for navigating services in dashboard and trigger start/stop/restart
|
||||
- Add support for trigger backup
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "cm-dashboard-agent"
|
||||
version = "0.1.29"
|
||||
version = "0.1.131"
|
||||
edition = "2021"
|
||||
|
||||
[dependencies]
|
||||
|
||||
@@ -4,12 +4,13 @@ use std::time::Duration;
|
||||
use tokio::time::interval;
|
||||
use tracing::{debug, error, info};
|
||||
|
||||
use crate::communication::{AgentCommand, ServiceAction, ZmqHandler};
|
||||
use crate::communication::{AgentCommand, ZmqHandler};
|
||||
use crate::config::AgentConfig;
|
||||
use crate::metrics::MetricCollectionManager;
|
||||
use crate::notifications::NotificationManager;
|
||||
use crate::service_tracker::UserStoppedServiceTracker;
|
||||
use crate::status::HostStatusManager;
|
||||
use cm_dashboard_shared::{Metric, MetricMessage, MetricValue, Status};
|
||||
use cm_dashboard_shared::{AgentData, Metric, MetricValue, Status, TmpfsData, DriveData, FilesystemData, ServiceData};
|
||||
|
||||
pub struct Agent {
|
||||
hostname: String,
|
||||
@@ -18,6 +19,7 @@ pub struct Agent {
|
||||
metric_manager: MetricCollectionManager,
|
||||
notification_manager: NotificationManager,
|
||||
host_status_manager: HostStatusManager,
|
||||
service_tracker: UserStoppedServiceTracker,
|
||||
}
|
||||
|
||||
impl Agent {
|
||||
@@ -50,6 +52,10 @@ impl Agent {
|
||||
let host_status_manager = HostStatusManager::new(config.status_aggregation.clone());
|
||||
info!("Host status manager initialized");
|
||||
|
||||
// Initialize user-stopped service tracker
|
||||
let service_tracker = UserStoppedServiceTracker::init_global()?;
|
||||
info!("User-stopped service tracker initialized");
|
||||
|
||||
Ok(Self {
|
||||
hostname,
|
||||
config,
|
||||
@@ -57,6 +63,7 @@ impl Agent {
|
||||
metric_manager,
|
||||
notification_manager,
|
||||
host_status_manager,
|
||||
service_tracker,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -71,10 +78,11 @@ impl Agent {
|
||||
info!("Initial metric collection completed - all data cached and ready");
|
||||
}
|
||||
|
||||
// Separate intervals for collection, transmission, and email notifications
|
||||
// Separate intervals for collection, transmission, heartbeat, and email notifications
|
||||
let mut collection_interval =
|
||||
interval(Duration::from_secs(self.config.collection_interval_seconds));
|
||||
let mut transmission_interval = interval(Duration::from_secs(self.config.zmq.transmission_interval_seconds));
|
||||
let mut heartbeat_interval = interval(Duration::from_secs(self.config.zmq.heartbeat_interval_seconds));
|
||||
let mut notification_interval = interval(Duration::from_secs(self.config.notifications.aggregation_interval_seconds));
|
||||
|
||||
loop {
|
||||
@@ -91,6 +99,12 @@ impl Agent {
|
||||
error!("Failed to broadcast metrics: {}", e);
|
||||
}
|
||||
}
|
||||
_ = heartbeat_interval.tick() => {
|
||||
// Send standalone heartbeat for host connectivity detection
|
||||
if let Err(e) = self.send_heartbeat().await {
|
||||
error!("Failed to send heartbeat: {}", e);
|
||||
}
|
||||
}
|
||||
_ = notification_interval.tick() => {
|
||||
// Process batched email notifications (separate from dashboard updates)
|
||||
if let Err(e) = self.host_status_manager.process_pending_notifications(&mut self.notification_manager).await {
|
||||
@@ -173,24 +187,331 @@ impl Agent {
|
||||
let version_metric = self.get_agent_version_metric();
|
||||
metrics.push(version_metric);
|
||||
|
||||
// Add heartbeat metric for host connectivity detection
|
||||
let heartbeat_metric = self.get_heartbeat_metric();
|
||||
metrics.push(heartbeat_metric);
|
||||
|
||||
// Check for user-stopped services that are now active and clear their flags
|
||||
self.clear_user_stopped_flags_for_active_services(&metrics);
|
||||
|
||||
if metrics.is_empty() {
|
||||
debug!("No metrics to broadcast");
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
debug!("Broadcasting {} cached metrics (including host status summary)", metrics.len());
|
||||
debug!("Broadcasting {} cached metrics as structured data", metrics.len());
|
||||
|
||||
// Create and send message with all current data
|
||||
let message = MetricMessage::new(self.hostname.clone(), metrics);
|
||||
self.zmq_handler.publish_metrics(&message).await?;
|
||||
// Convert metrics to structured data and send
|
||||
let agent_data = self.metrics_to_structured_data(&metrics)?;
|
||||
self.zmq_handler.publish_agent_data(&agent_data).await?;
|
||||
|
||||
debug!("Metrics broadcasted successfully");
|
||||
debug!("Structured data broadcasted successfully");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Convert legacy metrics to structured data format
|
||||
fn metrics_to_structured_data(&self, metrics: &[Metric]) -> Result<AgentData> {
|
||||
let mut agent_data = AgentData::new(self.hostname.clone(), self.get_agent_version());
|
||||
|
||||
// Parse metrics into structured data
|
||||
for metric in metrics {
|
||||
self.parse_metric_into_agent_data(&mut agent_data, metric)?;
|
||||
}
|
||||
|
||||
Ok(agent_data)
|
||||
}
|
||||
|
||||
/// Parse a single metric into the appropriate structured data field
|
||||
fn parse_metric_into_agent_data(&self, agent_data: &mut AgentData, metric: &Metric) -> Result<()> {
|
||||
// CPU metrics
|
||||
if metric.name == "cpu_load_1min" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
agent_data.system.cpu.load_1min = value;
|
||||
}
|
||||
} else if metric.name == "cpu_load_5min" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
agent_data.system.cpu.load_5min = value;
|
||||
}
|
||||
} else if metric.name == "cpu_load_15min" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
agent_data.system.cpu.load_15min = value;
|
||||
}
|
||||
} else if metric.name == "cpu_frequency_mhz" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
agent_data.system.cpu.frequency_mhz = value;
|
||||
}
|
||||
} else if metric.name == "cpu_temperature_celsius" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
agent_data.system.cpu.temperature_celsius = Some(value);
|
||||
}
|
||||
}
|
||||
// Memory metrics
|
||||
else if metric.name == "memory_usage_percent" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
agent_data.system.memory.usage_percent = value;
|
||||
}
|
||||
} else if metric.name == "memory_total_gb" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
agent_data.system.memory.total_gb = value;
|
||||
}
|
||||
} else if metric.name == "memory_used_gb" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
agent_data.system.memory.used_gb = value;
|
||||
}
|
||||
} else if metric.name == "memory_available_gb" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
agent_data.system.memory.available_gb = value;
|
||||
}
|
||||
} else if metric.name == "memory_swap_total_gb" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
agent_data.system.memory.swap_total_gb = value;
|
||||
}
|
||||
} else if metric.name == "memory_swap_used_gb" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
agent_data.system.memory.swap_used_gb = value;
|
||||
}
|
||||
}
|
||||
// Tmpfs metrics
|
||||
else if metric.name.starts_with("memory_tmp_") {
|
||||
// For now, create a single /tmp tmpfs entry
|
||||
if metric.name == "memory_tmp_usage_percent" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
if let Some(tmpfs) = agent_data.system.memory.tmpfs.get_mut(0) {
|
||||
tmpfs.usage_percent = value;
|
||||
} else {
|
||||
agent_data.system.memory.tmpfs.push(TmpfsData {
|
||||
mount: "/tmp".to_string(),
|
||||
usage_percent: value,
|
||||
used_gb: 0.0,
|
||||
total_gb: 0.0,
|
||||
});
|
||||
}
|
||||
}
|
||||
} else if metric.name == "memory_tmp_used_gb" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
if let Some(tmpfs) = agent_data.system.memory.tmpfs.get_mut(0) {
|
||||
tmpfs.used_gb = value;
|
||||
} else {
|
||||
agent_data.system.memory.tmpfs.push(TmpfsData {
|
||||
mount: "/tmp".to_string(),
|
||||
usage_percent: 0.0,
|
||||
used_gb: value,
|
||||
total_gb: 0.0,
|
||||
});
|
||||
}
|
||||
}
|
||||
} else if metric.name == "memory_tmp_total_gb" {
|
||||
if let Some(value) = metric.value.as_f32() {
|
||||
if let Some(tmpfs) = agent_data.system.memory.tmpfs.get_mut(0) {
|
||||
tmpfs.total_gb = value;
|
||||
} else {
|
||||
agent_data.system.memory.tmpfs.push(TmpfsData {
|
||||
mount: "/tmp".to_string(),
|
||||
usage_percent: 0.0,
|
||||
used_gb: 0.0,
|
||||
total_gb: value,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Storage metrics
|
||||
else if metric.name.starts_with("disk_") {
|
||||
if metric.name.contains("_temperature") {
|
||||
if let Some(drive_name) = self.extract_drive_name(&metric.name) {
|
||||
if let Some(temp) = metric.value.as_f32() {
|
||||
self.ensure_drive_exists(agent_data, &drive_name);
|
||||
if let Some(drive) = agent_data.system.storage.drives.iter_mut().find(|d| d.name == drive_name) {
|
||||
drive.temperature_celsius = Some(temp);
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if metric.name.contains("_wear_percent") {
|
||||
if let Some(drive_name) = self.extract_drive_name(&metric.name) {
|
||||
if let Some(wear) = metric.value.as_f32() {
|
||||
self.ensure_drive_exists(agent_data, &drive_name);
|
||||
if let Some(drive) = agent_data.system.storage.drives.iter_mut().find(|d| d.name == drive_name) {
|
||||
drive.wear_percent = Some(wear);
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if metric.name.contains("_health") {
|
||||
if let Some(drive_name) = self.extract_drive_name(&metric.name) {
|
||||
let health = metric.value.as_string();
|
||||
self.ensure_drive_exists(agent_data, &drive_name);
|
||||
if let Some(drive) = agent_data.system.storage.drives.iter_mut().find(|d| d.name == drive_name) {
|
||||
drive.health = health;
|
||||
}
|
||||
}
|
||||
} else if metric.name.contains("_fs_") {
|
||||
// Filesystem metrics: disk_{pool}_fs_{filesystem}_{metric}
|
||||
if let Some((pool_name, fs_name)) = self.extract_pool_and_filesystem(&metric.name) {
|
||||
if metric.name.contains("_usage_percent") {
|
||||
if let Some(usage) = metric.value.as_f32() {
|
||||
self.ensure_filesystem_exists(agent_data, &pool_name, &fs_name, usage, 0.0, 0.0);
|
||||
}
|
||||
} else if metric.name.contains("_used_gb") {
|
||||
if let Some(used) = metric.value.as_f32() {
|
||||
self.update_filesystem_field(agent_data, &pool_name, &fs_name, |fs| fs.used_gb = used);
|
||||
}
|
||||
} else if metric.name.contains("_total_gb") {
|
||||
if let Some(total) = metric.value.as_f32() {
|
||||
self.update_filesystem_field(agent_data, &pool_name, &fs_name, |fs| fs.total_gb = total);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Service metrics
|
||||
else if metric.name.starts_with("service_") {
|
||||
if let Some(service_name) = self.extract_service_name(&metric.name) {
|
||||
if metric.name.contains("_status") {
|
||||
let status = metric.value.as_string();
|
||||
self.ensure_service_exists(agent_data, &service_name, &status);
|
||||
} else if metric.name.contains("_memory_mb") {
|
||||
if let Some(memory) = metric.value.as_f32() {
|
||||
self.update_service_field(agent_data, &service_name, |svc| svc.memory_mb = memory);
|
||||
}
|
||||
} else if metric.name.contains("_disk_gb") {
|
||||
if let Some(disk) = metric.value.as_f32() {
|
||||
self.update_service_field(agent_data, &service_name, |svc| svc.disk_gb = disk);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Backup metrics
|
||||
else if metric.name.starts_with("backup_") {
|
||||
if metric.name == "backup_status" {
|
||||
agent_data.backup.status = metric.value.as_string();
|
||||
} else if metric.name == "backup_last_run_timestamp" {
|
||||
if let Some(timestamp) = metric.value.as_i64() {
|
||||
agent_data.backup.last_run = Some(timestamp as u64);
|
||||
}
|
||||
} else if metric.name == "backup_next_scheduled_timestamp" {
|
||||
if let Some(timestamp) = metric.value.as_i64() {
|
||||
agent_data.backup.next_scheduled = Some(timestamp as u64);
|
||||
}
|
||||
} else if metric.name == "backup_size_gb" {
|
||||
if let Some(size) = metric.value.as_f32() {
|
||||
agent_data.backup.total_size_gb = Some(size);
|
||||
}
|
||||
} else if metric.name == "backup_repository_health" {
|
||||
agent_data.backup.repository_health = Some(metric.value.as_string());
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Extract drive name from metric like "disk_nvme0n1_temperature"
|
||||
fn extract_drive_name(&self, metric_name: &str) -> Option<String> {
|
||||
if metric_name.starts_with("disk_") {
|
||||
let suffixes = ["_temperature", "_wear_percent", "_health"];
|
||||
for suffix in suffixes {
|
||||
if let Some(suffix_pos) = metric_name.rfind(suffix) {
|
||||
return Some(metric_name[5..suffix_pos].to_string()); // Skip "disk_"
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Extract pool and filesystem from "disk_{pool}_fs_{filesystem}_{metric}"
|
||||
fn extract_pool_and_filesystem(&self, metric_name: &str) -> Option<(String, String)> {
|
||||
if let Some(fs_pos) = metric_name.find("_fs_") {
|
||||
let pool_name = metric_name[5..fs_pos].to_string(); // Skip "disk_"
|
||||
let after_fs = &metric_name[fs_pos + 4..]; // Skip "_fs_"
|
||||
if let Some(metric_pos) = after_fs.find('_') {
|
||||
let fs_name = after_fs[..metric_pos].to_string();
|
||||
return Some((pool_name, fs_name));
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Extract service name from "service_{name}_{metric}"
|
||||
fn extract_service_name(&self, metric_name: &str) -> Option<String> {
|
||||
if metric_name.starts_with("service_") {
|
||||
let suffixes = ["_status", "_memory_mb", "_disk_gb"];
|
||||
for suffix in suffixes {
|
||||
if let Some(suffix_pos) = metric_name.rfind(suffix) {
|
||||
return Some(metric_name[8..suffix_pos].to_string()); // Skip "service_"
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Ensure drive exists in agent_data
|
||||
fn ensure_drive_exists(&self, agent_data: &mut AgentData, drive_name: &str) {
|
||||
if !agent_data.system.storage.drives.iter().any(|d| d.name == drive_name) {
|
||||
agent_data.system.storage.drives.push(DriveData {
|
||||
name: drive_name.to_string(),
|
||||
health: "UNKNOWN".to_string(),
|
||||
temperature_celsius: None,
|
||||
wear_percent: None,
|
||||
filesystems: Vec::new(),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/// Ensure filesystem exists in the correct drive
|
||||
fn ensure_filesystem_exists(&self, agent_data: &mut AgentData, pool_name: &str, fs_name: &str, usage_percent: f32, used_gb: f32, total_gb: f32) {
|
||||
self.ensure_drive_exists(agent_data, pool_name);
|
||||
if let Some(drive) = agent_data.system.storage.drives.iter_mut().find(|d| d.name == pool_name) {
|
||||
if !drive.filesystems.iter().any(|fs| fs.mount == fs_name) {
|
||||
drive.filesystems.push(FilesystemData {
|
||||
mount: fs_name.to_string(),
|
||||
usage_percent,
|
||||
used_gb,
|
||||
total_gb,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Update filesystem field
|
||||
fn update_filesystem_field<F>(&self, agent_data: &mut AgentData, pool_name: &str, fs_name: &str, update_fn: F)
|
||||
where F: FnOnce(&mut FilesystemData) {
|
||||
if let Some(drive) = agent_data.system.storage.drives.iter_mut().find(|d| d.name == pool_name) {
|
||||
if let Some(fs) = drive.filesystems.iter_mut().find(|fs| fs.mount == fs_name) {
|
||||
update_fn(fs);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Ensure service exists
|
||||
fn ensure_service_exists(&self, agent_data: &mut AgentData, service_name: &str, status: &str) {
|
||||
if !agent_data.services.iter().any(|s| s.name == service_name) {
|
||||
agent_data.services.push(ServiceData {
|
||||
name: service_name.to_string(),
|
||||
status: status.to_string(),
|
||||
memory_mb: 0.0,
|
||||
disk_gb: 0.0,
|
||||
user_stopped: false, // TODO: Get from service tracker
|
||||
});
|
||||
} else if let Some(service) = agent_data.services.iter_mut().find(|s| s.name == service_name) {
|
||||
service.status = status.to_string();
|
||||
}
|
||||
}
|
||||
|
||||
/// Update service field
|
||||
fn update_service_field<F>(&self, agent_data: &mut AgentData, service_name: &str, update_fn: F)
|
||||
where F: FnOnce(&mut ServiceData) {
|
||||
if let Some(service) = agent_data.services.iter_mut().find(|s| s.name == service_name) {
|
||||
update_fn(service);
|
||||
}
|
||||
}
|
||||
|
||||
async fn process_metrics(&mut self, metrics: &[Metric]) -> bool {
|
||||
let mut status_changed = false;
|
||||
for metric in metrics {
|
||||
// Filter excluded metrics from email notification processing only
|
||||
if self.config.notifications.exclude_email_metrics.contains(&metric.name) {
|
||||
debug!("Excluding metric '{}' from email notification processing", metric.name);
|
||||
continue;
|
||||
}
|
||||
|
||||
if self.host_status_manager.process_metric(metric, &mut self.notification_manager).await {
|
||||
status_changed = true;
|
||||
}
|
||||
@@ -216,6 +537,33 @@ impl Agent {
|
||||
format!("v{}", env!("CARGO_PKG_VERSION"))
|
||||
}
|
||||
|
||||
/// Create heartbeat metric for host connectivity detection
|
||||
fn get_heartbeat_metric(&self) -> Metric {
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
let timestamp = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_secs();
|
||||
|
||||
Metric::new(
|
||||
"agent_heartbeat".to_string(),
|
||||
MetricValue::Integer(timestamp as i64),
|
||||
Status::Ok,
|
||||
)
|
||||
}
|
||||
|
||||
/// Send standalone heartbeat for connectivity detection
|
||||
async fn send_heartbeat(&mut self) -> Result<()> {
|
||||
// Create minimal agent data with just heartbeat
|
||||
let agent_data = AgentData::new(self.hostname.clone(), self.get_agent_version());
|
||||
// Heartbeat timestamp is already set in AgentData::new()
|
||||
|
||||
self.zmq_handler.publish_agent_data(&agent_data).await?;
|
||||
debug!("Sent standalone heartbeat for connectivity detection");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn handle_commands(&mut self) -> Result<()> {
|
||||
// Try to receive commands (non-blocking)
|
||||
match self.zmq_handler.try_receive_command() {
|
||||
@@ -259,56 +607,38 @@ impl Agent {
|
||||
info!("Processing Ping command - agent is alive");
|
||||
// Could send a response back via ZMQ if needed
|
||||
}
|
||||
AgentCommand::ServiceControl { service_name, action } => {
|
||||
info!("Processing ServiceControl command: {} {:?}", service_name, action);
|
||||
if let Err(e) = self.handle_service_control(&service_name, &action).await {
|
||||
error!("Failed to execute service control: {}", e);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
/// Check metrics for user-stopped services that are now active and clear their flags
|
||||
fn clear_user_stopped_flags_for_active_services(&mut self, metrics: &[Metric]) {
|
||||
for metric in metrics {
|
||||
// Look for service status metrics that are active
|
||||
if metric.name.starts_with("service_") && metric.name.ends_with("_status") {
|
||||
if let MetricValue::String(status) = &metric.value {
|
||||
if status == "active" {
|
||||
// Extract service name from metric name (service_nginx_status -> nginx)
|
||||
let service_name = metric.name
|
||||
.strip_prefix("service_")
|
||||
.and_then(|s| s.strip_suffix("_status"))
|
||||
.unwrap_or("");
|
||||
|
||||
if !service_name.is_empty() && UserStoppedServiceTracker::is_service_user_stopped(service_name) {
|
||||
info!("Service '{}' is now active - clearing user-stopped flag", service_name);
|
||||
if let Err(e) = self.service_tracker.clear_user_stopped(service_name) {
|
||||
error!("Failed to clear user-stopped flag for '{}': {}", service_name, e);
|
||||
} else {
|
||||
// Sync to global tracker
|
||||
UserStoppedServiceTracker::update_global(&self.service_tracker);
|
||||
debug!("Cleared user-stopped flag for service '{}'", service_name);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Handle systemd service control commands
|
||||
async fn handle_service_control(&mut self, service_name: &str, action: &ServiceAction) -> Result<()> {
|
||||
let action_str = match action {
|
||||
ServiceAction::Start => "start",
|
||||
ServiceAction::Stop => "stop",
|
||||
ServiceAction::Restart => "restart",
|
||||
ServiceAction::Status => "status",
|
||||
};
|
||||
|
||||
info!("Executing systemctl {} {}", action_str, service_name);
|
||||
|
||||
let output = tokio::process::Command::new("sudo")
|
||||
.arg("systemctl")
|
||||
.arg(action_str)
|
||||
.arg(service_name)
|
||||
.output()
|
||||
.await?;
|
||||
|
||||
if output.status.success() {
|
||||
info!("Service {} {} completed successfully", service_name, action_str);
|
||||
if !output.stdout.is_empty() {
|
||||
debug!("stdout: {}", String::from_utf8_lossy(&output.stdout));
|
||||
}
|
||||
} else {
|
||||
let stderr = String::from_utf8_lossy(&output.stderr);
|
||||
error!("Service {} {} failed: {}", service_name, action_str, stderr);
|
||||
return Err(anyhow::anyhow!("systemctl {} {} failed: {}", action_str, service_name, stderr));
|
||||
}
|
||||
|
||||
// Force refresh metrics after service control to update service status
|
||||
if matches!(action, ServiceAction::Start | ServiceAction::Stop | ServiceAction::Restart) {
|
||||
info!("Triggering immediate metric refresh after service control");
|
||||
if let Err(e) = self.collect_metrics_only().await {
|
||||
error!("Failed to refresh metrics after service control: {}", e);
|
||||
} else {
|
||||
info!("Service status refreshed immediately after {} {}", action_str, service_name);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
}
|
||||
@@ -25,6 +25,25 @@ impl BackupCollector {
|
||||
}
|
||||
|
||||
async fn read_backup_status(&self) -> Result<Option<BackupStatusToml>, CollectorError> {
|
||||
// Check if we're in maintenance mode
|
||||
if std::fs::metadata("/tmp/cm-maintenance").is_ok() {
|
||||
// Return special maintenance mode status
|
||||
let maintenance_status = BackupStatusToml {
|
||||
backup_name: "maintenance".to_string(),
|
||||
start_time: chrono::Utc::now().format("%Y-%m-%d %H:%M:%S UTC").to_string(),
|
||||
current_time: chrono::Utc::now().format("%Y-%m-%d %H:%M:%S UTC").to_string(),
|
||||
duration_seconds: 0,
|
||||
status: "pending".to_string(),
|
||||
last_updated: chrono::Utc::now().format("%Y-%m-%d %H:%M:%S UTC").to_string(),
|
||||
disk_space: None,
|
||||
disk_product_name: None,
|
||||
disk_serial_number: None,
|
||||
disk_wear_percent: None,
|
||||
services: HashMap::new(),
|
||||
};
|
||||
return Ok(Some(maintenance_status));
|
||||
}
|
||||
|
||||
// Check if backup status file exists
|
||||
if !std::path::Path::new(&self.backup_status_file).exists() {
|
||||
return Ok(None); // File doesn't exist, but this is not an error
|
||||
@@ -79,7 +98,9 @@ impl BackupCollector {
|
||||
}
|
||||
}
|
||||
"failed" => Status::Critical,
|
||||
"warning" => Status::Warning, // Backup completed with warnings
|
||||
"running" => Status::Ok, // Currently running is OK
|
||||
"pending" => Status::Pending, // Maintenance mode or backup starting
|
||||
_ => Status::Unknown,
|
||||
}
|
||||
}
|
||||
@@ -136,10 +157,12 @@ impl Collector for BackupCollector {
|
||||
name: "backup_overall_status".to_string(),
|
||||
value: MetricValue::String(match overall_status {
|
||||
Status::Ok => "ok".to_string(),
|
||||
Status::Inactive => "inactive".to_string(),
|
||||
Status::Pending => "pending".to_string(),
|
||||
Status::Warning => "warning".to_string(),
|
||||
Status::Critical => "critical".to_string(),
|
||||
Status::Unknown => "unknown".to_string(),
|
||||
Status::Offline => "offline".to_string(),
|
||||
}),
|
||||
status: overall_status,
|
||||
timestamp,
|
||||
@@ -198,10 +221,12 @@ impl Collector for BackupCollector {
|
||||
name: format!("backup_service_{}_status", service_name),
|
||||
value: MetricValue::String(match service_status {
|
||||
Status::Ok => "ok".to_string(),
|
||||
Status::Inactive => "inactive".to_string(),
|
||||
Status::Pending => "pending".to_string(),
|
||||
Status::Warning => "warning".to_string(),
|
||||
Status::Critical => "critical".to_string(),
|
||||
Status::Unknown => "unknown".to_string(),
|
||||
Status::Offline => "offline".to_string(),
|
||||
}),
|
||||
status: service_status,
|
||||
timestamp,
|
||||
@@ -375,6 +400,25 @@ impl Collector for BackupCollector {
|
||||
});
|
||||
}
|
||||
|
||||
if let Some(wear_percent) = backup_status.disk_wear_percent {
|
||||
let wear_status = if wear_percent >= 90.0 {
|
||||
Status::Critical
|
||||
} else if wear_percent >= 75.0 {
|
||||
Status::Warning
|
||||
} else {
|
||||
Status::Ok
|
||||
};
|
||||
|
||||
metrics.push(Metric {
|
||||
name: "backup_disk_wear_percent".to_string(),
|
||||
value: MetricValue::Float(wear_percent),
|
||||
status: wear_status,
|
||||
timestamp,
|
||||
description: Some("Backup disk wear percentage from SMART data".to_string()),
|
||||
unit: Some("percent".to_string()),
|
||||
});
|
||||
}
|
||||
|
||||
// Count services by status
|
||||
let mut status_counts = HashMap::new();
|
||||
for service in backup_status.services.values() {
|
||||
@@ -408,6 +452,7 @@ pub struct BackupStatusToml {
|
||||
pub disk_space: Option<DiskSpace>,
|
||||
pub disk_product_name: Option<String>,
|
||||
pub disk_serial_number: Option<String>,
|
||||
pub disk_wear_percent: Option<f32>,
|
||||
pub services: HashMap<String, ServiceStatus>,
|
||||
}
|
||||
|
||||
|
||||
@@ -5,353 +5,159 @@ use cm_dashboard_shared::{Metric, MetricValue, Status, StatusTracker, Hysteresis
|
||||
use crate::config::DiskConfig;
|
||||
use std::process::Command;
|
||||
use std::time::Instant;
|
||||
use std::collections::HashMap;
|
||||
use tracing::debug;
|
||||
|
||||
use super::{Collector, CollectorError};
|
||||
|
||||
/// Information about a storage pool (mount point with underlying drives)
|
||||
#[derive(Debug, Clone)]
|
||||
struct StoragePool {
|
||||
name: String, // e.g., "steampool", "root"
|
||||
mount_point: String, // e.g., "/mnt/steampool", "/"
|
||||
filesystem: String, // e.g., "mergerfs", "ext4", "zfs", "btrfs"
|
||||
storage_type: String, // e.g., "mergerfs", "single", "raid", "zfs"
|
||||
size: String, // e.g., "2.5TB"
|
||||
used: String, // e.g., "2.1TB"
|
||||
available: String, // e.g., "400GB"
|
||||
usage_percent: f32, // e.g., 85.0
|
||||
underlying_drives: Vec<DriveInfo>, // Individual physical drives
|
||||
}
|
||||
|
||||
/// Information about an individual physical drive
|
||||
#[derive(Debug, Clone)]
|
||||
struct DriveInfo {
|
||||
device: String, // e.g., "sda", "nvme0n1"
|
||||
health_status: String, // e.g., "PASSED", "FAILED"
|
||||
temperature: Option<f32>, // e.g., 45.0°C
|
||||
wear_level: Option<f32>, // e.g., 12.0% (for SSDs)
|
||||
}
|
||||
|
||||
/// Disk usage collector for monitoring filesystem sizes
|
||||
/// Storage collector with clean architecture
|
||||
pub struct DiskCollector {
|
||||
config: DiskConfig,
|
||||
temperature_thresholds: HysteresisThresholds,
|
||||
detected_devices: std::collections::HashMap<String, Vec<String>>, // mount_point -> devices
|
||||
}
|
||||
|
||||
/// A physical drive with its filesystems
|
||||
#[derive(Debug, Clone)]
|
||||
struct PhysicalDrive {
|
||||
device: String, // e.g., "nvme0n1", "sda"
|
||||
filesystems: Vec<Filesystem>, // mounted filesystems on this drive
|
||||
temperature: Option<f32>, // drive temperature
|
||||
wear_level: Option<f32>, // SSD wear level
|
||||
health_status: String, // SMART health
|
||||
}
|
||||
|
||||
/// A mergerfs pool
|
||||
#[derive(Debug, Clone)]
|
||||
struct MergerfsPool {
|
||||
mount_point: String, // e.g., "/srv/media"
|
||||
total_bytes: u64, // pool total capacity
|
||||
used_bytes: u64, // pool used space
|
||||
data_drives: Vec<DriveInfo>, // data member drives
|
||||
parity_drives: Vec<DriveInfo>, // parity drives
|
||||
}
|
||||
|
||||
/// Individual filesystem on a drive
|
||||
#[derive(Debug, Clone)]
|
||||
struct Filesystem {
|
||||
mount_point: String, // e.g., "/", "/boot"
|
||||
total_bytes: u64, // filesystem capacity
|
||||
used_bytes: u64, // filesystem used space
|
||||
}
|
||||
|
||||
/// Drive information for pools
|
||||
#[derive(Debug, Clone)]
|
||||
struct DriveInfo {
|
||||
device: String, // e.g., "sdb", "sdc"
|
||||
mount_point: String, // e.g., "/mnt/disk1"
|
||||
temperature: Option<f32>, // drive temperature
|
||||
wear_level: Option<f32>, // SSD wear level
|
||||
health_status: String, // SMART health
|
||||
}
|
||||
|
||||
/// Discovered storage topology
|
||||
#[derive(Debug)]
|
||||
struct StorageTopology {
|
||||
physical_drives: Vec<PhysicalDrive>,
|
||||
mergerfs_pools: Vec<MergerfsPool>,
|
||||
}
|
||||
|
||||
impl DiskCollector {
|
||||
pub fn new(config: DiskConfig) -> Self {
|
||||
// Create hysteresis thresholds for disk temperature from config
|
||||
let temperature_thresholds = HysteresisThresholds::with_custom_gaps(
|
||||
config.temperature_warning_celsius,
|
||||
5.0, // 5°C gap for recovery
|
||||
5.0,
|
||||
config.temperature_critical_celsius,
|
||||
5.0, // 5°C gap for recovery
|
||||
5.0,
|
||||
);
|
||||
|
||||
// Detect devices for all configured filesystems at startup
|
||||
let mut detected_devices = std::collections::HashMap::new();
|
||||
for fs_config in &config.filesystems {
|
||||
if fs_config.monitor {
|
||||
if let Ok(devices) = Self::detect_device_for_mount_point_static(&fs_config.mount_point) {
|
||||
detected_devices.insert(fs_config.mount_point.clone(), devices);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Self {
|
||||
config,
|
||||
temperature_thresholds,
|
||||
detected_devices,
|
||||
}
|
||||
}
|
||||
|
||||
/// Calculate disk temperature status using hysteresis thresholds
|
||||
fn calculate_temperature_status(&self, metric_name: &str, temperature: f32, status_tracker: &mut StatusTracker) -> Status {
|
||||
status_tracker.calculate_with_hysteresis(metric_name, temperature, &self.temperature_thresholds)
|
||||
/// Discover all storage using clean workflow: lsblk → df → group
|
||||
fn discover_storage(&self) -> Result<StorageTopology> {
|
||||
debug!("Starting storage discovery");
|
||||
|
||||
// Step 1: Get all mount points and their backing devices using lsblk
|
||||
let mount_devices = self.get_mount_devices()?;
|
||||
debug!("Found {} mount points", mount_devices.len());
|
||||
|
||||
// Step 2: Get filesystem usage for each mount point using df
|
||||
let filesystem_usage = self.get_filesystem_usage(&mount_devices)?;
|
||||
debug!("Got usage data for {} filesystems", filesystem_usage.len());
|
||||
|
||||
// Step 3: Detect mergerfs pools from /proc/mounts
|
||||
let mergerfs_pools = self.discover_mergerfs_pools()?;
|
||||
debug!("Found {} mergerfs pools", mergerfs_pools.len());
|
||||
|
||||
// Step 4: Group regular filesystems by physical drive
|
||||
let physical_drives = self.group_by_physical_drive(&mount_devices, &filesystem_usage, &mergerfs_pools)?;
|
||||
debug!("Grouped into {} physical drives", physical_drives.len());
|
||||
|
||||
Ok(StorageTopology {
|
||||
physical_drives,
|
||||
mergerfs_pools,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
/// Get configured storage pools with individual drive information
|
||||
fn get_configured_storage_pools(&self) -> Result<Vec<StoragePool>> {
|
||||
let mut storage_pools = Vec::new();
|
||||
|
||||
for fs_config in &self.config.filesystems {
|
||||
if !fs_config.monitor {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Get filesystem stats for the mount point
|
||||
match self.get_filesystem_info(&fs_config.mount_point) {
|
||||
Ok((total_bytes, used_bytes)) => {
|
||||
let available_bytes = total_bytes - used_bytes;
|
||||
let usage_percent = if total_bytes > 0 {
|
||||
(used_bytes as f64 / total_bytes as f64) * 100.0
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
|
||||
// Convert bytes to human-readable format
|
||||
let size = self.bytes_to_human_readable(total_bytes);
|
||||
let used = self.bytes_to_human_readable(used_bytes);
|
||||
let available = self.bytes_to_human_readable(available_bytes);
|
||||
|
||||
// Get individual drive information using pre-detected devices
|
||||
let device_names = self.detected_devices.get(&fs_config.mount_point).cloned().unwrap_or_default();
|
||||
let underlying_drives = self.get_drive_info_for_devices(&device_names)?;
|
||||
|
||||
storage_pools.push(StoragePool {
|
||||
name: fs_config.name.clone(),
|
||||
mount_point: fs_config.mount_point.clone(),
|
||||
filesystem: fs_config.fs_type.clone(),
|
||||
storage_type: fs_config.storage_type.clone(),
|
||||
size,
|
||||
used,
|
||||
available,
|
||||
usage_percent: usage_percent as f32,
|
||||
underlying_drives,
|
||||
});
|
||||
|
||||
debug!(
|
||||
"Storage pool '{}' ({}) at {} with {} detected drives",
|
||||
fs_config.name, fs_config.storage_type, fs_config.mount_point, device_names.len()
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
debug!(
|
||||
"Failed to get filesystem info for storage pool '{}': {}",
|
||||
fs_config.name, e
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(storage_pools)
|
||||
}
|
||||
|
||||
/// Get drive information for a list of device names
|
||||
fn get_drive_info_for_devices(&self, device_names: &[String]) -> Result<Vec<DriveInfo>> {
|
||||
let mut drives = Vec::new();
|
||||
|
||||
for device_name in device_names {
|
||||
let device_path = format!("/dev/{}", device_name);
|
||||
|
||||
// Get SMART data for this drive
|
||||
let (health_status, temperature, wear_level) = self.get_smart_data(&device_path);
|
||||
|
||||
drives.push(DriveInfo {
|
||||
device: device_name.clone(),
|
||||
health_status: health_status.clone(),
|
||||
temperature,
|
||||
wear_level,
|
||||
});
|
||||
|
||||
debug!(
|
||||
"Drive info for {}: health={}, temp={:?}°C, wear={:?}%",
|
||||
device_name, health_status, temperature, wear_level
|
||||
);
|
||||
}
|
||||
|
||||
Ok(drives)
|
||||
}
|
||||
|
||||
/// Get SMART data for a drive (health, temperature, wear level)
|
||||
fn get_smart_data(&self, device_path: &str) -> (String, Option<f32>, Option<f32>) {
|
||||
// Try to get SMART data using smartctl
|
||||
let output = Command::new("sudo")
|
||||
.arg("smartctl")
|
||||
.arg("-a")
|
||||
.arg(device_path)
|
||||
.output();
|
||||
|
||||
match output {
|
||||
Ok(result) if result.status.success() => {
|
||||
let stdout = String::from_utf8_lossy(&result.stdout);
|
||||
|
||||
// Parse health status
|
||||
let health = if stdout.contains("PASSED") {
|
||||
"PASSED".to_string()
|
||||
} else if stdout.contains("FAILED") {
|
||||
"FAILED".to_string()
|
||||
} else {
|
||||
"UNKNOWN".to_string()
|
||||
};
|
||||
|
||||
// Parse temperature (look for various temperature indicators)
|
||||
let temperature = self.parse_temperature_from_smart(&stdout);
|
||||
|
||||
// Parse wear level (for SSDs)
|
||||
let wear_level = self.parse_wear_level_from_smart(&stdout);
|
||||
|
||||
(health, temperature, wear_level)
|
||||
}
|
||||
_ => {
|
||||
debug!("Failed to get SMART data for {}", device_path);
|
||||
("UNKNOWN".to_string(), None, None)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse temperature from SMART output
|
||||
fn parse_temperature_from_smart(&self, smart_output: &str) -> Option<f32> {
|
||||
for line in smart_output.lines() {
|
||||
// Look for temperature in various formats
|
||||
if line.contains("Temperature_Celsius") || line.contains("Temperature") {
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
if parts.len() >= 10 {
|
||||
if let Ok(temp) = parts[9].parse::<f32>() {
|
||||
return Some(temp);
|
||||
}
|
||||
}
|
||||
}
|
||||
// NVMe drives might show temperature differently
|
||||
if line.contains("temperature:") {
|
||||
if let Some(temp_part) = line.split("temperature:").nth(1) {
|
||||
if let Some(temp_str) = temp_part.split_whitespace().next() {
|
||||
if let Ok(temp) = temp_str.parse::<f32>() {
|
||||
return Some(temp);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Parse wear level from SMART output (SSD wear leveling)
|
||||
/// Supports both NVMe and SATA SSD wear indicators
|
||||
fn parse_wear_level_from_smart(&self, smart_output: &str) -> Option<f32> {
|
||||
for line in smart_output.lines() {
|
||||
let line = line.trim();
|
||||
|
||||
// NVMe drives - direct percentage used
|
||||
if line.contains("Percentage Used:") {
|
||||
if let Some(wear_part) = line.split("Percentage Used:").nth(1) {
|
||||
if let Some(wear_str) = wear_part.split('%').next() {
|
||||
if let Ok(wear) = wear_str.trim().parse::<f32>() {
|
||||
return Some(wear);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// SATA SSD attributes - parse SMART table format
|
||||
// Format: ID ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
if parts.len() >= 10 {
|
||||
// SSD Life Left / Percent Lifetime Remaining (higher = less wear)
|
||||
if line.contains("SSD_Life_Left") || line.contains("Percent_Lifetime_Remain") {
|
||||
if let Ok(remaining) = parts[3].parse::<f32>() { // VALUE column
|
||||
return Some(100.0 - remaining); // Convert remaining to used
|
||||
}
|
||||
}
|
||||
|
||||
// Media Wearout Indicator (lower = more wear, normalize to 0-100)
|
||||
if line.contains("Media_Wearout_Indicator") {
|
||||
if let Ok(remaining) = parts[3].parse::<f32>() { // VALUE column
|
||||
return Some(100.0 - remaining); // Convert remaining to used
|
||||
}
|
||||
}
|
||||
|
||||
// Wear Leveling Count (higher = less wear, but varies by manufacturer)
|
||||
if line.contains("Wear_Leveling_Count") {
|
||||
if let Ok(wear_count) = parts[3].parse::<f32>() { // VALUE column
|
||||
// Most SSDs: 100 = new, decreases with wear
|
||||
if wear_count <= 100.0 {
|
||||
return Some(100.0 - wear_count);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Total LBAs Written - calculate against typical endurance if available
|
||||
// This is more complex and manufacturer-specific, so we skip for now
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Convert bytes to human-readable format
|
||||
fn bytes_to_human_readable(&self, bytes: u64) -> String {
|
||||
const UNITS: &[&str] = &["B", "K", "M", "G", "T"];
|
||||
let mut size = bytes as f64;
|
||||
let mut unit_index = 0;
|
||||
|
||||
while size >= 1024.0 && unit_index < UNITS.len() - 1 {
|
||||
size /= 1024.0;
|
||||
unit_index += 1;
|
||||
}
|
||||
|
||||
if unit_index == 0 {
|
||||
format!("{:.0}{}", size, UNITS[unit_index])
|
||||
} else {
|
||||
format!("{:.1}{}", size, UNITS[unit_index])
|
||||
}
|
||||
}
|
||||
|
||||
/// Detect device backing a mount point using lsblk (static version for startup)
|
||||
fn detect_device_for_mount_point_static(mount_point: &str) -> Result<Vec<String>> {
|
||||
/// Use lsblk to get mount points and their backing devices
|
||||
fn get_mount_devices(&self) -> Result<HashMap<String, String>> {
|
||||
let output = Command::new("lsblk")
|
||||
.args(&["-n", "-o", "NAME,MOUNTPOINT"])
|
||||
.output()?;
|
||||
|
||||
if !output.status.success() {
|
||||
return Ok(Vec::new());
|
||||
return Err(anyhow::anyhow!("lsblk command failed"));
|
||||
}
|
||||
|
||||
let mut mount_devices = HashMap::new();
|
||||
let output_str = String::from_utf8_lossy(&output.stdout);
|
||||
|
||||
for line in output_str.lines() {
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
if parts.len() >= 2 && parts[1] == mount_point {
|
||||
// Remove tree symbols and extract device name (e.g., "├─nvme0n1p2" -> "nvme0n1p2")
|
||||
if parts.len() >= 2 {
|
||||
let device_name = parts[0]
|
||||
.trim_start_matches('├')
|
||||
.trim_start_matches('└')
|
||||
.trim_start_matches('─')
|
||||
.trim();
|
||||
.trim_start_matches(&['├', '└', '─', ' '][..]);
|
||||
let mount_point = parts[1];
|
||||
|
||||
// Extract base device name (e.g., "nvme0n1p2" -> "nvme0n1")
|
||||
if let Some(base_device) = Self::extract_base_device(device_name) {
|
||||
return Ok(vec![base_device]);
|
||||
// Skip unwanted mount points
|
||||
if self.should_skip_mount_point(mount_point) {
|
||||
continue;
|
||||
}
|
||||
|
||||
mount_devices.insert(mount_point.to_string(), device_name.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
Ok(mount_devices)
|
||||
}
|
||||
|
||||
/// Check if we should skip this mount point
|
||||
fn should_skip_mount_point(&self, mount_point: &str) -> bool {
|
||||
let skip_prefixes = ["/proc", "/sys", "/dev", "/tmp", "/run"];
|
||||
skip_prefixes.iter().any(|prefix| mount_point.starts_with(prefix))
|
||||
}
|
||||
|
||||
/// Use df to get filesystem usage for mount points
|
||||
fn get_filesystem_usage(&self, mount_devices: &HashMap<String, String>) -> Result<HashMap<String, (u64, u64)>> {
|
||||
let mut filesystem_usage = HashMap::new();
|
||||
|
||||
for mount_point in mount_devices.keys() {
|
||||
match self.get_filesystem_info(mount_point) {
|
||||
Ok((total, used)) => {
|
||||
filesystem_usage.insert(mount_point.clone(), (total, used));
|
||||
}
|
||||
Err(e) => {
|
||||
debug!("Failed to get filesystem info for {}: {}", mount_point, e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(Vec::new())
|
||||
Ok(filesystem_usage)
|
||||
}
|
||||
|
||||
/// Extract base device name from partition (e.g., "nvme0n1p2" -> "nvme0n1", "sda1" -> "sda")
|
||||
fn extract_base_device(device_name: &str) -> Option<String> {
|
||||
// Handle NVMe devices (nvme0n1p1 -> nvme0n1)
|
||||
if device_name.starts_with("nvme") {
|
||||
if let Some(p_pos) = device_name.find('p') {
|
||||
return Some(device_name[..p_pos].to_string());
|
||||
}
|
||||
}
|
||||
|
||||
// Handle traditional devices (sda1 -> sda)
|
||||
if device_name.len() > 1 {
|
||||
let chars: Vec<char> = device_name.chars().collect();
|
||||
let mut end_idx = chars.len();
|
||||
|
||||
// Find where the device name ends and partition number begins
|
||||
for (i, &c) in chars.iter().enumerate().rev() {
|
||||
if !c.is_ascii_digit() {
|
||||
end_idx = i + 1;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if end_idx > 0 && end_idx < chars.len() {
|
||||
return Some(chars[..end_idx].iter().collect());
|
||||
}
|
||||
}
|
||||
|
||||
// If no partition detected, return as-is
|
||||
Some(device_name.to_string())
|
||||
}
|
||||
|
||||
|
||||
/// Get filesystem info using df command
|
||||
fn get_filesystem_info(&self, path: &str) -> Result<(u64, u64)> {
|
||||
let output = Command::new("df")
|
||||
@@ -381,216 +187,815 @@ impl DiskCollector {
|
||||
Ok((total_bytes, used_bytes))
|
||||
}
|
||||
|
||||
/// Discover mergerfs pools from /proc/mounts
|
||||
fn discover_mergerfs_pools(&self) -> Result<Vec<MergerfsPool>> {
|
||||
let mounts_content = std::fs::read_to_string("/proc/mounts")?;
|
||||
let mut pools = Vec::new();
|
||||
|
||||
/// Parse size string (e.g., "120G", "45M") to GB value
|
||||
fn parse_size_to_gb(&self, size_str: &str) -> f32 {
|
||||
let size_str = size_str.trim();
|
||||
if size_str.is_empty() || size_str == "-" {
|
||||
return 0.0;
|
||||
}
|
||||
for line in mounts_content.lines() {
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
if parts.len() >= 3 && parts[2] == "fuse.mergerfs" {
|
||||
let mount_point = parts[1].to_string();
|
||||
let device_sources = parts[0]; // e.g., "/mnt/disk1:/mnt/disk2"
|
||||
|
||||
// Extract numeric part and unit
|
||||
let (num_str, unit) = if let Some(last_char) = size_str.chars().last() {
|
||||
if last_char.is_alphabetic() {
|
||||
let num_part = &size_str[..size_str.len() - 1];
|
||||
let unit_part = &size_str[size_str.len() - 1..];
|
||||
(num_part, unit_part)
|
||||
} else {
|
||||
(size_str, "")
|
||||
// Get pool usage
|
||||
let (total_bytes, used_bytes) = self.get_filesystem_info(&mount_point)
|
||||
.unwrap_or((0, 0));
|
||||
|
||||
// Parse member paths - handle both full paths and numeric references
|
||||
let raw_paths: Vec<String> = device_sources
|
||||
.split(':')
|
||||
.map(|s| s.trim().to_string())
|
||||
.filter(|s| !s.is_empty())
|
||||
.collect();
|
||||
|
||||
// Convert numeric references to actual mount points if needed
|
||||
let mut member_paths = if raw_paths.iter().any(|path| !path.starts_with('/')) {
|
||||
// Handle numeric format like "1:2" by finding corresponding /mnt/disk* paths
|
||||
self.resolve_numeric_mergerfs_paths(&raw_paths)?
|
||||
} else {
|
||||
// Already full paths
|
||||
raw_paths
|
||||
};
|
||||
|
||||
// For SnapRAID setups, include parity drives that are related to this pool's data drives
|
||||
let related_parity_paths = self.discover_related_parity_drives(&member_paths)?;
|
||||
member_paths.extend(related_parity_paths);
|
||||
|
||||
// Categorize as data vs parity drives
|
||||
let (data_drives, parity_drives) = match self.categorize_pool_drives(&member_paths) {
|
||||
Ok(drives) => drives,
|
||||
Err(e) => {
|
||||
debug!("Failed to categorize drives for pool {}: {}. Skipping.", mount_point, e);
|
||||
continue;
|
||||
}
|
||||
};
|
||||
|
||||
pools.push(MergerfsPool {
|
||||
mount_point,
|
||||
total_bytes,
|
||||
used_bytes,
|
||||
data_drives,
|
||||
parity_drives,
|
||||
});
|
||||
}
|
||||
} else {
|
||||
(size_str, "")
|
||||
};
|
||||
|
||||
let number: f32 = num_str.parse().unwrap_or(0.0);
|
||||
|
||||
match unit.to_uppercase().as_str() {
|
||||
"T" | "TB" => number * 1024.0,
|
||||
"G" | "GB" => number,
|
||||
"M" | "MB" => number / 1024.0,
|
||||
"K" | "KB" => number / (1024.0 * 1024.0),
|
||||
"B" | "" => number / (1024.0 * 1024.0 * 1024.0),
|
||||
_ => number, // Assume GB if unknown unit
|
||||
}
|
||||
|
||||
Ok(pools)
|
||||
}
|
||||
|
||||
/// Discover parity drives that are related to the given data drives
|
||||
fn discover_related_parity_drives(&self, data_drives: &[String]) -> Result<Vec<String>> {
|
||||
let mount_devices = self.get_mount_devices()?;
|
||||
let mut related_parity = Vec::new();
|
||||
|
||||
// Find parity drives that share the same parent directory as the data drives
|
||||
for data_path in data_drives {
|
||||
if let Some(parent_dir) = self.get_parent_directory(data_path) {
|
||||
// Look for parity drives in the same parent directory
|
||||
for (mount_point, _device) in &mount_devices {
|
||||
if mount_point.contains("parity") && mount_point.starts_with(&parent_dir) {
|
||||
if !related_parity.contains(mount_point) {
|
||||
related_parity.push(mount_point.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(related_parity)
|
||||
}
|
||||
|
||||
/// Get parent directory of a mount path (e.g., "/mnt/disk1" -> "/mnt")
|
||||
fn get_parent_directory(&self, path: &str) -> Option<String> {
|
||||
if let Some(last_slash) = path.rfind('/') {
|
||||
if last_slash > 0 {
|
||||
return Some(path[..last_slash].to_string());
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Categorize pool member drives as data vs parity
|
||||
fn categorize_pool_drives(&self, member_paths: &[String]) -> Result<(Vec<DriveInfo>, Vec<DriveInfo>)> {
|
||||
let mut data_drives = Vec::new();
|
||||
let mut parity_drives = Vec::new();
|
||||
|
||||
for path in member_paths {
|
||||
let drive_info = self.get_drive_info_for_path(path)?;
|
||||
|
||||
// Heuristic: if path contains "parity", it's parity
|
||||
if path.to_lowercase().contains("parity") {
|
||||
parity_drives.push(drive_info);
|
||||
} else {
|
||||
data_drives.push(drive_info);
|
||||
}
|
||||
}
|
||||
|
||||
Ok((data_drives, parity_drives))
|
||||
}
|
||||
|
||||
/// Get drive information for a mount path
|
||||
fn get_drive_info_for_path(&self, path: &str) -> Result<DriveInfo> {
|
||||
// Use lsblk to find the backing device
|
||||
let output = Command::new("lsblk")
|
||||
.args(&["-n", "-o", "NAME,MOUNTPOINT"])
|
||||
.output()?;
|
||||
|
||||
let output_str = String::from_utf8_lossy(&output.stdout);
|
||||
let mut device = String::new();
|
||||
|
||||
for line in output_str.lines() {
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
if parts.len() >= 2 && parts[1] == path {
|
||||
device = parts[0]
|
||||
.trim_start_matches('├')
|
||||
.trim_start_matches('└')
|
||||
.trim_start_matches('─')
|
||||
.trim()
|
||||
.to_string();
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if device.is_empty() {
|
||||
return Err(anyhow::anyhow!("Could not find device for path {}", path));
|
||||
}
|
||||
|
||||
// Extract base device name (e.g., "sda1" -> "sda")
|
||||
let base_device = self.extract_base_device(&device);
|
||||
|
||||
// Get SMART data
|
||||
let (health, temperature, wear) = self.get_smart_data(&format!("/dev/{}", base_device));
|
||||
|
||||
Ok(DriveInfo {
|
||||
device: base_device,
|
||||
mount_point: path.to_string(),
|
||||
temperature,
|
||||
wear_level: wear,
|
||||
health_status: health,
|
||||
})
|
||||
}
|
||||
|
||||
/// Resolve numeric mergerfs references like "1:2" to actual mount paths
|
||||
fn resolve_numeric_mergerfs_paths(&self, numeric_refs: &[String]) -> Result<Vec<String>> {
|
||||
let mut resolved_paths = Vec::new();
|
||||
|
||||
// Get all mount points that look like /mnt/disk* or /mnt/parity*
|
||||
let mount_devices = self.get_mount_devices()?;
|
||||
let mut disk_mounts: Vec<String> = mount_devices.keys()
|
||||
.filter(|path| path.starts_with("/mnt/disk") || path.starts_with("/mnt/parity"))
|
||||
.cloned()
|
||||
.collect();
|
||||
disk_mounts.sort(); // Ensure consistent ordering
|
||||
|
||||
for num_ref in numeric_refs {
|
||||
if let Ok(index) = num_ref.parse::<usize>() {
|
||||
// Convert 1-based index to 0-based
|
||||
if index > 0 && index <= disk_mounts.len() {
|
||||
resolved_paths.push(disk_mounts[index - 1].clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback: if we couldn't resolve, return the original paths
|
||||
if resolved_paths.is_empty() {
|
||||
resolved_paths = numeric_refs.to_vec();
|
||||
}
|
||||
|
||||
Ok(resolved_paths)
|
||||
}
|
||||
|
||||
/// Extract base device name from partition (e.g., "nvme0n1p2" -> "nvme0n1", "sda1" -> "sda")
|
||||
fn extract_base_device(&self, device_name: &str) -> String {
|
||||
// Handle NVMe devices (nvme0n1p1 -> nvme0n1)
|
||||
if device_name.starts_with("nvme") {
|
||||
if let Some(p_pos) = device_name.find('p') {
|
||||
return device_name[..p_pos].to_string();
|
||||
}
|
||||
}
|
||||
|
||||
// Handle traditional devices (sda1 -> sda)
|
||||
if device_name.len() > 1 {
|
||||
let chars: Vec<char> = device_name.chars().collect();
|
||||
let mut end_idx = chars.len();
|
||||
|
||||
// Find where the device name ends and partition number begins
|
||||
for (i, &c) in chars.iter().enumerate().rev() {
|
||||
if !c.is_ascii_digit() {
|
||||
end_idx = i + 1;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if end_idx > 0 && end_idx < chars.len() {
|
||||
return chars[..end_idx].iter().collect();
|
||||
}
|
||||
}
|
||||
|
||||
// If no partition detected, return as-is
|
||||
device_name.to_string()
|
||||
}
|
||||
|
||||
/// Group filesystems by physical drive (excluding mergerfs members)
|
||||
fn group_by_physical_drive(
|
||||
&self,
|
||||
mount_devices: &HashMap<String, String>,
|
||||
filesystem_usage: &HashMap<String, (u64, u64)>,
|
||||
mergerfs_pools: &[MergerfsPool]
|
||||
) -> Result<Vec<PhysicalDrive>> {
|
||||
let mut drive_groups: HashMap<String, Vec<Filesystem>> = HashMap::new();
|
||||
|
||||
// Get all mergerfs member paths to exclude them
|
||||
let mut mergerfs_members = std::collections::HashSet::new();
|
||||
for pool in mergerfs_pools {
|
||||
for drive in &pool.data_drives {
|
||||
mergerfs_members.insert(drive.mount_point.clone());
|
||||
}
|
||||
for drive in &pool.parity_drives {
|
||||
mergerfs_members.insert(drive.mount_point.clone());
|
||||
}
|
||||
}
|
||||
|
||||
// Group filesystems by base device
|
||||
for (mount_point, device) in mount_devices {
|
||||
// Skip mergerfs member mounts
|
||||
if mergerfs_members.contains(mount_point) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let base_device = self.extract_base_device(device);
|
||||
|
||||
if let Some((total, used)) = filesystem_usage.get(mount_point) {
|
||||
let filesystem = Filesystem {
|
||||
mount_point: mount_point.clone(),
|
||||
total_bytes: *total,
|
||||
used_bytes: *used,
|
||||
};
|
||||
|
||||
drive_groups.entry(base_device).or_insert_with(Vec::new).push(filesystem);
|
||||
}
|
||||
}
|
||||
|
||||
// Convert to PhysicalDrive structs with SMART data
|
||||
let mut physical_drives = Vec::new();
|
||||
for (device, filesystems) in drive_groups {
|
||||
let (health, temperature, wear) = self.get_smart_data(&format!("/dev/{}", device));
|
||||
|
||||
physical_drives.push(PhysicalDrive {
|
||||
device,
|
||||
filesystems,
|
||||
temperature,
|
||||
wear_level: wear,
|
||||
health_status: health,
|
||||
});
|
||||
}
|
||||
|
||||
Ok(physical_drives)
|
||||
}
|
||||
|
||||
/// Get SMART data for a drive
|
||||
fn get_smart_data(&self, device_path: &str) -> (String, Option<f32>, Option<f32>) {
|
||||
let output = Command::new("sudo")
|
||||
.arg("smartctl")
|
||||
.arg("-a")
|
||||
.arg(device_path)
|
||||
.output();
|
||||
|
||||
match output {
|
||||
Ok(result) if result.status.success() => {
|
||||
let stdout = String::from_utf8_lossy(&result.stdout);
|
||||
|
||||
// Parse health status
|
||||
let health = if stdout.contains("PASSED") {
|
||||
"PASSED".to_string()
|
||||
} else if stdout.contains("FAILED") {
|
||||
"FAILED".to_string()
|
||||
} else {
|
||||
"UNKNOWN".to_string()
|
||||
};
|
||||
|
||||
// Parse temperature and wear level
|
||||
let temperature = self.parse_temperature_from_smart(&stdout);
|
||||
let wear_level = self.parse_wear_level_from_smart(&stdout);
|
||||
|
||||
(health, temperature, wear_level)
|
||||
}
|
||||
_ => {
|
||||
debug!("Failed to get SMART data for {}", device_path);
|
||||
("UNKNOWN".to_string(), None, None)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse temperature from SMART output
|
||||
fn parse_temperature_from_smart(&self, smart_output: &str) -> Option<f32> {
|
||||
for line in smart_output.lines() {
|
||||
if line.contains("Temperature_Celsius") || line.contains("Temperature") {
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
if parts.len() >= 10 {
|
||||
if let Ok(temp) = parts[9].parse::<f32>() {
|
||||
return Some(temp);
|
||||
}
|
||||
}
|
||||
}
|
||||
// NVMe format: "Temperature:" (capital T)
|
||||
if line.contains("Temperature:") {
|
||||
if let Some(temp_part) = line.split("Temperature:").nth(1) {
|
||||
if let Some(temp_str) = temp_part.split_whitespace().next() {
|
||||
if let Ok(temp) = temp_str.parse::<f32>() {
|
||||
return Some(temp);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
// Legacy format: "temperature:" (lowercase)
|
||||
if line.contains("temperature:") {
|
||||
if let Some(temp_part) = line.split("temperature:").nth(1) {
|
||||
if let Some(temp_str) = temp_part.split_whitespace().next() {
|
||||
if let Ok(temp) = temp_str.parse::<f32>() {
|
||||
return Some(temp);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Parse wear level from SMART output
|
||||
fn parse_wear_level_from_smart(&self, smart_output: &str) -> Option<f32> {
|
||||
for line in smart_output.lines() {
|
||||
if line.contains("Percentage Used:") {
|
||||
if let Some(wear_part) = line.split("Percentage Used:").nth(1) {
|
||||
if let Some(wear_str) = wear_part.split('%').next() {
|
||||
if let Ok(wear) = wear_str.trim().parse::<f32>() {
|
||||
return Some(wear);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
if parts.len() >= 10 {
|
||||
if line.contains("SSD_Life_Left") || line.contains("Percent_Lifetime_Remain") {
|
||||
if let Ok(remaining) = parts[3].parse::<f32>() {
|
||||
return Some(100.0 - remaining);
|
||||
}
|
||||
}
|
||||
if line.contains("Wear_Leveling_Count") {
|
||||
if let Ok(wear_count) = parts[3].parse::<f32>() {
|
||||
if wear_count <= 100.0 {
|
||||
return Some(100.0 - wear_count);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Calculate temperature status with hysteresis
|
||||
fn calculate_temperature_status(&self, metric_name: &str, temperature: f32, status_tracker: &mut StatusTracker) -> Status {
|
||||
status_tracker.calculate_with_hysteresis(metric_name, temperature, &self.temperature_thresholds)
|
||||
}
|
||||
|
||||
/// Convert bytes to human readable format
|
||||
fn bytes_to_human_readable(&self, bytes: u64) -> String {
|
||||
const UNITS: &[&str] = &["B", "K", "M", "G", "T"];
|
||||
let mut size = bytes as f64;
|
||||
let mut unit_index = 0;
|
||||
|
||||
while size >= 1024.0 && unit_index < UNITS.len() - 1 {
|
||||
size /= 1024.0;
|
||||
unit_index += 1;
|
||||
}
|
||||
|
||||
if unit_index == 0 {
|
||||
format!("{:.0}{}", size, UNITS[unit_index])
|
||||
} else {
|
||||
format!("{:.1}{}", size, UNITS[unit_index])
|
||||
}
|
||||
}
|
||||
|
||||
/// Convert bytes to gigabytes
|
||||
fn bytes_to_gb(&self, bytes: u64) -> f32 {
|
||||
bytes as f32 / (1024.0 * 1024.0 * 1024.0)
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl Collector for DiskCollector {
|
||||
|
||||
async fn collect(&self, status_tracker: &mut StatusTracker) -> Result<Vec<Metric>, CollectorError> {
|
||||
let start_time = Instant::now();
|
||||
debug!("Collecting storage pool and individual drive metrics");
|
||||
debug!("Starting clean storage collection");
|
||||
|
||||
let mut metrics = Vec::new();
|
||||
let timestamp = chrono::Utc::now().timestamp() as u64;
|
||||
|
||||
// Get configured storage pools with individual drive data
|
||||
let storage_pools = match self.get_configured_storage_pools() {
|
||||
Ok(pools) => {
|
||||
debug!("Found {} storage pools", pools.len());
|
||||
pools
|
||||
}
|
||||
// Discover storage topology
|
||||
let topology = match self.discover_storage() {
|
||||
Ok(topology) => topology,
|
||||
Err(e) => {
|
||||
debug!("Failed to get storage pools: {}", e);
|
||||
Vec::new()
|
||||
tracing::error!("Storage discovery failed: {}", e);
|
||||
return Ok(metrics);
|
||||
}
|
||||
};
|
||||
|
||||
// Generate metrics for each storage pool and its underlying drives
|
||||
for storage_pool in &storage_pools {
|
||||
let timestamp = chrono::Utc::now().timestamp() as u64;
|
||||
// Generate metrics for physical drives
|
||||
for drive in &topology.physical_drives {
|
||||
self.generate_physical_drive_metrics(&mut metrics, drive, timestamp, status_tracker);
|
||||
}
|
||||
|
||||
// Storage pool overall metrics
|
||||
let pool_name = &storage_pool.name;
|
||||
// Generate metrics for mergerfs pools
|
||||
for pool in &topology.mergerfs_pools {
|
||||
self.generate_mergerfs_pool_metrics(&mut metrics, pool, timestamp, status_tracker);
|
||||
}
|
||||
|
||||
// Parse size strings to get actual values for calculations
|
||||
let size_gb = self.parse_size_to_gb(&storage_pool.size);
|
||||
let used_gb = self.parse_size_to_gb(&storage_pool.used);
|
||||
let avail_gb = self.parse_size_to_gb(&storage_pool.available);
|
||||
// Add total storage count
|
||||
let total_storage = topology.physical_drives.len() + topology.mergerfs_pools.len();
|
||||
metrics.push(Metric {
|
||||
name: "disk_count".to_string(),
|
||||
value: MetricValue::Integer(total_storage as i64),
|
||||
unit: None,
|
||||
description: Some(format!("Total storage: {} drives, {} pools", topology.physical_drives.len(), topology.mergerfs_pools.len())),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Calculate status based on configured thresholds
|
||||
let pool_status = if storage_pool.usage_percent >= self.config.usage_critical_percent {
|
||||
let collection_time = start_time.elapsed();
|
||||
debug!("Clean storage collection completed in {:?} with {} metrics", collection_time, metrics.len());
|
||||
|
||||
Ok(metrics)
|
||||
}
|
||||
}
|
||||
|
||||
impl DiskCollector {
|
||||
/// Generate metrics for a physical drive and its filesystems
|
||||
fn generate_physical_drive_metrics(
|
||||
&self,
|
||||
metrics: &mut Vec<Metric>,
|
||||
drive: &PhysicalDrive,
|
||||
timestamp: u64,
|
||||
status_tracker: &mut StatusTracker
|
||||
) {
|
||||
let drive_name = &drive.device;
|
||||
|
||||
// Calculate drive totals
|
||||
let total_capacity: u64 = drive.filesystems.iter().map(|fs| fs.total_bytes).sum();
|
||||
let total_used: u64 = drive.filesystems.iter().map(|fs| fs.used_bytes).sum();
|
||||
let total_available = total_capacity.saturating_sub(total_used);
|
||||
let usage_percent = if total_capacity > 0 {
|
||||
(total_used as f64 / total_capacity as f64) * 100.0
|
||||
} else { 0.0 };
|
||||
|
||||
// Drive health status
|
||||
let health_status = if drive.health_status == "PASSED" { Status::Ok }
|
||||
else if drive.health_status == "FAILED" { Status::Critical }
|
||||
else { Status::Unknown };
|
||||
|
||||
// Usage status
|
||||
let usage_status = if usage_percent >= self.config.usage_critical_percent as f64 {
|
||||
Status::Critical
|
||||
} else if usage_percent >= self.config.usage_warning_percent as f64 {
|
||||
Status::Warning
|
||||
} else {
|
||||
Status::Ok
|
||||
};
|
||||
|
||||
let drive_status = if health_status == Status::Critical { Status::Critical } else { usage_status };
|
||||
|
||||
// Drive info metrics
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_health", drive_name),
|
||||
value: MetricValue::String(drive.health_status.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("{}: {}", drive_name, drive.health_status)),
|
||||
status: health_status,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Drive temperature
|
||||
if let Some(temp) = drive.temperature {
|
||||
let temp_status = self.calculate_temperature_status(
|
||||
&format!("disk_{}_temperature", drive_name), temp, status_tracker
|
||||
);
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_temperature", drive_name),
|
||||
value: MetricValue::Float(temp),
|
||||
unit: Some("°C".to_string()),
|
||||
description: Some(format!("{}: {:.0}°C", drive_name, temp)),
|
||||
status: temp_status,
|
||||
timestamp,
|
||||
});
|
||||
}
|
||||
|
||||
// Drive wear level
|
||||
if let Some(wear) = drive.wear_level {
|
||||
let wear_status = if wear >= self.config.wear_critical_percent { Status::Critical }
|
||||
else if wear >= self.config.wear_warning_percent { Status::Warning }
|
||||
else { Status::Ok };
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_wear_percent", drive_name),
|
||||
value: MetricValue::Float(wear),
|
||||
unit: Some("%".to_string()),
|
||||
description: Some(format!("{}: {:.0}% wear", drive_name, wear)),
|
||||
status: wear_status,
|
||||
timestamp,
|
||||
});
|
||||
}
|
||||
|
||||
// Drive capacity metrics
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_total_gb", drive_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(total_capacity)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("{}: {}", drive_name, self.bytes_to_human_readable(total_capacity))),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_used_gb", drive_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(total_used)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("{}: {}", drive_name, self.bytes_to_human_readable(total_used))),
|
||||
status: drive_status.clone(),
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_available_gb", drive_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(total_available)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("{}: {}", drive_name, self.bytes_to_human_readable(total_available))),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_usage_percent", drive_name),
|
||||
value: MetricValue::Float(usage_percent as f32),
|
||||
unit: Some("%".to_string()),
|
||||
description: Some(format!("{}: {:.1}%", drive_name, usage_percent)),
|
||||
status: drive_status,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Pool type indicator
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_pool_type", drive_name),
|
||||
value: MetricValue::String(format!("drive ({})", drive.filesystems.len())),
|
||||
unit: None,
|
||||
description: Some(format!("Type: physical drive")),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Individual filesystem metrics
|
||||
for filesystem in &drive.filesystems {
|
||||
let fs_name = if filesystem.mount_point == "/" {
|
||||
"root".to_string()
|
||||
} else {
|
||||
filesystem.mount_point.trim_start_matches('/').replace('/', "_")
|
||||
};
|
||||
|
||||
let fs_usage_percent = if filesystem.total_bytes > 0 {
|
||||
(filesystem.used_bytes as f64 / filesystem.total_bytes as f64) * 100.0
|
||||
} else { 0.0 };
|
||||
|
||||
let fs_status = if fs_usage_percent >= self.config.usage_critical_percent as f64 {
|
||||
Status::Critical
|
||||
} else if storage_pool.usage_percent >= self.config.usage_warning_percent {
|
||||
} else if fs_usage_percent >= self.config.usage_warning_percent as f64 {
|
||||
Status::Warning
|
||||
} else {
|
||||
Status::Ok
|
||||
};
|
||||
|
||||
// Storage pool info metrics
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_mount_point", pool_name),
|
||||
value: MetricValue::String(storage_pool.mount_point.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("Mount: {}", storage_pool.mount_point)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_filesystem", pool_name),
|
||||
value: MetricValue::String(storage_pool.filesystem.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("FS: {}", storage_pool.filesystem)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_storage_type", pool_name),
|
||||
value: MetricValue::String(storage_pool.storage_type.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("Type: {}", storage_pool.storage_type)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Storage pool size metrics
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_total_gb", pool_name),
|
||||
value: MetricValue::Float(size_gb),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("Total: {}", storage_pool.size)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_used_gb", pool_name),
|
||||
value: MetricValue::Float(used_gb),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("Used: {}", storage_pool.used)),
|
||||
status: pool_status,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_available_gb", pool_name),
|
||||
value: MetricValue::Float(avail_gb),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("Available: {}", storage_pool.available)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_usage_percent", pool_name),
|
||||
value: MetricValue::Float(storage_pool.usage_percent),
|
||||
name: format!("disk_{}_fs_{}_usage_percent", drive_name, fs_name),
|
||||
value: MetricValue::Float(fs_usage_percent as f32),
|
||||
unit: Some("%".to_string()),
|
||||
description: Some(format!("Usage: {:.1}%", storage_pool.usage_percent)),
|
||||
status: pool_status,
|
||||
description: Some(format!("{}: {:.0}%", filesystem.mount_point, fs_usage_percent)),
|
||||
status: fs_status.clone(),
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Individual drive metrics for this storage pool
|
||||
for drive in &storage_pool.underlying_drives {
|
||||
// Drive health status
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_{}_health", pool_name, drive.device),
|
||||
value: MetricValue::String(drive.health_status.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("{}: {}", drive.device, drive.health_status)),
|
||||
status: if drive.health_status == "PASSED" { Status::Ok }
|
||||
else if drive.health_status == "FAILED" { Status::Critical }
|
||||
else { Status::Unknown },
|
||||
timestamp,
|
||||
});
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_fs_{}_used_gb", drive_name, fs_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(filesystem.used_bytes)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("{}: {}", filesystem.mount_point, self.bytes_to_human_readable(filesystem.used_bytes))),
|
||||
status: fs_status.clone(),
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Drive temperature
|
||||
if let Some(temp) = drive.temperature {
|
||||
let temp_status = self.calculate_temperature_status(
|
||||
&format!("disk_{}_{}_temperature", pool_name, drive.device),
|
||||
temp,
|
||||
status_tracker
|
||||
);
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_fs_{}_total_gb", drive_name, fs_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(filesystem.total_bytes)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("{}: {}", filesystem.mount_point, self.bytes_to_human_readable(filesystem.total_bytes))),
|
||||
status: fs_status.clone(),
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_{}_temperature", pool_name, drive.device),
|
||||
value: MetricValue::Float(temp),
|
||||
unit: Some("°C".to_string()),
|
||||
description: Some(format!("{}: {:.0}°C", drive.device, temp)),
|
||||
status: temp_status,
|
||||
timestamp,
|
||||
});
|
||||
}
|
||||
let fs_available = filesystem.total_bytes.saturating_sub(filesystem.used_bytes);
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_fs_{}_available_gb", drive_name, fs_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(fs_available)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("{}: {}", filesystem.mount_point, self.bytes_to_human_readable(fs_available))),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Drive wear level (for SSDs)
|
||||
if let Some(wear) = drive.wear_level {
|
||||
let wear_status = if wear >= self.config.wear_critical_percent { Status::Critical }
|
||||
else if wear >= self.config.wear_warning_percent { Status::Warning }
|
||||
else { Status::Ok };
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_{}_wear_percent", pool_name, drive.device),
|
||||
value: MetricValue::Float(wear),
|
||||
unit: Some("%".to_string()),
|
||||
description: Some(format!("{}: {:.0}% wear", drive.device, wear)),
|
||||
status: wear_status,
|
||||
timestamp,
|
||||
});
|
||||
}
|
||||
}
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_fs_{}_mount_point", drive_name, fs_name),
|
||||
value: MetricValue::String(filesystem.mount_point.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("Mount: {}", filesystem.mount_point)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
}
|
||||
|
||||
// Add storage pool count metric
|
||||
metrics.push(Metric {
|
||||
name: "disk_count".to_string(),
|
||||
value: MetricValue::Integer(storage_pools.len() as i64),
|
||||
unit: None,
|
||||
description: Some(format!("Total storage pools: {}", storage_pools.len())),
|
||||
status: Status::Ok,
|
||||
timestamp: chrono::Utc::now().timestamp() as u64,
|
||||
});
|
||||
|
||||
|
||||
let collection_time = start_time.elapsed();
|
||||
debug!(
|
||||
"Multi-disk collection completed in {:?} with {} metrics",
|
||||
collection_time,
|
||||
metrics.len()
|
||||
);
|
||||
|
||||
Ok(metrics)
|
||||
}
|
||||
|
||||
/// Generate metrics for a mergerfs pool
|
||||
fn generate_mergerfs_pool_metrics(
|
||||
&self,
|
||||
metrics: &mut Vec<Metric>,
|
||||
pool: &MergerfsPool,
|
||||
timestamp: u64,
|
||||
status_tracker: &mut StatusTracker
|
||||
) {
|
||||
// Use consistent pool naming: extract mount point without leading slash
|
||||
let pool_name = if pool.mount_point == "/" {
|
||||
"root".to_string()
|
||||
} else {
|
||||
pool.mount_point.trim_start_matches('/').replace('/', "_")
|
||||
};
|
||||
|
||||
if pool_name.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
let usage_percent = if pool.total_bytes > 0 {
|
||||
(pool.used_bytes as f64 / pool.total_bytes as f64) * 100.0
|
||||
} else { 0.0 };
|
||||
|
||||
// Calculate pool health based on drive health
|
||||
let failed_data = pool.data_drives.iter()
|
||||
.filter(|d| d.health_status != "PASSED")
|
||||
.count();
|
||||
let failed_parity = pool.parity_drives.iter()
|
||||
.filter(|d| d.health_status != "PASSED")
|
||||
.count();
|
||||
|
||||
let pool_health = match (failed_data, failed_parity) {
|
||||
(0, 0) => Status::Ok,
|
||||
(1, 0) | (0, 1) => Status::Warning,
|
||||
_ => Status::Critical,
|
||||
};
|
||||
|
||||
let usage_status = if usage_percent >= self.config.usage_critical_percent as f64 {
|
||||
Status::Critical
|
||||
} else if usage_percent >= self.config.usage_warning_percent as f64 {
|
||||
Status::Warning
|
||||
} else {
|
||||
Status::Ok
|
||||
};
|
||||
|
||||
let pool_status = if pool_health == Status::Critical { Status::Critical } else { usage_status };
|
||||
|
||||
// Pool metrics
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_mount_point", pool_name),
|
||||
value: MetricValue::String(pool.mount_point.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("Mount: {}", pool.mount_point)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_pool_type", pool_name),
|
||||
value: MetricValue::String(format!("mergerfs ({}+{})", pool.data_drives.len(), pool.parity_drives.len())),
|
||||
unit: None,
|
||||
description: Some("Type: mergerfs".to_string()),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_pool_health", pool_name),
|
||||
value: MetricValue::String(match pool_health {
|
||||
Status::Ok => "healthy".to_string(),
|
||||
Status::Warning => "degraded".to_string(),
|
||||
Status::Critical => "critical".to_string(),
|
||||
_ => "unknown".to_string(),
|
||||
}),
|
||||
unit: None,
|
||||
description: Some("Pool health".to_string()),
|
||||
status: pool_health,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_total_gb", pool_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(pool.total_bytes)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("Total: {}", self.bytes_to_human_readable(pool.total_bytes))),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_used_gb", pool_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(pool.used_bytes)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("Used: {}", self.bytes_to_human_readable(pool.used_bytes))),
|
||||
status: pool_status.clone(),
|
||||
timestamp,
|
||||
});
|
||||
|
||||
let available_bytes = pool.total_bytes.saturating_sub(pool.used_bytes);
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_available_gb", pool_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(available_bytes)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("Available: {}", self.bytes_to_human_readable(available_bytes))),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_usage_percent", pool_name),
|
||||
value: MetricValue::Float(usage_percent as f32),
|
||||
unit: Some("%".to_string()),
|
||||
description: Some(format!("Usage: {:.1}%", usage_percent)),
|
||||
status: pool_status,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Individual drive metrics
|
||||
for drive in &pool.data_drives {
|
||||
self.generate_pool_drive_metrics(metrics, &pool_name, &drive.device, drive, timestamp, status_tracker);
|
||||
}
|
||||
|
||||
for drive in &pool.parity_drives {
|
||||
self.generate_pool_drive_metrics(metrics, &pool_name, &drive.device, drive, timestamp, status_tracker);
|
||||
}
|
||||
}
|
||||
|
||||
/// Generate metrics for drives in mergerfs pools
|
||||
fn generate_pool_drive_metrics(
|
||||
&self,
|
||||
metrics: &mut Vec<Metric>,
|
||||
pool_name: &str,
|
||||
drive_role: &str,
|
||||
drive: &DriveInfo,
|
||||
timestamp: u64,
|
||||
status_tracker: &mut StatusTracker
|
||||
) {
|
||||
let drive_health = if drive.health_status == "PASSED" { Status::Ok }
|
||||
else if drive.health_status == "FAILED" { Status::Critical }
|
||||
else { Status::Unknown };
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_{}_health", pool_name, drive_role),
|
||||
value: MetricValue::String(drive.health_status.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("{}: {}", drive.device, drive.health_status)),
|
||||
status: drive_health,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
if let Some(temp) = drive.temperature {
|
||||
let temp_status = self.calculate_temperature_status(
|
||||
&format!("disk_{}_{}_temperature", pool_name, drive_role), temp, status_tracker
|
||||
);
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_{}_temperature", pool_name, drive_role),
|
||||
value: MetricValue::Float(temp),
|
||||
unit: Some("°C".to_string()),
|
||||
description: Some(format!("{}: {:.0}°C", drive.device, temp)),
|
||||
status: temp_status,
|
||||
timestamp,
|
||||
});
|
||||
}
|
||||
|
||||
if let Some(wear) = drive.wear_level {
|
||||
let wear_status = if wear >= self.config.wear_critical_percent { Status::Critical }
|
||||
else if wear >= self.config.wear_warning_percent { Status::Warning }
|
||||
else { Status::Ok };
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_{}_wear_percent", pool_name, drive_role),
|
||||
value: MetricValue::Float(wear),
|
||||
unit: Some("%".to_string()),
|
||||
description: Some(format!("{}: {:.0}% wear", drive.device, wear)),
|
||||
status: wear_status,
|
||||
timestamp,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
1327
agent/src/collectors/disk_old.rs
Normal file
1327
agent/src/collectors/disk_old.rs
Normal file
@@ -0,0 +1,1327 @@
|
||||
use anyhow::Result;
|
||||
use async_trait::async_trait;
|
||||
use cm_dashboard_shared::{Metric, MetricValue, Status, StatusTracker, HysteresisThresholds};
|
||||
|
||||
use crate::config::DiskConfig;
|
||||
use std::process::Command;
|
||||
use std::time::Instant;
|
||||
use std::fs;
|
||||
use tracing::debug;
|
||||
|
||||
use super::{Collector, CollectorError};
|
||||
|
||||
/// Mount point information from /proc/mounts
|
||||
#[derive(Debug, Clone)]
|
||||
struct MountInfo {
|
||||
device: String, // e.g., "/dev/sda1" or "/mnt/disk1:/mnt/disk2"
|
||||
mount_point: String, // e.g., "/", "/srv/media"
|
||||
fs_type: String, // e.g., "ext4", "xfs", "fuse.mergerfs"
|
||||
}
|
||||
|
||||
/// Auto-discovered storage topology
|
||||
#[derive(Debug, Clone)]
|
||||
struct StorageTopology {
|
||||
single_disks: Vec<MountInfo>,
|
||||
mergerfs_pools: Vec<MergerfsPoolInfo>,
|
||||
}
|
||||
|
||||
/// MergerFS pool information
|
||||
#[derive(Debug, Clone)]
|
||||
struct MergerfsPoolInfo {
|
||||
mount_point: String, // e.g., "/srv/media"
|
||||
data_members: Vec<String>, // e.g., ["/mnt/disk1", "/mnt/disk2"]
|
||||
parity_disks: Vec<String>, // e.g., ["/mnt/parity"]
|
||||
}
|
||||
|
||||
/// Information about a storage pool (mount point with underlying drives)
|
||||
#[derive(Debug, Clone)]
|
||||
struct StoragePool {
|
||||
name: String, // e.g., "steampool", "root"
|
||||
mount_point: String, // e.g., "/mnt/steampool", "/"
|
||||
filesystem: String, // e.g., "mergerfs", "ext4", "zfs", "btrfs"
|
||||
pool_type: StoragePoolType, // Enhanced pool type with configuration
|
||||
size: String, // e.g., "2.5TB"
|
||||
used: String, // e.g., "2.1TB"
|
||||
available: String, // e.g., "400GB"
|
||||
usage_percent: f32, // e.g., 85.0
|
||||
underlying_drives: Vec<DriveInfo>, // Individual physical drives
|
||||
pool_health: PoolHealth, // Overall pool health status
|
||||
}
|
||||
|
||||
/// Enhanced storage pool types with specific configurations
|
||||
#[derive(Debug, Clone)]
|
||||
enum StoragePoolType {
|
||||
Single, // Traditional single disk (legacy)
|
||||
PhysicalDrive { // Physical drive with multiple filesystems
|
||||
filesystems: Vec<String>, // Mount points on this drive
|
||||
},
|
||||
MergerfsPool { // MergerFS with optional parity
|
||||
data_disks: Vec<String>, // Member disk names (sdb, sdd)
|
||||
parity_disks: Vec<String>, // Parity disk names (sdc)
|
||||
},
|
||||
#[allow(dead_code)]
|
||||
RaidArray { // Hardware RAID (future)
|
||||
level: String, // "RAID1", "RAID5", etc.
|
||||
member_disks: Vec<String>,
|
||||
spare_disks: Vec<String>,
|
||||
},
|
||||
#[allow(dead_code)]
|
||||
ZfsPool { // ZFS pool (future)
|
||||
pool_name: String,
|
||||
vdevs: Vec<String>,
|
||||
}
|
||||
}
|
||||
|
||||
/// Pool health status for redundant storage
|
||||
#[derive(Debug, Clone, Copy, PartialEq)]
|
||||
enum PoolHealth {
|
||||
Healthy, // All drives OK, parity current
|
||||
Degraded, // One drive failed or parity outdated, still functional
|
||||
Critical, // Multiple failures, data at risk
|
||||
#[allow(dead_code)]
|
||||
Rebuilding, // Actively rebuilding/scrubbing (future: SnapRAID status integration)
|
||||
Unknown, // Cannot determine status
|
||||
}
|
||||
|
||||
/// Information about an individual physical drive
|
||||
#[derive(Debug, Clone)]
|
||||
struct DriveInfo {
|
||||
device: String, // e.g., "sda", "nvme0n1"
|
||||
health_status: String, // e.g., "PASSED", "FAILED"
|
||||
temperature: Option<f32>, // e.g., 45.0°C
|
||||
wear_level: Option<f32>, // e.g., 12.0% (for SSDs)
|
||||
}
|
||||
|
||||
/// Disk usage collector for monitoring filesystem sizes
|
||||
pub struct DiskCollector {
|
||||
config: DiskConfig,
|
||||
temperature_thresholds: HysteresisThresholds,
|
||||
detected_devices: std::collections::HashMap<String, Vec<String>>, // mount_point -> devices
|
||||
storage_topology: Option<StorageTopology>, // Auto-discovered storage layout
|
||||
}
|
||||
|
||||
impl DiskCollector {
|
||||
pub fn new(config: DiskConfig) -> Self {
|
||||
// Create hysteresis thresholds for disk temperature from config
|
||||
let temperature_thresholds = HysteresisThresholds::with_custom_gaps(
|
||||
config.temperature_warning_celsius,
|
||||
5.0, // 5°C gap for recovery
|
||||
config.temperature_critical_celsius,
|
||||
5.0, // 5°C gap for recovery
|
||||
);
|
||||
|
||||
// Perform auto-discovery of storage topology
|
||||
let storage_topology = match Self::auto_discover_storage() {
|
||||
Ok(topology) => {
|
||||
debug!("Auto-discovered storage topology: {} single disks, {} mergerfs pools",
|
||||
topology.single_disks.len(), topology.mergerfs_pools.len());
|
||||
Some(topology)
|
||||
}
|
||||
Err(e) => {
|
||||
debug!("Failed to auto-discover storage topology: {}", e);
|
||||
None
|
||||
}
|
||||
};
|
||||
|
||||
// Detect devices for discovered storage
|
||||
let mut detected_devices = std::collections::HashMap::new();
|
||||
if let Some(ref topology) = storage_topology {
|
||||
// Add single disks
|
||||
for disk in &topology.single_disks {
|
||||
if let Ok(devices) = Self::detect_device_for_mount_point_static(&disk.mount_point) {
|
||||
detected_devices.insert(disk.mount_point.clone(), devices);
|
||||
}
|
||||
}
|
||||
|
||||
// Add mergerfs pools and their members
|
||||
for pool in &topology.mergerfs_pools {
|
||||
// Detect devices for the pool itself
|
||||
if let Ok(devices) = Self::detect_device_for_mount_point_static(&pool.mount_point) {
|
||||
detected_devices.insert(pool.mount_point.clone(), devices);
|
||||
}
|
||||
|
||||
// Detect devices for member disks
|
||||
for member in &pool.data_members {
|
||||
if let Ok(devices) = Self::detect_device_for_mount_point_static(member) {
|
||||
detected_devices.insert(member.clone(), devices);
|
||||
}
|
||||
}
|
||||
|
||||
// Detect devices for parity disks
|
||||
for parity in &pool.parity_disks {
|
||||
if let Ok(devices) = Self::detect_device_for_mount_point_static(parity) {
|
||||
detected_devices.insert(parity.clone(), devices);
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Fallback: use legacy filesystem config detection
|
||||
for fs_config in &config.filesystems {
|
||||
if fs_config.monitor {
|
||||
if let Ok(devices) = Self::detect_device_for_mount_point_static(&fs_config.mount_point) {
|
||||
detected_devices.insert(fs_config.mount_point.clone(), devices);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Self {
|
||||
config,
|
||||
temperature_thresholds,
|
||||
detected_devices,
|
||||
storage_topology,
|
||||
}
|
||||
}
|
||||
|
||||
/// Auto-discover storage topology by parsing system information
|
||||
fn auto_discover_storage() -> Result<StorageTopology> {
|
||||
let mounts = Self::parse_proc_mounts()?;
|
||||
let mut single_disks = Vec::new();
|
||||
let mut mergerfs_pools = Vec::new();
|
||||
|
||||
// Filter out unwanted filesystem types and mount points
|
||||
let exclude_fs_types = ["tmpfs", "devtmpfs", "sysfs", "proc", "cgroup", "cgroup2", "devpts"];
|
||||
let exclude_mount_prefixes = ["/proc", "/sys", "/dev", "/tmp", "/run"];
|
||||
|
||||
for mount in mounts {
|
||||
// Skip excluded filesystem types
|
||||
if exclude_fs_types.contains(&mount.fs_type.as_str()) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Skip excluded mount point prefixes
|
||||
if exclude_mount_prefixes.iter().any(|prefix| mount.mount_point.starts_with(prefix)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
match mount.fs_type.as_str() {
|
||||
"fuse.mergerfs" => {
|
||||
// Parse mergerfs pool
|
||||
let data_members = Self::parse_mergerfs_sources(&mount.device);
|
||||
let parity_disks = Self::detect_parity_disks(&data_members);
|
||||
|
||||
mergerfs_pools.push(MergerfsPoolInfo {
|
||||
mount_point: mount.mount_point.clone(),
|
||||
data_members,
|
||||
parity_disks,
|
||||
});
|
||||
|
||||
debug!("Discovered mergerfs pool at {}", mount.mount_point);
|
||||
}
|
||||
"ext4" | "xfs" | "btrfs" | "ntfs" | "vfat" => {
|
||||
// Check if this mount is part of a mergerfs pool
|
||||
let is_mergerfs_member = mergerfs_pools.iter()
|
||||
.any(|pool| pool.data_members.contains(&mount.mount_point) ||
|
||||
pool.parity_disks.contains(&mount.mount_point));
|
||||
|
||||
if !is_mergerfs_member {
|
||||
debug!("Discovered single disk at {}", mount.mount_point);
|
||||
single_disks.push(mount);
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
debug!("Skipping unsupported filesystem type: {}", mount.fs_type);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(StorageTopology {
|
||||
single_disks,
|
||||
mergerfs_pools,
|
||||
})
|
||||
}
|
||||
|
||||
/// Parse /proc/mounts to get all mount information
|
||||
fn parse_proc_mounts() -> Result<Vec<MountInfo>> {
|
||||
let mounts_content = fs::read_to_string("/proc/mounts")?;
|
||||
let mut mounts = Vec::new();
|
||||
|
||||
for line in mounts_content.lines() {
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
if parts.len() >= 3 {
|
||||
mounts.push(MountInfo {
|
||||
device: parts[0].to_string(),
|
||||
mount_point: parts[1].to_string(),
|
||||
fs_type: parts[2].to_string(),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
Ok(mounts)
|
||||
}
|
||||
|
||||
/// Parse mergerfs source string to extract member paths
|
||||
fn parse_mergerfs_sources(source: &str) -> Vec<String> {
|
||||
// MergerFS source format: "/mnt/disk1:/mnt/disk2:/mnt/disk3"
|
||||
source.split(':')
|
||||
.map(|s| s.trim().to_string())
|
||||
.filter(|s| !s.is_empty())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Detect potential parity disks based on data member heuristics
|
||||
fn detect_parity_disks(data_members: &[String]) -> Vec<String> {
|
||||
let mut parity_disks = Vec::new();
|
||||
|
||||
// Heuristic 1: Look for mount points with "parity" in the name
|
||||
if let Ok(mounts) = Self::parse_proc_mounts() {
|
||||
for mount in mounts {
|
||||
if mount.mount_point.to_lowercase().contains("parity") &&
|
||||
(mount.fs_type == "xfs" || mount.fs_type == "ext4") {
|
||||
debug!("Detected parity disk by name: {}", mount.mount_point);
|
||||
parity_disks.push(mount.mount_point);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Heuristic 2: Look for sequential device pattern
|
||||
// If data members are /mnt/disk1, /mnt/disk2, look for /mnt/disk* that's not in data
|
||||
if parity_disks.is_empty() {
|
||||
if let Some(pattern) = Self::extract_mount_pattern(data_members) {
|
||||
if let Ok(mounts) = Self::parse_proc_mounts() {
|
||||
for mount in mounts {
|
||||
if mount.mount_point.starts_with(&pattern) &&
|
||||
!data_members.contains(&mount.mount_point) &&
|
||||
(mount.fs_type == "xfs" || mount.fs_type == "ext4") {
|
||||
debug!("Detected parity disk by pattern: {}", mount.mount_point);
|
||||
parity_disks.push(mount.mount_point);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
parity_disks
|
||||
}
|
||||
|
||||
/// Extract common mount point pattern from data members
|
||||
fn extract_mount_pattern(data_members: &[String]) -> Option<String> {
|
||||
if data_members.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
// Find common prefix (e.g., "/mnt/disk" from "/mnt/disk1", "/mnt/disk2")
|
||||
let first = &data_members[0];
|
||||
if let Some(last_slash) = first.rfind('/') {
|
||||
let base = &first[..last_slash + 1]; // Include the slash
|
||||
|
||||
// Check if all members share this base
|
||||
if data_members.iter().all(|member| member.starts_with(base)) {
|
||||
return Some(base.to_string());
|
||||
}
|
||||
}
|
||||
|
||||
None
|
||||
}
|
||||
|
||||
/// Calculate disk temperature status using hysteresis thresholds
|
||||
fn calculate_temperature_status(&self, metric_name: &str, temperature: f32, status_tracker: &mut StatusTracker) -> Status {
|
||||
status_tracker.calculate_with_hysteresis(metric_name, temperature, &self.temperature_thresholds)
|
||||
}
|
||||
|
||||
|
||||
/// Get storage pools using auto-discovered topology or fallback to configuration
|
||||
fn get_configured_storage_pools(&self) -> Result<Vec<StoragePool>> {
|
||||
if let Some(ref topology) = self.storage_topology {
|
||||
self.get_auto_discovered_storage_pools(topology)
|
||||
} else {
|
||||
self.get_legacy_configured_storage_pools()
|
||||
}
|
||||
}
|
||||
|
||||
/// Get storage pools from auto-discovered topology
|
||||
fn get_auto_discovered_storage_pools(&self, topology: &StorageTopology) -> Result<Vec<StoragePool>> {
|
||||
let mut storage_pools = Vec::new();
|
||||
|
||||
// Group single disks by physical drive for unified pool display
|
||||
let grouped_disks = self.group_filesystems_by_physical_drive(&topology.single_disks)?;
|
||||
|
||||
// Process grouped single disks (each physical drive becomes a pool)
|
||||
for (drive_name, filesystems) in grouped_disks {
|
||||
// Create a unified pool for this physical drive
|
||||
let pool = self.create_physical_drive_pool(&drive_name, &filesystems)?;
|
||||
storage_pools.push(pool);
|
||||
}
|
||||
|
||||
// IMPORTANT: Do not create individual filesystem pools when using auto-discovery
|
||||
// All single disk filesystems should be grouped into physical drive pools above
|
||||
|
||||
// Process mergerfs pools (these remain as logical pools)
|
||||
for pool_info in &topology.mergerfs_pools {
|
||||
if let Ok((total_bytes, used_bytes)) = self.get_filesystem_info(&pool_info.mount_point) {
|
||||
let available_bytes = total_bytes - used_bytes;
|
||||
let usage_percent = if total_bytes > 0 {
|
||||
(used_bytes as f64 / total_bytes as f64) * 100.0
|
||||
} else { 0.0 };
|
||||
|
||||
let size = self.bytes_to_human_readable(total_bytes);
|
||||
let used = self.bytes_to_human_readable(used_bytes);
|
||||
let available = self.bytes_to_human_readable(available_bytes);
|
||||
|
||||
// Collect all member and parity drives
|
||||
let mut all_drives = Vec::new();
|
||||
|
||||
// Add data member drives
|
||||
for member in &pool_info.data_members {
|
||||
if let Some(devices) = self.detected_devices.get(member) {
|
||||
all_drives.extend(devices.clone());
|
||||
}
|
||||
}
|
||||
|
||||
// Add parity drives
|
||||
for parity in &pool_info.parity_disks {
|
||||
if let Some(devices) = self.detected_devices.get(parity) {
|
||||
all_drives.extend(devices.clone());
|
||||
}
|
||||
}
|
||||
|
||||
let underlying_drives = self.get_drive_info_for_devices(&all_drives)?;
|
||||
|
||||
// Calculate pool health
|
||||
let pool_health = self.calculate_mergerfs_pool_health(&pool_info.data_members, &pool_info.parity_disks, &underlying_drives);
|
||||
|
||||
// Generate pool name from mount point
|
||||
let name = pool_info.mount_point.trim_start_matches('/').replace('/', "_");
|
||||
|
||||
storage_pools.push(StoragePool {
|
||||
name,
|
||||
mount_point: pool_info.mount_point.clone(),
|
||||
filesystem: "fuse.mergerfs".to_string(),
|
||||
pool_type: StoragePoolType::MergerfsPool {
|
||||
data_disks: pool_info.data_members.iter()
|
||||
.filter_map(|member| self.detected_devices.get(member).and_then(|devices| devices.first().cloned()))
|
||||
.collect(),
|
||||
parity_disks: pool_info.parity_disks.iter()
|
||||
.filter_map(|parity| self.detected_devices.get(parity).and_then(|devices| devices.first().cloned()))
|
||||
.collect(),
|
||||
},
|
||||
size,
|
||||
used,
|
||||
available,
|
||||
usage_percent: usage_percent as f32,
|
||||
underlying_drives,
|
||||
pool_health,
|
||||
});
|
||||
|
||||
debug!("Auto-discovered mergerfs pool: {} with {} data + {} parity disks",
|
||||
pool_info.mount_point, pool_info.data_members.len(), pool_info.parity_disks.len());
|
||||
}
|
||||
}
|
||||
|
||||
Ok(storage_pools)
|
||||
}
|
||||
|
||||
/// Group filesystems by their backing physical drive
|
||||
fn group_filesystems_by_physical_drive(&self, filesystems: &[MountInfo]) -> Result<std::collections::HashMap<String, Vec<MountInfo>>> {
|
||||
let mut grouped = std::collections::HashMap::new();
|
||||
|
||||
for fs in filesystems {
|
||||
// Get the physical drive name for this mount point
|
||||
if let Some(devices) = self.detected_devices.get(&fs.mount_point) {
|
||||
if let Some(device_name) = devices.first() {
|
||||
// Extract base drive name from detected device
|
||||
let drive_name = Self::extract_base_device(device_name)
|
||||
.unwrap_or_else(|| device_name.clone());
|
||||
|
||||
debug!("Grouping filesystem {} (device: {}) under drive: {}",
|
||||
fs.mount_point, device_name, drive_name);
|
||||
|
||||
grouped.entry(drive_name).or_insert_with(Vec::new).push(fs.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
debug!("Filesystem grouping result: {} drives with filesystems: {:?}",
|
||||
grouped.len(),
|
||||
grouped.keys().collect::<Vec<_>>());
|
||||
|
||||
Ok(grouped)
|
||||
}
|
||||
|
||||
/// Create a physical drive pool containing multiple filesystems
|
||||
fn create_physical_drive_pool(&self, drive_name: &str, filesystems: &[MountInfo]) -> Result<StoragePool> {
|
||||
if filesystems.is_empty() {
|
||||
return Err(anyhow::anyhow!("No filesystems for drive {}", drive_name));
|
||||
}
|
||||
|
||||
// Calculate total usage across all filesystems on this drive
|
||||
let mut total_capacity = 0u64;
|
||||
let mut total_used = 0u64;
|
||||
|
||||
for fs in filesystems {
|
||||
if let Ok((capacity, used)) = self.get_filesystem_info(&fs.mount_point) {
|
||||
total_capacity += capacity;
|
||||
total_used += used;
|
||||
}
|
||||
}
|
||||
|
||||
let total_available = total_capacity.saturating_sub(total_used);
|
||||
let usage_percent = if total_capacity > 0 {
|
||||
(total_used as f64 / total_capacity as f64) * 100.0
|
||||
} else { 0.0 };
|
||||
|
||||
// Get drive information for SMART data
|
||||
let device_names = vec![drive_name.to_string()];
|
||||
let underlying_drives = self.get_drive_info_for_devices(&device_names)?;
|
||||
|
||||
// Collect filesystem mount points for this drive
|
||||
let filesystem_mount_points: Vec<String> = filesystems.iter()
|
||||
.map(|fs| fs.mount_point.clone())
|
||||
.collect();
|
||||
|
||||
Ok(StoragePool {
|
||||
name: drive_name.to_string(),
|
||||
mount_point: format!("(physical drive)"), // Special marker for physical drives
|
||||
filesystem: "physical".to_string(),
|
||||
pool_type: StoragePoolType::PhysicalDrive {
|
||||
filesystems: filesystem_mount_points,
|
||||
},
|
||||
size: self.bytes_to_human_readable(total_capacity),
|
||||
used: self.bytes_to_human_readable(total_used),
|
||||
available: self.bytes_to_human_readable(total_available),
|
||||
usage_percent: usage_percent as f32,
|
||||
pool_health: if underlying_drives.iter().all(|d| d.health_status == "PASSED") {
|
||||
PoolHealth::Healthy
|
||||
} else {
|
||||
PoolHealth::Critical
|
||||
},
|
||||
underlying_drives,
|
||||
})
|
||||
}
|
||||
|
||||
/// Calculate pool health specifically for mergerfs pools
|
||||
fn calculate_mergerfs_pool_health(&self, data_members: &[String], parity_disks: &[String], drives: &[DriveInfo]) -> PoolHealth {
|
||||
// Get device names for data and parity drives
|
||||
let mut data_device_names = Vec::new();
|
||||
let mut parity_device_names = Vec::new();
|
||||
|
||||
for member in data_members {
|
||||
if let Some(devices) = self.detected_devices.get(member) {
|
||||
data_device_names.extend(devices.clone());
|
||||
}
|
||||
}
|
||||
|
||||
for parity in parity_disks {
|
||||
if let Some(devices) = self.detected_devices.get(parity) {
|
||||
parity_device_names.extend(devices.clone());
|
||||
}
|
||||
}
|
||||
|
||||
let failed_data = drives.iter()
|
||||
.filter(|d| data_device_names.contains(&d.device) && d.health_status != "PASSED")
|
||||
.count();
|
||||
let failed_parity = drives.iter()
|
||||
.filter(|d| parity_device_names.contains(&d.device) && d.health_status != "PASSED")
|
||||
.count();
|
||||
|
||||
match (failed_data, failed_parity) {
|
||||
(0, 0) => PoolHealth::Healthy,
|
||||
(1, 0) => PoolHealth::Degraded, // Can recover with parity
|
||||
(0, 1) => PoolHealth::Degraded, // Lost parity protection
|
||||
_ => PoolHealth::Critical, // Multiple failures
|
||||
}
|
||||
}
|
||||
|
||||
/// Fallback to legacy configuration-based storage pools
|
||||
fn get_legacy_configured_storage_pools(&self) -> Result<Vec<StoragePool>> {
|
||||
let mut storage_pools = Vec::new();
|
||||
let mut processed_pools = std::collections::HashSet::new();
|
||||
|
||||
// Legacy implementation: use filesystem configuration
|
||||
for fs_config in &self.config.filesystems {
|
||||
if !fs_config.monitor {
|
||||
continue;
|
||||
}
|
||||
|
||||
let (pool_type, skip_in_single_mode) = self.determine_pool_type(&fs_config.storage_type);
|
||||
|
||||
// Skip member disks if they're part of a pool
|
||||
if skip_in_single_mode {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Check if this pool was already processed (in case of multiple member disks)
|
||||
let pool_key = match &pool_type {
|
||||
StoragePoolType::MergerfsPool { .. } => {
|
||||
// For mergerfs pools, use the main mount point
|
||||
if fs_config.fs_type == "fuse.mergerfs" {
|
||||
fs_config.mount_point.clone()
|
||||
} else {
|
||||
continue; // Skip member disks
|
||||
}
|
||||
}
|
||||
_ => fs_config.mount_point.clone()
|
||||
};
|
||||
|
||||
if processed_pools.contains(&pool_key) {
|
||||
continue;
|
||||
}
|
||||
processed_pools.insert(pool_key.clone());
|
||||
|
||||
// Get filesystem stats for the mount point
|
||||
match self.get_filesystem_info(&fs_config.mount_point) {
|
||||
Ok((total_bytes, used_bytes)) => {
|
||||
let available_bytes = total_bytes - used_bytes;
|
||||
let usage_percent = if total_bytes > 0 {
|
||||
(used_bytes as f64 / total_bytes as f64) * 100.0
|
||||
} else { 0.0 };
|
||||
|
||||
// Convert bytes to human-readable format
|
||||
let size = self.bytes_to_human_readable(total_bytes);
|
||||
let used = self.bytes_to_human_readable(used_bytes);
|
||||
let available = self.bytes_to_human_readable(available_bytes);
|
||||
|
||||
// Get underlying drives based on pool type
|
||||
let underlying_drives = self.get_pool_drives(&pool_type, &fs_config.mount_point)?;
|
||||
|
||||
// Calculate pool health
|
||||
let pool_health = self.calculate_pool_health(&pool_type, &underlying_drives);
|
||||
let drive_count = underlying_drives.len();
|
||||
|
||||
storage_pools.push(StoragePool {
|
||||
name: fs_config.name.clone(),
|
||||
mount_point: fs_config.mount_point.clone(),
|
||||
filesystem: fs_config.fs_type.clone(),
|
||||
pool_type: pool_type.clone(),
|
||||
size,
|
||||
used,
|
||||
available,
|
||||
usage_percent: usage_percent as f32,
|
||||
underlying_drives,
|
||||
pool_health,
|
||||
});
|
||||
|
||||
debug!(
|
||||
"Legacy configured storage pool '{}' ({:?}) at {} with {} drives, health: {:?}",
|
||||
fs_config.name, pool_type, fs_config.mount_point, drive_count, pool_health
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
debug!(
|
||||
"Failed to get filesystem info for storage pool '{}': {}",
|
||||
fs_config.name, e
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(storage_pools)
|
||||
}
|
||||
|
||||
/// Determine the storage pool type from configuration
|
||||
fn determine_pool_type(&self, storage_type: &str) -> (StoragePoolType, bool) {
|
||||
match storage_type {
|
||||
"single" => (StoragePoolType::Single, false),
|
||||
"mergerfs_pool" | "mergerfs" => {
|
||||
// Find associated member disks
|
||||
let data_disks = self.find_pool_member_disks("mergerfs_member");
|
||||
let parity_disks = self.find_pool_member_disks("parity");
|
||||
(StoragePoolType::MergerfsPool { data_disks, parity_disks }, false)
|
||||
}
|
||||
"mergerfs_member" => (StoragePoolType::Single, true), // Skip, part of pool
|
||||
"parity" => (StoragePoolType::Single, true), // Skip, part of pool
|
||||
"raid1" | "raid5" | "raid6" => {
|
||||
let member_disks = self.find_pool_member_disks(&format!("{}_member", storage_type));
|
||||
(StoragePoolType::RaidArray {
|
||||
level: storage_type.to_uppercase(),
|
||||
member_disks,
|
||||
spare_disks: Vec::new()
|
||||
}, false)
|
||||
}
|
||||
_ => (StoragePoolType::Single, false) // Default to single
|
||||
}
|
||||
}
|
||||
|
||||
/// Find member disks for a specific storage type
|
||||
fn find_pool_member_disks(&self, member_type: &str) -> Vec<String> {
|
||||
let mut member_disks = Vec::new();
|
||||
|
||||
for fs_config in &self.config.filesystems {
|
||||
if fs_config.storage_type == member_type && fs_config.monitor {
|
||||
// Get device names for this mount point
|
||||
if let Some(devices) = self.detected_devices.get(&fs_config.mount_point) {
|
||||
member_disks.extend(devices.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
member_disks
|
||||
}
|
||||
|
||||
/// Get drive information for a specific pool type
|
||||
fn get_pool_drives(&self, pool_type: &StoragePoolType, mount_point: &str) -> Result<Vec<DriveInfo>> {
|
||||
match pool_type {
|
||||
StoragePoolType::Single => {
|
||||
// Single disk - use detected devices for this mount point
|
||||
let device_names = self.detected_devices.get(mount_point).cloned().unwrap_or_default();
|
||||
self.get_drive_info_for_devices(&device_names)
|
||||
}
|
||||
StoragePoolType::PhysicalDrive { .. } => {
|
||||
// Physical drive - get drive info for the drive directly (mount_point not used)
|
||||
let device_names = vec![mount_point.to_string()];
|
||||
self.get_drive_info_for_devices(&device_names)
|
||||
}
|
||||
StoragePoolType::MergerfsPool { data_disks, parity_disks } => {
|
||||
// Mergerfs pool - collect all member drives
|
||||
let mut all_disks = data_disks.clone();
|
||||
all_disks.extend(parity_disks.clone());
|
||||
self.get_drive_info_for_devices(&all_disks)
|
||||
}
|
||||
StoragePoolType::RaidArray { member_disks, spare_disks, .. } => {
|
||||
// RAID array - collect member and spare drives
|
||||
let mut all_disks = member_disks.clone();
|
||||
all_disks.extend(spare_disks.clone());
|
||||
self.get_drive_info_for_devices(&all_disks)
|
||||
}
|
||||
StoragePoolType::ZfsPool { .. } => {
|
||||
// ZFS pool - use detected devices (future implementation)
|
||||
let device_names = self.detected_devices.get(mount_point).cloned().unwrap_or_default();
|
||||
self.get_drive_info_for_devices(&device_names)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Calculate pool health based on drive status and pool type
|
||||
fn calculate_pool_health(&self, pool_type: &StoragePoolType, drives: &[DriveInfo]) -> PoolHealth {
|
||||
match pool_type {
|
||||
StoragePoolType::Single => {
|
||||
// Single disk - health is just the drive health
|
||||
if drives.is_empty() {
|
||||
PoolHealth::Unknown
|
||||
} else if drives.iter().all(|d| d.health_status == "PASSED") {
|
||||
PoolHealth::Healthy
|
||||
} else {
|
||||
PoolHealth::Critical
|
||||
}
|
||||
}
|
||||
StoragePoolType::PhysicalDrive { .. } => {
|
||||
// Physical drive - health is just the drive health (similar to Single)
|
||||
if drives.is_empty() {
|
||||
PoolHealth::Unknown
|
||||
} else if drives.iter().all(|d| d.health_status == "PASSED") {
|
||||
PoolHealth::Healthy
|
||||
} else {
|
||||
PoolHealth::Critical
|
||||
}
|
||||
}
|
||||
StoragePoolType::MergerfsPool { data_disks, parity_disks } => {
|
||||
let failed_data = drives.iter()
|
||||
.filter(|d| data_disks.contains(&d.device) && d.health_status != "PASSED")
|
||||
.count();
|
||||
let failed_parity = drives.iter()
|
||||
.filter(|d| parity_disks.contains(&d.device) && d.health_status != "PASSED")
|
||||
.count();
|
||||
|
||||
match (failed_data, failed_parity) {
|
||||
(0, 0) => PoolHealth::Healthy,
|
||||
(1, 0) => PoolHealth::Degraded, // Can recover with parity
|
||||
(0, 1) => PoolHealth::Degraded, // Lost parity protection
|
||||
_ => PoolHealth::Critical, // Multiple failures
|
||||
}
|
||||
}
|
||||
StoragePoolType::RaidArray { level, .. } => {
|
||||
let failed_drives = drives.iter().filter(|d| d.health_status != "PASSED").count();
|
||||
|
||||
// Basic RAID health logic (can be enhanced per RAID level)
|
||||
match failed_drives {
|
||||
0 => PoolHealth::Healthy,
|
||||
1 if level.contains('1') || level.contains('5') || level.contains('6') => PoolHealth::Degraded,
|
||||
_ => PoolHealth::Critical,
|
||||
}
|
||||
}
|
||||
StoragePoolType::ZfsPool { .. } => {
|
||||
// ZFS health would require zpool status parsing (future)
|
||||
if drives.iter().all(|d| d.health_status == "PASSED") {
|
||||
PoolHealth::Healthy
|
||||
} else {
|
||||
PoolHealth::Degraded
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Get drive information for a list of device names
|
||||
fn get_drive_info_for_devices(&self, device_names: &[String]) -> Result<Vec<DriveInfo>> {
|
||||
let mut drives = Vec::new();
|
||||
|
||||
for device_name in device_names {
|
||||
let device_path = format!("/dev/{}", device_name);
|
||||
|
||||
// Get SMART data for this drive
|
||||
let (health_status, temperature, wear_level) = self.get_smart_data(&device_path);
|
||||
|
||||
drives.push(DriveInfo {
|
||||
device: device_name.clone(),
|
||||
health_status: health_status.clone(),
|
||||
temperature,
|
||||
wear_level,
|
||||
});
|
||||
|
||||
debug!(
|
||||
"Drive info for {}: health={}, temp={:?}°C, wear={:?}%",
|
||||
device_name, health_status, temperature, wear_level
|
||||
);
|
||||
}
|
||||
|
||||
Ok(drives)
|
||||
}
|
||||
|
||||
/// Get SMART data for a drive (health, temperature, wear level)
|
||||
fn get_smart_data(&self, device_path: &str) -> (String, Option<f32>, Option<f32>) {
|
||||
// Try to get SMART data using smartctl
|
||||
let output = Command::new("sudo")
|
||||
.arg("smartctl")
|
||||
.arg("-a")
|
||||
.arg(device_path)
|
||||
.output();
|
||||
|
||||
match output {
|
||||
Ok(result) if result.status.success() => {
|
||||
let stdout = String::from_utf8_lossy(&result.stdout);
|
||||
|
||||
// Parse health status
|
||||
let health = if stdout.contains("PASSED") {
|
||||
"PASSED".to_string()
|
||||
} else if stdout.contains("FAILED") {
|
||||
"FAILED".to_string()
|
||||
} else {
|
||||
"UNKNOWN".to_string()
|
||||
};
|
||||
|
||||
// Parse temperature (look for various temperature indicators)
|
||||
let temperature = self.parse_temperature_from_smart(&stdout);
|
||||
|
||||
// Parse wear level (for SSDs)
|
||||
let wear_level = self.parse_wear_level_from_smart(&stdout);
|
||||
|
||||
(health, temperature, wear_level)
|
||||
}
|
||||
_ => {
|
||||
debug!("Failed to get SMART data for {}", device_path);
|
||||
("UNKNOWN".to_string(), None, None)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Parse temperature from SMART output
|
||||
fn parse_temperature_from_smart(&self, smart_output: &str) -> Option<f32> {
|
||||
for line in smart_output.lines() {
|
||||
// Look for temperature in various formats
|
||||
if line.contains("Temperature_Celsius") || line.contains("Temperature") {
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
if parts.len() >= 10 {
|
||||
if let Ok(temp) = parts[9].parse::<f32>() {
|
||||
return Some(temp);
|
||||
}
|
||||
}
|
||||
}
|
||||
// NVMe drives might show temperature differently
|
||||
if line.contains("temperature:") {
|
||||
if let Some(temp_part) = line.split("temperature:").nth(1) {
|
||||
if let Some(temp_str) = temp_part.split_whitespace().next() {
|
||||
if let Ok(temp) = temp_str.parse::<f32>() {
|
||||
return Some(temp);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Parse wear level from SMART output (SSD wear leveling)
|
||||
/// Supports both NVMe and SATA SSD wear indicators
|
||||
fn parse_wear_level_from_smart(&self, smart_output: &str) -> Option<f32> {
|
||||
for line in smart_output.lines() {
|
||||
let line = line.trim();
|
||||
|
||||
// NVMe drives - direct percentage used
|
||||
if line.contains("Percentage Used:") {
|
||||
if let Some(wear_part) = line.split("Percentage Used:").nth(1) {
|
||||
if let Some(wear_str) = wear_part.split('%').next() {
|
||||
if let Ok(wear) = wear_str.trim().parse::<f32>() {
|
||||
return Some(wear);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// SATA SSD attributes - parse SMART table format
|
||||
// Format: ID ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
if parts.len() >= 10 {
|
||||
// SSD Life Left / Percent Lifetime Remaining (higher = less wear)
|
||||
if line.contains("SSD_Life_Left") || line.contains("Percent_Lifetime_Remain") {
|
||||
if let Ok(remaining) = parts[3].parse::<f32>() { // VALUE column
|
||||
return Some(100.0 - remaining); // Convert remaining to used
|
||||
}
|
||||
}
|
||||
|
||||
// Media Wearout Indicator (lower = more wear, normalize to 0-100)
|
||||
if line.contains("Media_Wearout_Indicator") {
|
||||
if let Ok(remaining) = parts[3].parse::<f32>() { // VALUE column
|
||||
return Some(100.0 - remaining); // Convert remaining to used
|
||||
}
|
||||
}
|
||||
|
||||
// Wear Leveling Count (higher = less wear, but varies by manufacturer)
|
||||
if line.contains("Wear_Leveling_Count") {
|
||||
if let Ok(wear_count) = parts[3].parse::<f32>() { // VALUE column
|
||||
// Most SSDs: 100 = new, decreases with wear
|
||||
if wear_count <= 100.0 {
|
||||
return Some(100.0 - wear_count);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Total LBAs Written - calculate against typical endurance if available
|
||||
// This is more complex and manufacturer-specific, so we skip for now
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Convert bytes to human-readable format
|
||||
fn bytes_to_human_readable(&self, bytes: u64) -> String {
|
||||
const UNITS: &[&str] = &["B", "K", "M", "G", "T"];
|
||||
let mut size = bytes as f64;
|
||||
let mut unit_index = 0;
|
||||
|
||||
while size >= 1024.0 && unit_index < UNITS.len() - 1 {
|
||||
size /= 1024.0;
|
||||
unit_index += 1;
|
||||
}
|
||||
|
||||
if unit_index == 0 {
|
||||
format!("{:.0}{}", size, UNITS[unit_index])
|
||||
} else {
|
||||
format!("{:.1}{}", size, UNITS[unit_index])
|
||||
}
|
||||
}
|
||||
|
||||
/// Convert bytes to gigabytes
|
||||
fn bytes_to_gb(&self, bytes: u64) -> f32 {
|
||||
bytes as f32 / (1024.0 * 1024.0 * 1024.0)
|
||||
}
|
||||
|
||||
/// Detect device backing a mount point using lsblk (static version for startup)
|
||||
fn detect_device_for_mount_point_static(mount_point: &str) -> Result<Vec<String>> {
|
||||
let output = Command::new("lsblk")
|
||||
.args(&["-n", "-o", "NAME,MOUNTPOINT"])
|
||||
.output()?;
|
||||
|
||||
if !output.status.success() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
|
||||
let output_str = String::from_utf8_lossy(&output.stdout);
|
||||
|
||||
for line in output_str.lines() {
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
if parts.len() >= 2 && parts[1] == mount_point {
|
||||
// Remove tree symbols and extract device name (e.g., "├─nvme0n1p2" -> "nvme0n1p2")
|
||||
let device_name = parts[0]
|
||||
.trim_start_matches('├')
|
||||
.trim_start_matches('└')
|
||||
.trim_start_matches('─')
|
||||
.trim();
|
||||
|
||||
// Extract base device name (e.g., "nvme0n1p2" -> "nvme0n1")
|
||||
if let Some(base_device) = Self::extract_base_device(device_name) {
|
||||
return Ok(vec![base_device]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(Vec::new())
|
||||
}
|
||||
|
||||
/// Extract base device name from partition (e.g., "nvme0n1p2" -> "nvme0n1", "sda1" -> "sda")
|
||||
fn extract_base_device(device_name: &str) -> Option<String> {
|
||||
// Handle NVMe devices (nvme0n1p1 -> nvme0n1)
|
||||
if device_name.starts_with("nvme") {
|
||||
if let Some(p_pos) = device_name.find('p') {
|
||||
return Some(device_name[..p_pos].to_string());
|
||||
}
|
||||
}
|
||||
|
||||
// Handle traditional devices (sda1 -> sda)
|
||||
if device_name.len() > 1 {
|
||||
let chars: Vec<char> = device_name.chars().collect();
|
||||
let mut end_idx = chars.len();
|
||||
|
||||
// Find where the device name ends and partition number begins
|
||||
for (i, &c) in chars.iter().enumerate().rev() {
|
||||
if !c.is_ascii_digit() {
|
||||
end_idx = i + 1;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if end_idx > 0 && end_idx < chars.len() {
|
||||
return Some(chars[..end_idx].iter().collect());
|
||||
}
|
||||
}
|
||||
|
||||
// If no partition detected, return as-is
|
||||
Some(device_name.to_string())
|
||||
}
|
||||
|
||||
|
||||
/// Get filesystem info using df command
|
||||
fn get_filesystem_info(&self, path: &str) -> Result<(u64, u64)> {
|
||||
let output = Command::new("df")
|
||||
.arg("--block-size=1")
|
||||
.arg(path)
|
||||
.output()?;
|
||||
|
||||
if !output.status.success() {
|
||||
return Err(anyhow::anyhow!("df command failed for {}", path));
|
||||
}
|
||||
|
||||
let output_str = String::from_utf8(output.stdout)?;
|
||||
let lines: Vec<&str> = output_str.lines().collect();
|
||||
|
||||
if lines.len() < 2 {
|
||||
return Err(anyhow::anyhow!("Unexpected df output format"));
|
||||
}
|
||||
|
||||
let fields: Vec<&str> = lines[1].split_whitespace().collect();
|
||||
if fields.len() < 4 {
|
||||
return Err(anyhow::anyhow!("Unexpected df fields count"));
|
||||
}
|
||||
|
||||
let total_bytes = fields[1].parse::<u64>()?;
|
||||
let used_bytes = fields[2].parse::<u64>()?;
|
||||
|
||||
Ok((total_bytes, used_bytes))
|
||||
}
|
||||
|
||||
|
||||
/// Parse size string (e.g., "120G", "45M") to GB value
|
||||
fn parse_size_to_gb(&self, size_str: &str) -> f32 {
|
||||
let size_str = size_str.trim();
|
||||
if size_str.is_empty() || size_str == "-" {
|
||||
return 0.0;
|
||||
}
|
||||
|
||||
// Extract numeric part and unit
|
||||
let (num_str, unit) = if let Some(last_char) = size_str.chars().last() {
|
||||
if last_char.is_alphabetic() {
|
||||
let num_part = &size_str[..size_str.len() - 1];
|
||||
let unit_part = &size_str[size_str.len() - 1..];
|
||||
(num_part, unit_part)
|
||||
} else {
|
||||
(size_str, "")
|
||||
}
|
||||
} else {
|
||||
(size_str, "")
|
||||
};
|
||||
|
||||
let number: f32 = num_str.parse().unwrap_or(0.0);
|
||||
|
||||
match unit.to_uppercase().as_str() {
|
||||
"T" | "TB" => number * 1024.0,
|
||||
"G" | "GB" => number,
|
||||
"M" | "MB" => number / 1024.0,
|
||||
"K" | "KB" => number / (1024.0 * 1024.0),
|
||||
"B" | "" => number / (1024.0 * 1024.0 * 1024.0),
|
||||
_ => number, // Assume GB if unknown unit
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
impl Collector for DiskCollector {
|
||||
|
||||
async fn collect(&self, status_tracker: &mut StatusTracker) -> Result<Vec<Metric>, CollectorError> {
|
||||
let start_time = Instant::now();
|
||||
debug!("Collecting storage pool and individual drive metrics");
|
||||
|
||||
let mut metrics = Vec::new();
|
||||
|
||||
// Get configured storage pools with individual drive data
|
||||
let storage_pools = match self.get_configured_storage_pools() {
|
||||
Ok(pools) => {
|
||||
debug!("Found {} storage pools", pools.len());
|
||||
pools
|
||||
}
|
||||
Err(e) => {
|
||||
debug!("Failed to get storage pools: {}", e);
|
||||
Vec::new()
|
||||
}
|
||||
};
|
||||
|
||||
// Generate metrics for each storage pool and its underlying drives
|
||||
for storage_pool in &storage_pools {
|
||||
let timestamp = chrono::Utc::now().timestamp() as u64;
|
||||
|
||||
// Storage pool overall metrics
|
||||
let pool_name = &storage_pool.name;
|
||||
|
||||
// Parse size strings to get actual values for calculations
|
||||
let size_gb = self.parse_size_to_gb(&storage_pool.size);
|
||||
let used_gb = self.parse_size_to_gb(&storage_pool.used);
|
||||
let avail_gb = self.parse_size_to_gb(&storage_pool.available);
|
||||
|
||||
// Calculate status based on configured thresholds and pool health
|
||||
let usage_status = if storage_pool.usage_percent >= self.config.usage_critical_percent {
|
||||
Status::Critical
|
||||
} else if storage_pool.usage_percent >= self.config.usage_warning_percent {
|
||||
Status::Warning
|
||||
} else {
|
||||
Status::Ok
|
||||
};
|
||||
|
||||
let pool_status = match storage_pool.pool_health {
|
||||
PoolHealth::Critical => Status::Critical,
|
||||
PoolHealth::Degraded => Status::Warning,
|
||||
PoolHealth::Rebuilding => Status::Warning,
|
||||
PoolHealth::Healthy => usage_status,
|
||||
PoolHealth::Unknown => Status::Unknown,
|
||||
};
|
||||
|
||||
// Storage pool info metrics
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_mount_point", pool_name),
|
||||
value: MetricValue::String(storage_pool.mount_point.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("Mount: {}", storage_pool.mount_point)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_filesystem", pool_name),
|
||||
value: MetricValue::String(storage_pool.filesystem.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("FS: {}", storage_pool.filesystem)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Enhanced pool type information
|
||||
let pool_type_str = match &storage_pool.pool_type {
|
||||
StoragePoolType::Single => "single".to_string(),
|
||||
StoragePoolType::PhysicalDrive { filesystems } => {
|
||||
format!("drive ({})", filesystems.len())
|
||||
}
|
||||
StoragePoolType::MergerfsPool { data_disks, parity_disks } => {
|
||||
format!("mergerfs ({}+{})", data_disks.len(), parity_disks.len())
|
||||
}
|
||||
StoragePoolType::RaidArray { level, member_disks, spare_disks } => {
|
||||
format!("{} ({}+{})", level, member_disks.len(), spare_disks.len())
|
||||
}
|
||||
StoragePoolType::ZfsPool { pool_name, .. } => {
|
||||
format!("zfs ({})", pool_name)
|
||||
}
|
||||
};
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_pool_type", pool_name),
|
||||
value: MetricValue::String(pool_type_str.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("Type: {}", pool_type_str)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Pool health status
|
||||
let health_str = match storage_pool.pool_health {
|
||||
PoolHealth::Healthy => "healthy",
|
||||
PoolHealth::Degraded => "degraded",
|
||||
PoolHealth::Critical => "critical",
|
||||
PoolHealth::Rebuilding => "rebuilding",
|
||||
PoolHealth::Unknown => "unknown",
|
||||
};
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_pool_health", pool_name),
|
||||
value: MetricValue::String(health_str.to_string()),
|
||||
unit: None,
|
||||
description: Some(format!("Health: {}", health_str)),
|
||||
status: pool_status,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Storage pool size metrics
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_total_gb", pool_name),
|
||||
value: MetricValue::Float(size_gb),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("Total: {}", storage_pool.size)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_used_gb", pool_name),
|
||||
value: MetricValue::Float(used_gb),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("Used: {}", storage_pool.used)),
|
||||
status: pool_status,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_available_gb", pool_name),
|
||||
value: MetricValue::Float(avail_gb),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("Available: {}", storage_pool.available)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_usage_percent", pool_name),
|
||||
value: MetricValue::Float(storage_pool.usage_percent),
|
||||
unit: Some("%".to_string()),
|
||||
description: Some(format!("Usage: {:.1}%", storage_pool.usage_percent)),
|
||||
status: pool_status,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Individual drive metrics for this storage pool
|
||||
for drive in &storage_pool.underlying_drives {
|
||||
// Drive health status
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_{}_health", pool_name, drive.device),
|
||||
value: MetricValue::String(drive.health_status.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("{}: {}", drive.device, drive.health_status)),
|
||||
status: if drive.health_status == "PASSED" { Status::Ok }
|
||||
else if drive.health_status == "FAILED" { Status::Critical }
|
||||
else { Status::Unknown },
|
||||
timestamp,
|
||||
});
|
||||
|
||||
// Drive temperature
|
||||
if let Some(temp) = drive.temperature {
|
||||
let temp_status = self.calculate_temperature_status(
|
||||
&format!("disk_{}_{}_temperature", pool_name, drive.device),
|
||||
temp,
|
||||
status_tracker
|
||||
);
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_{}_temperature", pool_name, drive.device),
|
||||
value: MetricValue::Float(temp),
|
||||
unit: Some("°C".to_string()),
|
||||
description: Some(format!("{}: {:.0}°C", drive.device, temp)),
|
||||
status: temp_status,
|
||||
timestamp,
|
||||
});
|
||||
}
|
||||
|
||||
// Drive wear level (for SSDs)
|
||||
if let Some(wear) = drive.wear_level {
|
||||
let wear_status = if wear >= self.config.wear_critical_percent { Status::Critical }
|
||||
else if wear >= self.config.wear_warning_percent { Status::Warning }
|
||||
else { Status::Ok };
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_{}_wear_percent", pool_name, drive.device),
|
||||
value: MetricValue::Float(wear),
|
||||
unit: Some("%".to_string()),
|
||||
description: Some(format!("{}: {:.0}% wear", drive.device, wear)),
|
||||
status: wear_status,
|
||||
timestamp,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Individual filesystem metrics for PhysicalDrive pools
|
||||
if let StoragePoolType::PhysicalDrive { filesystems } = &storage_pool.pool_type {
|
||||
for filesystem_mount in filesystems {
|
||||
if let Ok((total_bytes, used_bytes)) = self.get_filesystem_info(filesystem_mount) {
|
||||
let available_bytes = total_bytes - used_bytes;
|
||||
let usage_percent = if total_bytes > 0 {
|
||||
(used_bytes as f64 / total_bytes as f64) * 100.0
|
||||
} else { 0.0 };
|
||||
|
||||
let filesystem_name = if filesystem_mount == "/" {
|
||||
"root".to_string()
|
||||
} else {
|
||||
filesystem_mount.trim_start_matches('/').replace('/', "_")
|
||||
};
|
||||
|
||||
// Calculate filesystem status based on usage
|
||||
let fs_status = if usage_percent >= self.config.usage_critical_percent as f64 {
|
||||
Status::Critical
|
||||
} else if usage_percent >= self.config.usage_warning_percent as f64 {
|
||||
Status::Warning
|
||||
} else {
|
||||
Status::Ok
|
||||
};
|
||||
|
||||
// Filesystem usage metrics
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_fs_{}_usage_percent", pool_name, filesystem_name),
|
||||
value: MetricValue::Float(usage_percent as f32),
|
||||
unit: Some("%".to_string()),
|
||||
description: Some(format!("{}: {:.0}%", filesystem_mount, usage_percent)),
|
||||
status: fs_status.clone(),
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_fs_{}_used_gb", pool_name, filesystem_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(used_bytes)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("{}: {}GB used", filesystem_mount, self.bytes_to_human_readable(used_bytes))),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_fs_{}_total_gb", pool_name, filesystem_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(total_bytes)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("{}: {}GB total", filesystem_mount, self.bytes_to_human_readable(total_bytes))),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_fs_{}_available_gb", pool_name, filesystem_name),
|
||||
value: MetricValue::Float(self.bytes_to_gb(available_bytes)),
|
||||
unit: Some("GB".to_string()),
|
||||
description: Some(format!("{}: {}GB available", filesystem_mount, self.bytes_to_human_readable(available_bytes))),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
metrics.push(Metric {
|
||||
name: format!("disk_{}_fs_{}_mount_point", pool_name, filesystem_name),
|
||||
value: MetricValue::String(filesystem_mount.clone()),
|
||||
unit: None,
|
||||
description: Some(format!("Mount: {}", filesystem_mount)),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Add storage pool count metric
|
||||
metrics.push(Metric {
|
||||
name: "disk_count".to_string(),
|
||||
value: MetricValue::Integer(storage_pools.len() as i64),
|
||||
unit: None,
|
||||
description: Some(format!("Total storage pools: {}", storage_pools.len())),
|
||||
status: Status::Ok,
|
||||
timestamp: chrono::Utc::now().timestamp() as u64,
|
||||
});
|
||||
|
||||
|
||||
let collection_time = start_time.elapsed();
|
||||
debug!(
|
||||
"Multi-disk collection completed in {:?} with {} metrics",
|
||||
collection_time,
|
||||
metrics.len()
|
||||
);
|
||||
|
||||
Ok(metrics)
|
||||
}
|
||||
}
|
||||
@@ -37,6 +37,22 @@ impl NixOSCollector {
|
||||
}
|
||||
|
||||
/// Get configuration hash from deployed nix store system
|
||||
/// Get git commit hash from rebuild process
|
||||
fn get_git_commit(&self) -> Result<String, Box<dyn std::error::Error>> {
|
||||
let commit_file = "/var/lib/cm-dashboard/git-commit";
|
||||
match std::fs::read_to_string(commit_file) {
|
||||
Ok(content) => {
|
||||
let commit_hash = content.trim();
|
||||
if commit_hash.len() >= 7 {
|
||||
Ok(commit_hash.to_string())
|
||||
} else {
|
||||
Err("Git commit hash too short".into())
|
||||
}
|
||||
}
|
||||
Err(e) => Err(format!("Failed to read git commit file: {}", e).into())
|
||||
}
|
||||
}
|
||||
|
||||
fn get_config_hash(&self) -> Result<String, Box<dyn std::error::Error>> {
|
||||
// Read the symlink target of /run/current-system to get nix store path
|
||||
let output = Command::new("readlink")
|
||||
@@ -74,25 +90,25 @@ impl Collector for NixOSCollector {
|
||||
let mut metrics = Vec::new();
|
||||
let timestamp = chrono::Utc::now().timestamp() as u64;
|
||||
|
||||
// Collect NixOS build information (config hash)
|
||||
match self.get_config_hash() {
|
||||
Ok(config_hash) => {
|
||||
// Collect git commit information (shows what's actually deployed)
|
||||
match self.get_git_commit() {
|
||||
Ok(git_commit) => {
|
||||
metrics.push(Metric {
|
||||
name: "system_nixos_build".to_string(),
|
||||
value: MetricValue::String(config_hash),
|
||||
value: MetricValue::String(git_commit),
|
||||
unit: None,
|
||||
description: Some("NixOS deployed configuration hash".to_string()),
|
||||
description: Some("Git commit hash of deployed configuration".to_string()),
|
||||
status: Status::Ok,
|
||||
timestamp,
|
||||
});
|
||||
}
|
||||
Err(e) => {
|
||||
debug!("Failed to get config hash: {}", e);
|
||||
debug!("Failed to get git commit: {}", e);
|
||||
metrics.push(Metric {
|
||||
name: "system_nixos_build".to_string(),
|
||||
value: MetricValue::String("unknown".to_string()),
|
||||
unit: None,
|
||||
description: Some("NixOS config hash (failed to detect)".to_string()),
|
||||
description: Some("Git commit hash (failed to detect)".to_string()),
|
||||
status: Status::Unknown,
|
||||
timestamp,
|
||||
});
|
||||
|
||||
@@ -136,8 +136,21 @@ impl SystemdCollector {
|
||||
/// Auto-discover interesting services to monitor (internal version that doesn't update state)
|
||||
fn discover_services_internal(&self) -> Result<(Vec<String>, std::collections::HashMap<String, ServiceStatusInfo>)> {
|
||||
debug!("Starting systemd service discovery with status caching");
|
||||
// Get all services (includes inactive, running, failed - everything)
|
||||
let units_output = Command::new("systemctl")
|
||||
|
||||
// First: Get all service unit files (includes services that have never been started)
|
||||
let unit_files_output = Command::new("systemctl")
|
||||
.arg("list-unit-files")
|
||||
.arg("--type=service")
|
||||
.arg("--no-pager")
|
||||
.arg("--plain")
|
||||
.output()?;
|
||||
|
||||
if !unit_files_output.status.success() {
|
||||
return Err(anyhow::anyhow!("systemctl list-unit-files command failed"));
|
||||
}
|
||||
|
||||
// Second: Get runtime status of all units
|
||||
let units_status_output = Command::new("systemctl")
|
||||
.arg("list-units")
|
||||
.arg("--type=service")
|
||||
.arg("--all")
|
||||
@@ -145,22 +158,33 @@ impl SystemdCollector {
|
||||
.arg("--plain")
|
||||
.output()?;
|
||||
|
||||
if !units_output.status.success() {
|
||||
return Err(anyhow::anyhow!("systemctl system command failed"));
|
||||
if !units_status_output.status.success() {
|
||||
return Err(anyhow::anyhow!("systemctl list-units command failed"));
|
||||
}
|
||||
|
||||
let units_str = String::from_utf8(units_output.stdout)?;
|
||||
let unit_files_str = String::from_utf8(unit_files_output.stdout)?;
|
||||
let units_status_str = String::from_utf8(units_status_output.stdout)?;
|
||||
let mut services = Vec::new();
|
||||
|
||||
// Use configuration instead of hardcoded values
|
||||
let excluded_services = &self.config.excluded_services;
|
||||
let service_name_filters = &self.config.service_name_filters;
|
||||
|
||||
// Parse all services and cache their status information
|
||||
// Parse all service unit files to get complete service list
|
||||
let mut all_service_names = std::collections::HashSet::new();
|
||||
let mut status_cache = std::collections::HashMap::new();
|
||||
|
||||
for line in units_str.lines() {
|
||||
for line in unit_files_str.lines() {
|
||||
let fields: Vec<&str> = line.split_whitespace().collect();
|
||||
if fields.len() >= 2 && fields[0].ends_with(".service") {
|
||||
let service_name = fields[0].trim_end_matches(".service");
|
||||
all_service_names.insert(service_name.to_string());
|
||||
debug!("Found service unit file: {}", service_name);
|
||||
}
|
||||
}
|
||||
|
||||
// Parse runtime status for all units
|
||||
let mut status_cache = std::collections::HashMap::new();
|
||||
for line in units_status_str.lines() {
|
||||
let fields: Vec<&str> = line.split_whitespace().collect();
|
||||
if fields.len() >= 4 && fields[0].ends_with(".service") {
|
||||
let service_name = fields[0].trim_end_matches(".service");
|
||||
@@ -177,8 +201,19 @@ impl SystemdCollector {
|
||||
sub_state: sub_state.clone(),
|
||||
});
|
||||
|
||||
all_service_names.insert(service_name.to_string());
|
||||
debug!("Parsed service: {} (load:{}, active:{}, sub:{})", service_name, load_state, active_state, sub_state);
|
||||
debug!("Got runtime status for service: {} (load:{}, active:{}, sub:{})", service_name, load_state, active_state, sub_state);
|
||||
}
|
||||
}
|
||||
|
||||
// For services found in unit files but not in runtime status, set default inactive status
|
||||
for service_name in &all_service_names {
|
||||
if !status_cache.contains_key(service_name) {
|
||||
status_cache.insert(service_name.to_string(), ServiceStatusInfo {
|
||||
load_state: "not-loaded".to_string(),
|
||||
active_state: "inactive".to_string(),
|
||||
sub_state: "dead".to_string(),
|
||||
});
|
||||
debug!("Service {} found in unit files but not runtime - marked as inactive", service_name);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -318,13 +353,19 @@ impl SystemdCollector {
|
||||
Ok((active_status, detailed_info))
|
||||
}
|
||||
|
||||
/// Calculate service status
|
||||
fn calculate_service_status(&self, active_status: &str) -> Status {
|
||||
/// Calculate service status, taking user-stopped services into account
|
||||
fn calculate_service_status(&self, service_name: &str, active_status: &str) -> Status {
|
||||
match active_status.to_lowercase().as_str() {
|
||||
"active" => Status::Ok,
|
||||
"inactive" | "dead" => Status::Warning,
|
||||
"inactive" | "dead" => {
|
||||
debug!("Service '{}' is inactive - treating as Inactive status", service_name);
|
||||
Status::Inactive
|
||||
},
|
||||
"failed" | "error" => Status::Critical,
|
||||
"activating" | "deactivating" | "reloading" | "start" | "stop" | "restart" => Status::Pending,
|
||||
"activating" | "deactivating" | "reloading" | "start" | "stop" | "restart" => {
|
||||
debug!("Service '{}' is transitioning - treating as Pending", service_name);
|
||||
Status::Pending
|
||||
},
|
||||
_ => Status::Unknown,
|
||||
}
|
||||
}
|
||||
@@ -445,7 +486,7 @@ impl Collector for SystemdCollector {
|
||||
for service in &monitored_services {
|
||||
match self.get_service_status(service) {
|
||||
Ok((active_status, _detailed_info)) => {
|
||||
let status = self.calculate_service_status(&active_status);
|
||||
let status = self.calculate_service_status(service, &active_status);
|
||||
|
||||
// Individual service status metric
|
||||
metrics.push(Metric {
|
||||
@@ -520,10 +561,8 @@ impl SystemdCollector {
|
||||
for (site_name, url) in &sites {
|
||||
match self.check_site_latency(url) {
|
||||
Ok(latency_ms) => {
|
||||
let status = if latency_ms < 500.0 {
|
||||
let status = if latency_ms < self.config.nginx_latency_critical_ms {
|
||||
Status::Ok
|
||||
} else if latency_ms < 2000.0 {
|
||||
Status::Warning
|
||||
} else {
|
||||
Status::Critical
|
||||
};
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
use anyhow::Result;
|
||||
use cm_dashboard_shared::{MessageEnvelope, MetricMessage};
|
||||
use cm_dashboard_shared::{AgentData, MessageEnvelope};
|
||||
use tracing::{debug, info};
|
||||
use zmq::{Context, Socket, SocketType};
|
||||
|
||||
@@ -43,17 +43,17 @@ impl ZmqHandler {
|
||||
})
|
||||
}
|
||||
|
||||
/// Publish metrics message via ZMQ
|
||||
pub async fn publish_metrics(&self, message: &MetricMessage) -> Result<()> {
|
||||
|
||||
/// Publish agent data via ZMQ
|
||||
pub async fn publish_agent_data(&self, data: &AgentData) -> Result<()> {
|
||||
debug!(
|
||||
"Publishing {} metrics for host {}",
|
||||
message.metrics.len(),
|
||||
message.hostname
|
||||
"Publishing agent data for host {}",
|
||||
data.hostname
|
||||
);
|
||||
|
||||
// Create message envelope
|
||||
let envelope = MessageEnvelope::metrics(message.clone())
|
||||
.map_err(|e| anyhow::anyhow!("Failed to create message envelope: {}", e))?;
|
||||
// Create message envelope for agent data
|
||||
let envelope = MessageEnvelope::agent_data(data.clone())
|
||||
.map_err(|e| anyhow::anyhow!("Failed to create agent data envelope: {}", e))?;
|
||||
|
||||
// Serialize envelope
|
||||
let serialized = serde_json::to_vec(&envelope)?;
|
||||
@@ -61,13 +61,10 @@ impl ZmqHandler {
|
||||
// Send via ZMQ
|
||||
self.publisher.send(&serialized, 0)?;
|
||||
|
||||
debug!("Published metrics message ({} bytes)", serialized.len());
|
||||
debug!("Published agent data message ({} bytes)", serialized.len());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
|
||||
/// Send heartbeat (placeholder for future use)
|
||||
|
||||
/// Try to receive a command (non-blocking)
|
||||
pub fn try_receive_command(&self) -> Result<Option<AgentCommand>> {
|
||||
match self.command_receiver.recv_bytes(zmq::DONTWAIT) {
|
||||
@@ -100,18 +97,4 @@ pub enum AgentCommand {
|
||||
ToggleCollector { name: String, enabled: bool },
|
||||
/// Request status/health check
|
||||
Ping,
|
||||
/// Control systemd service
|
||||
ServiceControl {
|
||||
service_name: String,
|
||||
action: ServiceAction,
|
||||
},
|
||||
}
|
||||
|
||||
/// Service control actions
|
||||
#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)]
|
||||
pub enum ServiceAction {
|
||||
Start,
|
||||
Stop,
|
||||
Restart,
|
||||
Status,
|
||||
}
|
||||
|
||||
@@ -25,9 +25,10 @@ pub struct ZmqConfig {
|
||||
pub publisher_port: u16,
|
||||
pub command_port: u16,
|
||||
pub bind_address: String,
|
||||
pub timeout_ms: u64,
|
||||
pub heartbeat_interval_ms: u64,
|
||||
pub transmission_interval_seconds: u64,
|
||||
/// Heartbeat transmission interval in seconds for host connectivity detection
|
||||
#[serde(default = "default_heartbeat_interval_seconds")]
|
||||
pub heartbeat_interval_seconds: u64,
|
||||
}
|
||||
|
||||
/// Collector configuration
|
||||
@@ -108,6 +109,7 @@ pub struct SystemdConfig {
|
||||
pub nginx_check_interval_seconds: u64,
|
||||
pub http_timeout_seconds: u64,
|
||||
pub http_connect_timeout_seconds: u64,
|
||||
pub nginx_latency_critical_ms: f32,
|
||||
}
|
||||
|
||||
|
||||
@@ -145,9 +147,23 @@ pub struct NotificationConfig {
|
||||
pub rate_limit_minutes: u64,
|
||||
/// Email notification batching interval in seconds (default: 60)
|
||||
pub aggregation_interval_seconds: u64,
|
||||
/// List of metric names to exclude from email notifications
|
||||
#[serde(default)]
|
||||
pub exclude_email_metrics: Vec<String>,
|
||||
/// Path to maintenance mode file that suppresses email notifications when present
|
||||
#[serde(default = "default_maintenance_mode_file")]
|
||||
pub maintenance_mode_file: String,
|
||||
}
|
||||
|
||||
|
||||
fn default_heartbeat_interval_seconds() -> u64 {
|
||||
5
|
||||
}
|
||||
|
||||
fn default_maintenance_mode_file() -> String {
|
||||
"/tmp/cm-maintenance".to_string()
|
||||
}
|
||||
|
||||
impl AgentConfig {
|
||||
pub fn from_file<P: AsRef<Path>>(path: P) -> Result<Self> {
|
||||
loader::load_config(path)
|
||||
|
||||
@@ -19,10 +19,6 @@ pub fn validate_config(config: &AgentConfig) -> Result<()> {
|
||||
bail!("ZMQ bind address cannot be empty");
|
||||
}
|
||||
|
||||
if config.zmq.timeout_ms == 0 {
|
||||
bail!("ZMQ timeout cannot be 0");
|
||||
}
|
||||
|
||||
// Validate collection interval
|
||||
if config.collection_interval_seconds == 0 {
|
||||
bail!("Collection interval cannot be 0");
|
||||
@@ -83,6 +79,13 @@ pub fn validate_config(config: &AgentConfig) -> Result<()> {
|
||||
}
|
||||
}
|
||||
|
||||
// Validate systemd configuration
|
||||
if config.collectors.systemd.enabled {
|
||||
if config.collectors.systemd.nginx_latency_critical_ms <= 0.0 {
|
||||
bail!("Nginx latency critical threshold must be positive");
|
||||
}
|
||||
}
|
||||
|
||||
// Validate SMTP configuration
|
||||
if config.notifications.enabled {
|
||||
if config.notifications.smtp_host.is_empty() {
|
||||
|
||||
@@ -9,6 +9,7 @@ mod communication;
|
||||
mod config;
|
||||
mod metrics;
|
||||
mod notifications;
|
||||
mod service_tracker;
|
||||
mod status;
|
||||
|
||||
use agent::Agent;
|
||||
|
||||
@@ -59,6 +59,6 @@ impl NotificationManager {
|
||||
}
|
||||
|
||||
fn is_maintenance_mode(&self) -> bool {
|
||||
std::fs::metadata("/tmp/cm-maintenance").is_ok()
|
||||
std::fs::metadata(&self.config.maintenance_mode_file).is_ok()
|
||||
}
|
||||
}
|
||||
164
agent/src/service_tracker.rs
Normal file
164
agent/src/service_tracker.rs
Normal file
@@ -0,0 +1,164 @@
|
||||
use anyhow::Result;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashSet;
|
||||
use std::fs;
|
||||
use std::path::Path;
|
||||
use std::sync::{Arc, Mutex, OnceLock};
|
||||
use tracing::{debug, info, warn};
|
||||
|
||||
/// Shared instance for global access
|
||||
static GLOBAL_TRACKER: OnceLock<Arc<Mutex<UserStoppedServiceTracker>>> = OnceLock::new();
|
||||
|
||||
/// Tracks services that have been stopped by user action
|
||||
/// These services should be treated as OK status instead of Warning
|
||||
#[derive(Debug)]
|
||||
pub struct UserStoppedServiceTracker {
|
||||
/// Set of services stopped by user action
|
||||
user_stopped_services: HashSet<String>,
|
||||
/// Path to persistent storage file
|
||||
storage_path: String,
|
||||
}
|
||||
|
||||
/// Serializable data structure for persistence
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
struct UserStoppedData {
|
||||
services: Vec<String>,
|
||||
}
|
||||
|
||||
impl UserStoppedServiceTracker {
|
||||
/// Create new tracker with default storage path
|
||||
pub fn new() -> Self {
|
||||
Self::with_storage_path("/var/lib/cm-dashboard/user-stopped-services.json")
|
||||
}
|
||||
|
||||
/// Initialize global instance (called by agent)
|
||||
pub fn init_global() -> Result<Self> {
|
||||
let tracker = Self::new();
|
||||
|
||||
// Set global instance
|
||||
let global_instance = Arc::new(Mutex::new(tracker));
|
||||
if GLOBAL_TRACKER.set(global_instance).is_err() {
|
||||
warn!("Global service tracker was already initialized");
|
||||
}
|
||||
|
||||
// Return a new instance for the agent to use
|
||||
Ok(Self::new())
|
||||
}
|
||||
|
||||
/// Check if a service is user-stopped (global access for collectors)
|
||||
pub fn is_service_user_stopped(service_name: &str) -> bool {
|
||||
if let Some(global) = GLOBAL_TRACKER.get() {
|
||||
if let Ok(tracker) = global.lock() {
|
||||
tracker.is_user_stopped(service_name)
|
||||
} else {
|
||||
debug!("Failed to lock global service tracker");
|
||||
false
|
||||
}
|
||||
} else {
|
||||
debug!("Global service tracker not initialized");
|
||||
false
|
||||
}
|
||||
}
|
||||
|
||||
/// Update global tracker (called by agent when tracker state changes)
|
||||
pub fn update_global(updated_tracker: &UserStoppedServiceTracker) {
|
||||
if let Some(global) = GLOBAL_TRACKER.get() {
|
||||
if let Ok(mut tracker) = global.lock() {
|
||||
tracker.user_stopped_services = updated_tracker.user_stopped_services.clone();
|
||||
} else {
|
||||
debug!("Failed to lock global service tracker for update");
|
||||
}
|
||||
} else {
|
||||
debug!("Global service tracker not initialized for update");
|
||||
}
|
||||
}
|
||||
|
||||
/// Create new tracker with custom storage path
|
||||
pub fn with_storage_path<P: AsRef<Path>>(storage_path: P) -> Self {
|
||||
let storage_path = storage_path.as_ref().to_string_lossy().to_string();
|
||||
let mut tracker = Self {
|
||||
user_stopped_services: HashSet::new(),
|
||||
storage_path,
|
||||
};
|
||||
|
||||
// Load existing data from storage
|
||||
if let Err(e) = tracker.load_from_storage() {
|
||||
warn!("Failed to load user-stopped services from storage: {}", e);
|
||||
info!("Starting with empty user-stopped services list");
|
||||
}
|
||||
|
||||
tracker
|
||||
}
|
||||
|
||||
|
||||
/// Clear user-stopped flag for a service (when user starts it)
|
||||
pub fn clear_user_stopped(&mut self, service_name: &str) -> Result<()> {
|
||||
if self.user_stopped_services.remove(service_name) {
|
||||
info!("Cleared user-stopped flag for service '{}'", service_name);
|
||||
self.save_to_storage()?;
|
||||
debug!("Service '{}' user-stopped flag cleared and saved to storage", service_name);
|
||||
} else {
|
||||
debug!("Service '{}' was not marked as user-stopped", service_name);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Check if a service is marked as user-stopped
|
||||
pub fn is_user_stopped(&self, service_name: &str) -> bool {
|
||||
let is_stopped = self.user_stopped_services.contains(service_name);
|
||||
debug!("Service '{}' user-stopped status: {}", service_name, is_stopped);
|
||||
is_stopped
|
||||
}
|
||||
|
||||
|
||||
/// Save current state to persistent storage
|
||||
fn save_to_storage(&self) -> Result<()> {
|
||||
// Create parent directory if it doesn't exist
|
||||
if let Some(parent_dir) = Path::new(&self.storage_path).parent() {
|
||||
if !parent_dir.exists() {
|
||||
fs::create_dir_all(parent_dir)?;
|
||||
debug!("Created parent directory: {}", parent_dir.display());
|
||||
}
|
||||
}
|
||||
|
||||
let data = UserStoppedData {
|
||||
services: self.user_stopped_services.iter().cloned().collect(),
|
||||
};
|
||||
|
||||
let json_data = serde_json::to_string_pretty(&data)?;
|
||||
fs::write(&self.storage_path, json_data)?;
|
||||
|
||||
debug!(
|
||||
"Saved {} user-stopped services to {}",
|
||||
data.services.len(),
|
||||
self.storage_path
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Load state from persistent storage
|
||||
fn load_from_storage(&mut self) -> Result<()> {
|
||||
if !Path::new(&self.storage_path).exists() {
|
||||
debug!("Storage file {} does not exist, starting fresh", self.storage_path);
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
let json_data = fs::read_to_string(&self.storage_path)?;
|
||||
let data: UserStoppedData = serde_json::from_str(&json_data)?;
|
||||
|
||||
self.user_stopped_services = data.services.into_iter().collect();
|
||||
|
||||
info!(
|
||||
"Loaded {} user-stopped services from {}",
|
||||
self.user_stopped_services.len(),
|
||||
self.storage_path
|
||||
);
|
||||
|
||||
if !self.user_stopped_services.is_empty() {
|
||||
debug!("User-stopped services: {:?}", self.user_stopped_services);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -272,11 +272,13 @@ impl HostStatusManager {
|
||||
/// Check if a status change is significant enough for notification
|
||||
fn is_significant_change(&self, old_status: Status, new_status: Status) -> bool {
|
||||
match (old_status, new_status) {
|
||||
// Always notify on problems
|
||||
// Don't notify on transitions from Unknown (startup/restart scenario)
|
||||
(Status::Unknown, _) => false,
|
||||
// Always notify on problems (but not from Unknown)
|
||||
(_, Status::Warning) | (_, Status::Critical) => true,
|
||||
// Only notify on recovery if it's from a problem state to OK and all services are OK
|
||||
(Status::Warning | Status::Critical, Status::Ok) => self.current_host_status == Status::Ok,
|
||||
// Don't notify on startup or other transitions
|
||||
// Don't notify on other transitions
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
@@ -374,8 +376,8 @@ impl HostStatusManager {
|
||||
details.push('\n');
|
||||
}
|
||||
|
||||
// Show recoveries
|
||||
if !recovery_changes.is_empty() {
|
||||
// Show recoveries only if host status is now OK (all services recovered)
|
||||
if !recovery_changes.is_empty() && aggregated.host_status_final == Status::Ok {
|
||||
details.push_str(&format!("✅ RECOVERIES ({}):\n", recovery_changes.len()));
|
||||
for change in recovery_changes {
|
||||
details.push_str(&format!(" {}\n", change));
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "cm-dashboard"
|
||||
version = "0.1.29"
|
||||
version = "0.1.131"
|
||||
edition = "2021"
|
||||
|
||||
[dependencies]
|
||||
@@ -19,3 +19,4 @@ ratatui = { workspace = true }
|
||||
crossterm = { workspace = true }
|
||||
toml = { workspace = true }
|
||||
gethostname = { workspace = true }
|
||||
wake-on-lan = "0.2"
|
||||
@@ -9,24 +9,24 @@ use std::io;
|
||||
use std::time::{Duration, Instant};
|
||||
use tracing::{debug, error, info, warn};
|
||||
|
||||
use crate::communication::{AgentCommand, ServiceAction, ZmqCommandSender, ZmqConsumer};
|
||||
use crate::communication::{ZmqConsumer};
|
||||
use crate::config::DashboardConfig;
|
||||
use crate::metrics::MetricStore;
|
||||
use crate::ui::{TuiApp, UiCommand};
|
||||
use crate::ui::TuiApp;
|
||||
|
||||
pub struct Dashboard {
|
||||
zmq_consumer: ZmqConsumer,
|
||||
zmq_command_sender: ZmqCommandSender,
|
||||
metric_store: MetricStore,
|
||||
tui_app: Option<TuiApp>,
|
||||
terminal: Option<Terminal<CrosstermBackend<io::Stdout>>>,
|
||||
headless: bool,
|
||||
raw_data: bool,
|
||||
initial_commands_sent: std::collections::HashSet<String>,
|
||||
_config: DashboardConfig,
|
||||
config: DashboardConfig,
|
||||
}
|
||||
|
||||
impl Dashboard {
|
||||
pub async fn new(config_path: Option<String>, headless: bool) -> Result<Self> {
|
||||
pub async fn new(config_path: Option<String>, headless: bool, raw_data: bool) -> Result<Self> {
|
||||
info!("Initializing dashboard");
|
||||
|
||||
// Load configuration - try default path if not specified
|
||||
@@ -58,20 +58,9 @@ impl Dashboard {
|
||||
}
|
||||
};
|
||||
|
||||
// Initialize ZMQ command sender
|
||||
let zmq_command_sender = match ZmqCommandSender::new(&config.zmq) {
|
||||
Ok(sender) => sender,
|
||||
Err(e) => {
|
||||
error!("Failed to initialize ZMQ command sender: {}", e);
|
||||
return Err(e);
|
||||
}
|
||||
};
|
||||
|
||||
// Connect to predefined hosts from configuration
|
||||
let hosts = config.hosts.predefined_hosts.clone();
|
||||
|
||||
// Try to connect to hosts but don't fail if none are available
|
||||
match zmq_consumer.connect_to_predefined_hosts(&hosts).await {
|
||||
match zmq_consumer.connect_to_predefined_hosts(&config.hosts).await {
|
||||
Ok(_) => info!("Successfully connected to ZMQ hosts"),
|
||||
Err(e) => {
|
||||
warn!(
|
||||
@@ -127,28 +116,24 @@ impl Dashboard {
|
||||
|
||||
Ok(Self {
|
||||
zmq_consumer,
|
||||
zmq_command_sender,
|
||||
metric_store,
|
||||
tui_app,
|
||||
terminal,
|
||||
headless,
|
||||
raw_data,
|
||||
initial_commands_sent: std::collections::HashSet::new(),
|
||||
_config: config,
|
||||
config,
|
||||
})
|
||||
}
|
||||
|
||||
/// Send a command to a specific agent
|
||||
pub async fn send_command(&mut self, hostname: &str, command: AgentCommand) -> Result<()> {
|
||||
self.zmq_command_sender
|
||||
.send_command(hostname, command)
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn run(&mut self) -> Result<()> {
|
||||
info!("Starting dashboard main loop");
|
||||
|
||||
let mut last_metrics_check = Instant::now();
|
||||
let metrics_check_interval = Duration::from_millis(100); // Check for metrics every 100ms
|
||||
let mut last_heartbeat_check = Instant::now();
|
||||
let heartbeat_check_interval = Duration::from_secs(1); // Check for host connectivity every 1 second
|
||||
|
||||
loop {
|
||||
// Handle terminal events (keyboard input) only if not headless
|
||||
@@ -158,16 +143,10 @@ impl Dashboard {
|
||||
match event::read() {
|
||||
Ok(event) => {
|
||||
if let Some(ref mut tui_app) = self.tui_app {
|
||||
// Handle input and check for commands
|
||||
// Handle input
|
||||
match tui_app.handle_input(event) {
|
||||
Ok(Some(command)) => {
|
||||
// Execute the command
|
||||
if let Err(e) = self.execute_ui_command(command).await {
|
||||
error!("Failed to execute UI command: {}", e);
|
||||
}
|
||||
}
|
||||
Ok(None) => {
|
||||
// No command, check if we should quit
|
||||
Ok(_) => {
|
||||
// Check if we should quit
|
||||
if tui_app.should_quit() {
|
||||
info!("Quit requested, exiting dashboard");
|
||||
break;
|
||||
@@ -191,50 +170,50 @@ impl Dashboard {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Render UI immediately after handling input for responsive feedback
|
||||
if let Some(ref mut terminal) = self.terminal {
|
||||
if let Some(ref mut tui_app) = self.tui_app {
|
||||
if let Err(e) = terminal.draw(|frame| {
|
||||
tui_app.render(frame, &self.metric_store);
|
||||
}) {
|
||||
error!("Error rendering TUI after input: {}", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Check for new metrics
|
||||
if last_metrics_check.elapsed() >= metrics_check_interval {
|
||||
if let Ok(Some(metric_message)) = self.zmq_consumer.receive_metrics().await {
|
||||
if let Ok(Some(agent_data)) = self.zmq_consumer.receive_agent_data().await {
|
||||
debug!(
|
||||
"Received metrics from {}: {} metrics",
|
||||
metric_message.hostname,
|
||||
metric_message.metrics.len()
|
||||
"Received agent data from {}",
|
||||
agent_data.hostname
|
||||
);
|
||||
|
||||
// Check if this is the first time we've seen this host
|
||||
// Track first contact with host (no command needed - agent sends data every 2s)
|
||||
let is_new_host = !self
|
||||
.initial_commands_sent
|
||||
.contains(&metric_message.hostname);
|
||||
.contains(&agent_data.hostname);
|
||||
|
||||
if is_new_host {
|
||||
info!(
|
||||
"First contact with host {}, sending initial CollectNow command",
|
||||
metric_message.hostname
|
||||
"First contact with host {} - data will update automatically",
|
||||
agent_data.hostname
|
||||
);
|
||||
|
||||
// Send CollectNow command for immediate refresh
|
||||
if let Err(e) = self
|
||||
.send_command(&metric_message.hostname, AgentCommand::CollectNow)
|
||||
.await
|
||||
{
|
||||
error!(
|
||||
"Failed to send initial CollectNow command to {}: {}",
|
||||
metric_message.hostname, e
|
||||
);
|
||||
} else {
|
||||
info!(
|
||||
"✓ Sent initial CollectNow command to {}",
|
||||
metric_message.hostname
|
||||
);
|
||||
self.initial_commands_sent
|
||||
.insert(metric_message.hostname.clone());
|
||||
}
|
||||
self.initial_commands_sent
|
||||
.insert(agent_data.hostname.clone());
|
||||
}
|
||||
|
||||
// Update metric store
|
||||
self.metric_store
|
||||
.update_metrics(&metric_message.hostname, metric_message.metrics);
|
||||
// Show raw data if requested (before processing)
|
||||
if self.raw_data {
|
||||
println!("RAW AGENT DATA FROM {}:", agent_data.hostname);
|
||||
println!("{}", serde_json::to_string_pretty(&agent_data).unwrap_or_else(|e| format!("Serialization error: {}", e)));
|
||||
println!("{}", "─".repeat(80));
|
||||
}
|
||||
|
||||
// Update data store
|
||||
self.metric_store.process_agent_data(agent_data);
|
||||
|
||||
// Check for agent version mismatches across hosts
|
||||
if let Some((current_version, outdated_hosts)) = self.metric_store.get_version_mismatches() {
|
||||
@@ -243,14 +222,8 @@ impl Dashboard {
|
||||
}
|
||||
}
|
||||
|
||||
// Update TUI with new hosts and metrics (only if not headless)
|
||||
// Update TUI with new metrics (only if not headless)
|
||||
if let Some(ref mut tui_app) = self.tui_app {
|
||||
let connected_hosts = self
|
||||
.metric_store
|
||||
.get_connected_hosts(Duration::from_secs(30));
|
||||
|
||||
|
||||
tui_app.update_hosts(connected_hosts);
|
||||
tui_app.update_metrics(&self.metric_store);
|
||||
}
|
||||
}
|
||||
@@ -269,6 +242,20 @@ impl Dashboard {
|
||||
last_metrics_check = Instant::now();
|
||||
}
|
||||
|
||||
// Check for host connectivity changes (heartbeat timeouts) periodically
|
||||
if last_heartbeat_check.elapsed() >= heartbeat_check_interval {
|
||||
let timeout = Duration::from_secs(self.config.zmq.heartbeat_timeout_seconds);
|
||||
|
||||
// Clean up metrics for offline hosts
|
||||
self.metric_store.cleanup_offline_hosts(timeout);
|
||||
|
||||
if let Some(ref mut tui_app) = self.tui_app {
|
||||
let connected_hosts = self.metric_store.get_connected_hosts(timeout);
|
||||
tui_app.update_hosts(connected_hosts);
|
||||
}
|
||||
last_heartbeat_check = Instant::now();
|
||||
}
|
||||
|
||||
// Render TUI (only if not headless)
|
||||
if !self.headless {
|
||||
if let Some(ref mut terminal) = self.terminal {
|
||||
@@ -291,41 +278,6 @@ impl Dashboard {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Execute a UI command by sending it to the appropriate agent
|
||||
async fn execute_ui_command(&self, command: UiCommand) -> Result<()> {
|
||||
match command {
|
||||
UiCommand::ServiceRestart { hostname, service_name } => {
|
||||
info!("Sending restart command for service {} on {}", service_name, hostname);
|
||||
let agent_command = AgentCommand::ServiceControl {
|
||||
service_name,
|
||||
action: ServiceAction::Restart,
|
||||
};
|
||||
self.zmq_command_sender.send_command(&hostname, agent_command).await?;
|
||||
}
|
||||
UiCommand::ServiceStart { hostname, service_name } => {
|
||||
info!("Sending start command for service {} on {}", service_name, hostname);
|
||||
let agent_command = AgentCommand::ServiceControl {
|
||||
service_name: service_name.clone(),
|
||||
action: ServiceAction::Start,
|
||||
};
|
||||
self.zmq_command_sender.send_command(&hostname, agent_command).await?;
|
||||
}
|
||||
UiCommand::ServiceStop { hostname, service_name } => {
|
||||
info!("Sending stop command for service {} on {}", service_name, hostname);
|
||||
let agent_command = AgentCommand::ServiceControl {
|
||||
service_name: service_name.clone(),
|
||||
action: ServiceAction::Stop,
|
||||
};
|
||||
self.zmq_command_sender.send_command(&hostname, agent_command).await?;
|
||||
}
|
||||
UiCommand::TriggerBackup { hostname } => {
|
||||
info!("Trigger backup requested for {}", hostname);
|
||||
// TODO: Implement backup trigger command
|
||||
info!("Backup trigger not yet implemented");
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
|
||||
@@ -1,43 +1,10 @@
|
||||
use anyhow::Result;
|
||||
use cm_dashboard_shared::{CommandOutputMessage, MessageEnvelope, MessageType, MetricMessage};
|
||||
use cm_dashboard_shared::{AgentData, CommandOutputMessage, MessageEnvelope, MessageType};
|
||||
use tracing::{debug, error, info, warn};
|
||||
use zmq::{Context, Socket, SocketType};
|
||||
|
||||
use crate::config::ZmqConfig;
|
||||
|
||||
/// Commands that can be sent to agents
|
||||
#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)]
|
||||
pub enum AgentCommand {
|
||||
/// Request immediate metric collection
|
||||
CollectNow,
|
||||
/// Change collection interval
|
||||
SetInterval { seconds: u64 },
|
||||
/// Enable/disable a collector
|
||||
ToggleCollector { name: String, enabled: bool },
|
||||
/// Request status/health check
|
||||
Ping,
|
||||
/// Control systemd service
|
||||
ServiceControl {
|
||||
service_name: String,
|
||||
action: ServiceAction,
|
||||
},
|
||||
/// Rebuild NixOS system
|
||||
SystemRebuild {
|
||||
git_url: String,
|
||||
git_branch: String,
|
||||
working_dir: String,
|
||||
api_key_file: Option<String>,
|
||||
},
|
||||
}
|
||||
|
||||
/// Service control actions
|
||||
#[derive(Debug, Clone, serde::Deserialize, serde::Serialize)]
|
||||
pub enum ServiceAction {
|
||||
Start,
|
||||
Stop,
|
||||
Restart,
|
||||
Status,
|
||||
}
|
||||
|
||||
/// ZMQ consumer for receiving metrics from agents
|
||||
pub struct ZmqConsumer {
|
||||
@@ -83,13 +50,14 @@ impl ZmqConsumer {
|
||||
}
|
||||
}
|
||||
|
||||
/// Connect to predefined hosts
|
||||
pub async fn connect_to_predefined_hosts(&mut self, hosts: &[String]) -> Result<()> {
|
||||
|
||||
/// Connect to predefined hosts using their configuration
|
||||
pub async fn connect_to_predefined_hosts(&mut self, hosts: &std::collections::HashMap<String, crate::config::HostDetails>) -> Result<()> {
|
||||
let default_port = self.config.subscriber_ports[0];
|
||||
|
||||
for hostname in hosts {
|
||||
// Try to connect, but don't fail if some hosts are unreachable
|
||||
if let Err(e) = self.connect_to_host(hostname, default_port).await {
|
||||
for (hostname, host_details) in hosts {
|
||||
// Try to connect using configured IP, but don't fail if some hosts are unreachable
|
||||
if let Err(e) = self.connect_to_host_with_details(hostname, host_details, default_port).await {
|
||||
warn!("Could not connect to {}: {}", hostname, e);
|
||||
}
|
||||
}
|
||||
@@ -103,6 +71,15 @@ impl ZmqConsumer {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Connect to a host using its configuration details
|
||||
pub async fn connect_to_host_with_details(&mut self, hostname: &str, host_details: &crate::config::HostDetails, port: u16) -> Result<()> {
|
||||
// Get primary connection IP only - no fallbacks
|
||||
let primary_ip = host_details.get_connection_ip(hostname);
|
||||
|
||||
// Connect directly without fallback attempts
|
||||
self.connect_to_host(&primary_ip, port).await
|
||||
}
|
||||
|
||||
/// Receive command output from any connected agent (non-blocking)
|
||||
pub async fn receive_command_output(&mut self) -> Result<Option<CommandOutputMessage>> {
|
||||
match self.subscriber.recv_bytes(zmq::DONTWAIT) {
|
||||
@@ -140,8 +117,8 @@ impl ZmqConsumer {
|
||||
}
|
||||
}
|
||||
|
||||
/// Receive metrics from any connected agent (non-blocking)
|
||||
pub async fn receive_metrics(&mut self) -> Result<Option<MetricMessage>> {
|
||||
/// Receive agent data (non-blocking)
|
||||
pub async fn receive_agent_data(&mut self) -> Result<Option<AgentData>> {
|
||||
match self.subscriber.recv_bytes(zmq::DONTWAIT) {
|
||||
Ok(data) => {
|
||||
debug!("Received {} bytes from ZMQ", data.len());
|
||||
@@ -152,29 +129,27 @@ impl ZmqConsumer {
|
||||
|
||||
// Check message type
|
||||
match envelope.message_type {
|
||||
MessageType::Metrics => {
|
||||
let metrics = envelope
|
||||
.decode_metrics()
|
||||
.map_err(|e| anyhow::anyhow!("Failed to decode metrics: {}", e))?;
|
||||
MessageType::AgentData => {
|
||||
let agent_data = envelope
|
||||
.decode_agent_data()
|
||||
.map_err(|e| anyhow::anyhow!("Failed to decode agent data: {}", e))?;
|
||||
|
||||
debug!(
|
||||
"Received {} metrics from {}",
|
||||
metrics.metrics.len(),
|
||||
metrics.hostname
|
||||
"Received agent data from host {}",
|
||||
agent_data.hostname
|
||||
);
|
||||
|
||||
Ok(Some(metrics))
|
||||
Ok(Some(agent_data))
|
||||
}
|
||||
MessageType::Heartbeat => {
|
||||
debug!("Received heartbeat");
|
||||
Ok(None) // Don't return heartbeats as metrics
|
||||
Ok(None) // Don't return heartbeats
|
||||
}
|
||||
MessageType::CommandOutput => {
|
||||
debug!("Received command output (will be handled by receive_command_output)");
|
||||
Ok(None) // Command output handled by separate method
|
||||
}
|
||||
_ => {
|
||||
debug!("Received non-metrics message: {:?}", envelope.message_type);
|
||||
debug!("Received unsupported message: {:?}", envelope.message_type);
|
||||
Ok(None)
|
||||
}
|
||||
}
|
||||
@@ -189,44 +164,6 @@ impl ZmqConsumer {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/// ZMQ command sender for sending commands to agents
|
||||
pub struct ZmqCommandSender {
|
||||
context: Context,
|
||||
}
|
||||
|
||||
impl ZmqCommandSender {
|
||||
pub fn new(_config: &ZmqConfig) -> Result<Self> {
|
||||
let context = Context::new();
|
||||
|
||||
info!("ZMQ command sender initialized");
|
||||
|
||||
Ok(Self { context })
|
||||
}
|
||||
|
||||
/// Send a command to a specific agent
|
||||
pub async fn send_command(&self, hostname: &str, command: AgentCommand) -> Result<()> {
|
||||
// Create a new PUSH socket for this command (ZMQ best practice)
|
||||
let socket = self.context.socket(SocketType::PUSH)?;
|
||||
|
||||
// Set socket options
|
||||
socket.set_linger(1000)?; // Wait up to 1 second on close
|
||||
socket.set_sndtimeo(5000)?; // 5 second send timeout
|
||||
|
||||
// Connect to agent's command port (6131)
|
||||
let address = format!("tcp://{}:6131", hostname);
|
||||
socket.connect(&address)?;
|
||||
|
||||
// Serialize command
|
||||
let serialized = serde_json::to_vec(&command)?;
|
||||
|
||||
// Send command
|
||||
socket.send(&serialized, 0)?;
|
||||
|
||||
info!("Sent command {:?} to agent at {}", command, hostname);
|
||||
|
||||
// Socket will be automatically closed when dropped
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,21 +6,40 @@ use std::path::Path;
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct DashboardConfig {
|
||||
pub zmq: ZmqConfig,
|
||||
pub hosts: HostsConfig,
|
||||
pub hosts: std::collections::HashMap<String, HostDetails>,
|
||||
pub system: SystemConfig,
|
||||
pub ssh: SshConfig,
|
||||
pub service_logs: std::collections::HashMap<String, Vec<ServiceLogConfig>>,
|
||||
}
|
||||
|
||||
/// ZMQ consumer configuration
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ZmqConfig {
|
||||
pub subscriber_ports: Vec<u16>,
|
||||
/// Heartbeat timeout in seconds - hosts considered offline if no heartbeat received within this time
|
||||
#[serde(default = "default_heartbeat_timeout_seconds")]
|
||||
pub heartbeat_timeout_seconds: u64,
|
||||
}
|
||||
|
||||
/// Hosts configuration
|
||||
fn default_heartbeat_timeout_seconds() -> u64 {
|
||||
10 // Default to 10 seconds - allows for multiple missed heartbeats
|
||||
}
|
||||
|
||||
/// Individual host configuration details
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct HostsConfig {
|
||||
pub predefined_hosts: Vec<String>,
|
||||
pub struct HostDetails {
|
||||
pub mac_address: Option<String>,
|
||||
/// Primary IP address (local network)
|
||||
pub ip: Option<String>,
|
||||
}
|
||||
|
||||
|
||||
impl HostDetails {
|
||||
/// Get the IP address for connection (uses ip field or hostname as fallback)
|
||||
pub fn get_connection_ip(&self, hostname: &str) -> String {
|
||||
self.ip.as_ref().unwrap_or(&hostname.to_string()).clone()
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/// System configuration
|
||||
@@ -32,11 +51,19 @@ pub struct SystemConfig {
|
||||
pub nixos_config_api_key_file: Option<String>,
|
||||
}
|
||||
|
||||
/// SSH configuration for rebuild operations
|
||||
/// SSH configuration for rebuild and backup operations
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct SshConfig {
|
||||
pub rebuild_user: String,
|
||||
pub rebuild_alias: String,
|
||||
pub rebuild_cmd: String,
|
||||
pub service_manage_cmd: String,
|
||||
}
|
||||
|
||||
/// Service log file configuration per host
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ServiceLogConfig {
|
||||
pub service_name: String,
|
||||
pub log_file_path: String,
|
||||
}
|
||||
|
||||
impl DashboardConfig {
|
||||
@@ -60,8 +87,3 @@ impl Default for ZmqConfig {
|
||||
}
|
||||
}
|
||||
|
||||
impl Default for HostsConfig {
|
||||
fn default() -> Self {
|
||||
panic!("Dashboard configuration must be loaded from file - no hardcoded defaults allowed")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,10 +12,6 @@ mod ui;
|
||||
|
||||
use app::Dashboard;
|
||||
|
||||
/// Get hardcoded version
|
||||
fn get_version() -> &'static str {
|
||||
"v0.1.29"
|
||||
}
|
||||
|
||||
/// Check if running inside tmux session
|
||||
fn check_tmux_session() {
|
||||
@@ -42,7 +38,7 @@ fn check_tmux_session() {
|
||||
#[derive(Parser)]
|
||||
#[command(name = "cm-dashboard")]
|
||||
#[command(about = "CM Dashboard TUI with individual metric consumption")]
|
||||
#[command(version = get_version())]
|
||||
#[command(version)]
|
||||
struct Cli {
|
||||
/// Increase logging verbosity (-v, -vv)
|
||||
#[arg(short, long, action = clap::ArgAction::Count)]
|
||||
@@ -55,6 +51,10 @@ struct Cli {
|
||||
/// Run in headless mode (no TUI, just logging)
|
||||
#[arg(long)]
|
||||
headless: bool,
|
||||
|
||||
/// Show raw agent data in headless mode
|
||||
#[arg(long)]
|
||||
raw_data: bool,
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
@@ -90,7 +90,7 @@ async fn main() -> Result<()> {
|
||||
}
|
||||
|
||||
// Create and run dashboard
|
||||
let mut dashboard = Dashboard::new(cli.config, cli.headless).await?;
|
||||
let mut dashboard = Dashboard::new(cli.config, cli.headless, cli.raw_data).await?;
|
||||
|
||||
// Setup graceful shutdown
|
||||
let ctrl_c = async {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use cm_dashboard_shared::Metric;
|
||||
use cm_dashboard_shared::{AgentData, Metric};
|
||||
use std::collections::HashMap;
|
||||
use std::time::{Duration, Instant};
|
||||
use tracing::{debug, info, warn};
|
||||
@@ -11,8 +11,8 @@ pub struct MetricStore {
|
||||
current_metrics: HashMap<String, HashMap<String, Metric>>,
|
||||
/// Historical metrics for trending
|
||||
historical_metrics: HashMap<String, Vec<MetricDataPoint>>,
|
||||
/// Last update timestamp per host
|
||||
last_update: HashMap<String, Instant>,
|
||||
/// Last heartbeat timestamp per host
|
||||
last_heartbeat: HashMap<String, Instant>,
|
||||
/// Configuration
|
||||
max_metrics_per_host: usize,
|
||||
history_retention: Duration,
|
||||
@@ -23,7 +23,7 @@ impl MetricStore {
|
||||
Self {
|
||||
current_metrics: HashMap::new(),
|
||||
historical_metrics: HashMap::new(),
|
||||
last_update: HashMap::new(),
|
||||
last_heartbeat: HashMap::new(),
|
||||
max_metrics_per_host,
|
||||
history_retention: Duration::from_secs(history_retention_hours * 3600),
|
||||
}
|
||||
@@ -56,10 +56,13 @@ impl MetricStore {
|
||||
|
||||
// Add to history
|
||||
host_history.push(MetricDataPoint { received_at: now });
|
||||
}
|
||||
|
||||
// Update last update timestamp
|
||||
self.last_update.insert(hostname.to_string(), now);
|
||||
// Track heartbeat metrics for connectivity detection
|
||||
if metric_name == "agent_heartbeat" {
|
||||
self.last_heartbeat.insert(hostname.to_string(), now);
|
||||
debug!("Updated heartbeat for host {}", hostname);
|
||||
}
|
||||
}
|
||||
|
||||
// Get metrics count before cleanup
|
||||
let metrics_count = host_metrics.len();
|
||||
@@ -73,6 +76,286 @@ impl MetricStore {
|
||||
);
|
||||
}
|
||||
|
||||
/// Process structured agent data (temporary bridge - converts back to metrics)
|
||||
/// TODO: Replace entire metric system with direct structured data processing
|
||||
pub fn process_agent_data(&mut self, agent_data: AgentData) {
|
||||
let metrics = self.convert_agent_data_to_metrics(&agent_data);
|
||||
self.update_metrics(&agent_data.hostname, metrics);
|
||||
}
|
||||
|
||||
/// Convert structured agent data to legacy metrics (temporary bridge)
|
||||
fn convert_agent_data_to_metrics(&self, agent_data: &AgentData) -> Vec<Metric> {
|
||||
use cm_dashboard_shared::{Metric, MetricValue, Status};
|
||||
|
||||
let mut metrics = Vec::new();
|
||||
|
||||
// Convert CPU data
|
||||
metrics.push(Metric::new(
|
||||
"cpu_load_1min".to_string(),
|
||||
MetricValue::Float(agent_data.system.cpu.load_1min),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
"cpu_load_5min".to_string(),
|
||||
MetricValue::Float(agent_data.system.cpu.load_5min),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
"cpu_load_15min".to_string(),
|
||||
MetricValue::Float(agent_data.system.cpu.load_15min),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
"cpu_frequency_mhz".to_string(),
|
||||
MetricValue::Float(agent_data.system.cpu.frequency_mhz),
|
||||
Status::Ok,
|
||||
));
|
||||
if let Some(temp) = agent_data.system.cpu.temperature_celsius {
|
||||
metrics.push(Metric::new(
|
||||
"cpu_temperature_celsius".to_string(),
|
||||
MetricValue::Float(temp),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
|
||||
// Convert Memory data
|
||||
metrics.push(Metric::new(
|
||||
"memory_usage_percent".to_string(),
|
||||
MetricValue::Float(agent_data.system.memory.usage_percent),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
"memory_total_gb".to_string(),
|
||||
MetricValue::Float(agent_data.system.memory.total_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
"memory_used_gb".to_string(),
|
||||
MetricValue::Float(agent_data.system.memory.used_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
"memory_available_gb".to_string(),
|
||||
MetricValue::Float(agent_data.system.memory.available_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
"memory_swap_total_gb".to_string(),
|
||||
MetricValue::Float(agent_data.system.memory.swap_total_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
"memory_swap_used_gb".to_string(),
|
||||
MetricValue::Float(agent_data.system.memory.swap_used_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
|
||||
// Convert tmpfs data
|
||||
for tmpfs in &agent_data.system.memory.tmpfs {
|
||||
if tmpfs.mount == "/tmp" {
|
||||
metrics.push(Metric::new(
|
||||
"memory_tmp_usage_percent".to_string(),
|
||||
MetricValue::Float(tmpfs.usage_percent),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
"memory_tmp_used_gb".to_string(),
|
||||
MetricValue::Float(tmpfs.used_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
"memory_tmp_total_gb".to_string(),
|
||||
MetricValue::Float(tmpfs.total_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
// Add agent metadata
|
||||
metrics.push(Metric::new(
|
||||
"agent_version".to_string(),
|
||||
MetricValue::String(agent_data.agent_version.clone()),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
"agent_heartbeat".to_string(),
|
||||
MetricValue::Integer(agent_data.timestamp as i64),
|
||||
Status::Ok,
|
||||
));
|
||||
|
||||
// Convert storage data
|
||||
for drive in &agent_data.system.storage.drives {
|
||||
// Drive-level metrics
|
||||
if let Some(temp) = drive.temperature_celsius {
|
||||
metrics.push(Metric::new(
|
||||
format!("disk_{}_temperature", drive.name),
|
||||
MetricValue::Float(temp),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
if let Some(wear) = drive.wear_percent {
|
||||
metrics.push(Metric::new(
|
||||
format!("disk_{}_wear_percent", drive.name),
|
||||
MetricValue::Float(wear),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
metrics.push(Metric::new(
|
||||
format!("disk_{}_health", drive.name),
|
||||
MetricValue::String(drive.health.clone()),
|
||||
Status::Ok,
|
||||
));
|
||||
|
||||
// Filesystem metrics
|
||||
for fs in &drive.filesystems {
|
||||
let fs_base = format!("disk_{}_fs_{}", drive.name, fs.mount.replace('/', "root"));
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_usage_percent", fs_base),
|
||||
MetricValue::Float(fs.usage_percent),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_used_gb", fs_base),
|
||||
MetricValue::Float(fs.used_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_total_gb", fs_base),
|
||||
MetricValue::Float(fs.total_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
// Convert storage pools
|
||||
for pool in &agent_data.system.storage.pools {
|
||||
let pool_base = format!("disk_{}", pool.name);
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_usage_percent", pool_base),
|
||||
MetricValue::Float(pool.usage_percent),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_used_gb", pool_base),
|
||||
MetricValue::Float(pool.used_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_total_gb", pool_base),
|
||||
MetricValue::Float(pool.total_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_pool_type", pool_base),
|
||||
MetricValue::String(pool.pool_type.clone()),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_mount_point", pool_base),
|
||||
MetricValue::String(pool.mount.clone()),
|
||||
Status::Ok,
|
||||
));
|
||||
|
||||
// Pool drive data
|
||||
for drive in &pool.data_drives {
|
||||
if let Some(temp) = drive.temperature_celsius {
|
||||
metrics.push(Metric::new(
|
||||
format!("disk_{}_{}_temperature", pool.name, drive.name),
|
||||
MetricValue::Float(temp),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
if let Some(wear) = drive.wear_percent {
|
||||
metrics.push(Metric::new(
|
||||
format!("disk_{}_{}_wear_percent", pool.name, drive.name),
|
||||
MetricValue::Float(wear),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
}
|
||||
for drive in &pool.parity_drives {
|
||||
if let Some(temp) = drive.temperature_celsius {
|
||||
metrics.push(Metric::new(
|
||||
format!("disk_{}_{}_temperature", pool.name, drive.name),
|
||||
MetricValue::Float(temp),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
if let Some(wear) = drive.wear_percent {
|
||||
metrics.push(Metric::new(
|
||||
format!("disk_{}_{}_wear_percent", pool.name, drive.name),
|
||||
MetricValue::Float(wear),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Convert service data
|
||||
for service in &agent_data.services {
|
||||
let service_base = format!("service_{}", service.name);
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_status", service_base),
|
||||
MetricValue::String(service.status.clone()),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_memory_mb", service_base),
|
||||
MetricValue::Float(service.memory_mb),
|
||||
Status::Ok,
|
||||
));
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_disk_gb", service_base),
|
||||
MetricValue::Float(service.disk_gb),
|
||||
Status::Ok,
|
||||
));
|
||||
if service.user_stopped {
|
||||
metrics.push(Metric::new(
|
||||
format!("{}_user_stopped", service_base),
|
||||
MetricValue::Boolean(true),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
// Convert backup data
|
||||
metrics.push(Metric::new(
|
||||
"backup_status".to_string(),
|
||||
MetricValue::String(agent_data.backup.status.clone()),
|
||||
Status::Ok,
|
||||
));
|
||||
if let Some(last_run) = agent_data.backup.last_run {
|
||||
metrics.push(Metric::new(
|
||||
"backup_last_run_timestamp".to_string(),
|
||||
MetricValue::Integer(last_run as i64),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
if let Some(next_scheduled) = agent_data.backup.next_scheduled {
|
||||
metrics.push(Metric::new(
|
||||
"backup_next_scheduled_timestamp".to_string(),
|
||||
MetricValue::Integer(next_scheduled as i64),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
if let Some(size) = agent_data.backup.total_size_gb {
|
||||
metrics.push(Metric::new(
|
||||
"backup_size_gb".to_string(),
|
||||
MetricValue::Float(size),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
if let Some(health) = &agent_data.backup.repository_health {
|
||||
metrics.push(Metric::new(
|
||||
"backup_repository_health".to_string(),
|
||||
MetricValue::String(health.clone()),
|
||||
Status::Ok,
|
||||
));
|
||||
}
|
||||
|
||||
metrics
|
||||
}
|
||||
|
||||
/// Get current metric for a specific host
|
||||
pub fn get_metric(&self, hostname: &str, metric_name: &str) -> Option<&Metric> {
|
||||
self.current_metrics.get(hostname)?.get(metric_name)
|
||||
@@ -88,22 +371,46 @@ impl MetricStore {
|
||||
}
|
||||
}
|
||||
|
||||
/// Get connected hosts (hosts with recent updates)
|
||||
/// Get connected hosts (hosts with recent heartbeats)
|
||||
pub fn get_connected_hosts(&self, timeout: Duration) -> Vec<String> {
|
||||
let now = Instant::now();
|
||||
|
||||
self.last_update
|
||||
self.last_heartbeat
|
||||
.iter()
|
||||
.filter_map(|(hostname, &last_update)| {
|
||||
if now.duration_since(last_update) <= timeout {
|
||||
.filter_map(|(hostname, &last_heartbeat)| {
|
||||
if now.duration_since(last_heartbeat) <= timeout {
|
||||
Some(hostname.clone())
|
||||
} else {
|
||||
debug!("Host {} considered offline - last heartbeat was {:?} ago",
|
||||
hostname, now.duration_since(last_heartbeat));
|
||||
None
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Clean up data for offline hosts
|
||||
pub fn cleanup_offline_hosts(&mut self, timeout: Duration) {
|
||||
let now = Instant::now();
|
||||
let mut hosts_to_cleanup = Vec::new();
|
||||
|
||||
// Find hosts that are offline (no recent heartbeat)
|
||||
for (hostname, &last_heartbeat) in &self.last_heartbeat {
|
||||
if now.duration_since(last_heartbeat) > timeout {
|
||||
hosts_to_cleanup.push(hostname.clone());
|
||||
}
|
||||
}
|
||||
|
||||
// Clear metrics for offline hosts
|
||||
for hostname in hosts_to_cleanup {
|
||||
if let Some(metrics) = self.current_metrics.remove(&hostname) {
|
||||
info!("Cleared {} metrics for offline host: {}", metrics.len(), hostname);
|
||||
}
|
||||
// Keep heartbeat timestamp for reconnection detection
|
||||
// Don't remove from last_heartbeat to track when host was last seen
|
||||
}
|
||||
}
|
||||
|
||||
/// Cleanup old data and enforce limits
|
||||
fn cleanup_host_data(&mut self, hostname: &str) {
|
||||
let now = Instant::now();
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
use anyhow::Result;
|
||||
use crossterm::event::{Event, KeyCode, KeyModifiers};
|
||||
use crossterm::event::{Event, KeyCode};
|
||||
use ratatui::{
|
||||
layout::{Constraint, Direction, Layout, Rect},
|
||||
style::Style,
|
||||
@@ -9,45 +9,21 @@ use ratatui::{
|
||||
use std::collections::HashMap;
|
||||
use std::time::Instant;
|
||||
use tracing::info;
|
||||
use wake_on_lan::MagicPacket;
|
||||
|
||||
pub mod theme;
|
||||
pub mod widgets;
|
||||
|
||||
use crate::config::DashboardConfig;
|
||||
use crate::metrics::MetricStore;
|
||||
use cm_dashboard_shared::{Metric, Status};
|
||||
use cm_dashboard_shared::Status;
|
||||
use theme::{Components, Layout as ThemeLayout, Theme, Typography};
|
||||
use widgets::{BackupWidget, ServicesWidget, SystemWidget, Widget};
|
||||
|
||||
/// Commands that can be triggered from the UI
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum UiCommand {
|
||||
ServiceRestart { hostname: String, service_name: String },
|
||||
ServiceStart { hostname: String, service_name: String },
|
||||
ServiceStop { hostname: String, service_name: String },
|
||||
TriggerBackup { hostname: String },
|
||||
}
|
||||
|
||||
|
||||
/// Types of commands for status tracking
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum CommandType {
|
||||
ServiceRestart,
|
||||
ServiceStart,
|
||||
ServiceStop,
|
||||
BackupTrigger,
|
||||
}
|
||||
|
||||
/// Panel types for focus management
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum PanelType {
|
||||
System,
|
||||
Services,
|
||||
Backup,
|
||||
}
|
||||
|
||||
impl PanelType {
|
||||
}
|
||||
|
||||
/// Widget states for a specific host
|
||||
#[derive(Clone)]
|
||||
@@ -58,14 +34,8 @@ pub struct HostWidgets {
|
||||
pub services_widget: ServicesWidget,
|
||||
/// Backup widget state
|
||||
pub backup_widget: BackupWidget,
|
||||
/// Scroll offsets for each panel
|
||||
pub system_scroll_offset: usize,
|
||||
pub services_scroll_offset: usize,
|
||||
pub backup_scroll_offset: usize,
|
||||
/// Last update time for this host
|
||||
pub last_update: Option<Instant>,
|
||||
/// Pending service transitions for immediate visual feedback
|
||||
pub pending_service_transitions: HashMap<String, (CommandType, String, Instant)>, // service_name -> (command_type, original_status, start_time)
|
||||
}
|
||||
|
||||
impl HostWidgets {
|
||||
@@ -74,11 +44,7 @@ impl HostWidgets {
|
||||
system_widget: SystemWidget::new(),
|
||||
services_widget: ServicesWidget::new(),
|
||||
backup_widget: BackupWidget::new(),
|
||||
system_scroll_offset: 0,
|
||||
services_scroll_offset: 0,
|
||||
backup_scroll_offset: 0,
|
||||
last_update: None,
|
||||
pending_service_transitions: HashMap::new(),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -94,28 +60,39 @@ pub struct TuiApp {
|
||||
available_hosts: Vec<String>,
|
||||
/// Host index for navigation
|
||||
host_index: usize,
|
||||
/// Currently focused panel
|
||||
focused_panel: PanelType,
|
||||
/// Should quit application
|
||||
should_quit: bool,
|
||||
/// Track if user manually navigated away from localhost
|
||||
user_navigated_away: bool,
|
||||
/// Dashboard configuration
|
||||
config: DashboardConfig,
|
||||
/// Cached localhost hostname to avoid repeated system calls
|
||||
localhost: String,
|
||||
}
|
||||
|
||||
impl TuiApp {
|
||||
pub fn new(config: DashboardConfig) -> Self {
|
||||
Self {
|
||||
let localhost = gethostname::gethostname().to_string_lossy().to_string();
|
||||
let mut app = Self {
|
||||
host_widgets: HashMap::new(),
|
||||
current_host: None,
|
||||
available_hosts: Vec::new(),
|
||||
available_hosts: config.hosts.keys().cloned().collect(),
|
||||
host_index: 0,
|
||||
focused_panel: PanelType::System, // Start with System panel focused
|
||||
should_quit: false,
|
||||
user_navigated_away: false,
|
||||
config,
|
||||
localhost,
|
||||
};
|
||||
|
||||
// Sort predefined hosts
|
||||
app.available_hosts.sort();
|
||||
|
||||
// Initialize with first host if available
|
||||
if !app.available_hosts.is_empty() {
|
||||
app.current_host = Some(app.available_hosts[0].clone());
|
||||
}
|
||||
|
||||
app
|
||||
}
|
||||
|
||||
/// Get or create host widgets for the given hostname
|
||||
@@ -134,34 +111,32 @@ impl TuiApp {
|
||||
// Only update widgets if we have metrics for this host
|
||||
let all_metrics = metric_store.get_metrics_for_host(&hostname);
|
||||
if !all_metrics.is_empty() {
|
||||
// Get metrics first while hostname is borrowed
|
||||
let cpu_metrics: Vec<&Metric> = all_metrics
|
||||
.iter()
|
||||
.filter(|m| {
|
||||
m.name.starts_with("cpu_")
|
||||
|| m.name.contains("c_state_")
|
||||
|| m.name.starts_with("process_top_")
|
||||
})
|
||||
.copied()
|
||||
.collect();
|
||||
let memory_metrics: Vec<&Metric> = all_metrics
|
||||
.iter()
|
||||
.filter(|m| m.name.starts_with("memory_") || m.name.starts_with("disk_tmp_"))
|
||||
.copied()
|
||||
.collect();
|
||||
let service_metrics: Vec<&Metric> = all_metrics
|
||||
.iter()
|
||||
.filter(|m| m.name.starts_with("service_"))
|
||||
.copied()
|
||||
.collect();
|
||||
let all_backup_metrics: Vec<&Metric> = all_metrics
|
||||
.iter()
|
||||
.filter(|m| m.name.starts_with("backup_"))
|
||||
.copied()
|
||||
.collect();
|
||||
// Single pass metric categorization for better performance
|
||||
let mut cpu_metrics = Vec::new();
|
||||
let mut memory_metrics = Vec::new();
|
||||
let mut service_metrics = Vec::new();
|
||||
let mut backup_metrics = Vec::new();
|
||||
let mut nixos_metrics = Vec::new();
|
||||
let mut disk_metrics = Vec::new();
|
||||
|
||||
for metric in all_metrics {
|
||||
if metric.name.starts_with("cpu_")
|
||||
|| metric.name.contains("c_state_")
|
||||
|| metric.name.starts_with("process_top_") {
|
||||
cpu_metrics.push(metric);
|
||||
} else if metric.name.starts_with("memory_") || metric.name.starts_with("disk_tmp_") {
|
||||
memory_metrics.push(metric);
|
||||
} else if metric.name.starts_with("service_") {
|
||||
service_metrics.push(metric);
|
||||
} else if metric.name.starts_with("backup_") {
|
||||
backup_metrics.push(metric);
|
||||
} else if metric.name == "system_nixos_build" || metric.name == "system_active_users" || metric.name == "agent_version" {
|
||||
nixos_metrics.push(metric);
|
||||
} else if metric.name.starts_with("disk_") {
|
||||
disk_metrics.push(metric);
|
||||
}
|
||||
}
|
||||
|
||||
// Clear completed transitions first
|
||||
self.clear_completed_transitions(&hostname, &service_metrics);
|
||||
|
||||
// Now get host widgets and update them
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
@@ -169,21 +144,7 @@ impl TuiApp {
|
||||
// Collect all system metrics (CPU, memory, NixOS, disk/storage)
|
||||
let mut system_metrics = cpu_metrics;
|
||||
system_metrics.extend(memory_metrics);
|
||||
|
||||
// Add NixOS metrics - using exact matching for build display fix
|
||||
let nixos_metrics: Vec<&Metric> = all_metrics
|
||||
.iter()
|
||||
.filter(|m| m.name == "system_nixos_build" || m.name == "system_active_users" || m.name == "agent_version")
|
||||
.copied()
|
||||
.collect();
|
||||
system_metrics.extend(nixos_metrics);
|
||||
|
||||
// Add disk/storage metrics
|
||||
let disk_metrics: Vec<&Metric> = all_metrics
|
||||
.iter()
|
||||
.filter(|m| m.name.starts_with("disk_"))
|
||||
.copied()
|
||||
.collect();
|
||||
system_metrics.extend(disk_metrics);
|
||||
|
||||
host_widgets.system_widget.update_from_metrics(&system_metrics);
|
||||
@@ -192,7 +153,7 @@ impl TuiApp {
|
||||
.update_from_metrics(&service_metrics);
|
||||
host_widgets
|
||||
.backup_widget
|
||||
.update_from_metrics(&all_backup_metrics);
|
||||
.update_from_metrics(&backup_metrics);
|
||||
|
||||
host_widgets.last_update = Some(Instant::now());
|
||||
}
|
||||
@@ -200,30 +161,28 @@ impl TuiApp {
|
||||
}
|
||||
|
||||
/// Update available hosts with localhost prioritization
|
||||
pub fn update_hosts(&mut self, hosts: Vec<String>) {
|
||||
// Sort hosts alphabetically
|
||||
let mut sorted_hosts = hosts.clone();
|
||||
pub fn update_hosts(&mut self, discovered_hosts: Vec<String>) {
|
||||
// Start with configured hosts (always visible)
|
||||
let mut all_hosts: Vec<String> = self.config.hosts.keys().cloned().collect();
|
||||
|
||||
// Keep hosts that have pending transitions even if they're offline
|
||||
for (hostname, host_widgets) in &self.host_widgets {
|
||||
if !host_widgets.pending_service_transitions.is_empty() {
|
||||
if !sorted_hosts.contains(hostname) {
|
||||
sorted_hosts.push(hostname.clone());
|
||||
}
|
||||
// Add any discovered hosts that aren't already configured
|
||||
for host in discovered_hosts {
|
||||
if !all_hosts.contains(&host) {
|
||||
all_hosts.push(host);
|
||||
}
|
||||
}
|
||||
|
||||
sorted_hosts.sort();
|
||||
self.available_hosts = sorted_hosts;
|
||||
|
||||
all_hosts.sort();
|
||||
self.available_hosts = all_hosts;
|
||||
|
||||
// Get the current hostname (localhost) for auto-selection
|
||||
let localhost = gethostname::gethostname().to_string_lossy().to_string();
|
||||
if !self.available_hosts.is_empty() {
|
||||
if self.available_hosts.contains(&localhost) && !self.user_navigated_away {
|
||||
if self.available_hosts.contains(&self.localhost) && !self.user_navigated_away {
|
||||
// Localhost is available and user hasn't navigated away - switch to it
|
||||
self.current_host = Some(localhost.clone());
|
||||
self.current_host = Some(self.localhost.clone());
|
||||
// Find the actual index of localhost in the sorted list
|
||||
self.host_index = self.available_hosts.iter().position(|h| h == &localhost).unwrap_or(0);
|
||||
self.host_index = self.available_hosts.iter().position(|h| h == &self.localhost).unwrap_or(0);
|
||||
} else if self.current_host.is_none() {
|
||||
// No current host - select first available (which is localhost if available)
|
||||
self.current_host = Some(self.available_hosts[0].clone());
|
||||
@@ -243,7 +202,7 @@ impl TuiApp {
|
||||
}
|
||||
|
||||
/// Handle keyboard input
|
||||
pub fn handle_input(&mut self, event: Event) -> Result<Option<UiCommand>> {
|
||||
pub fn handle_input(&mut self, event: Event) -> Result<()> {
|
||||
if let Event::Key(key) = event {
|
||||
match key.code {
|
||||
KeyCode::Char('q') => {
|
||||
@@ -256,91 +215,197 @@ impl TuiApp {
|
||||
self.navigate_host(1);
|
||||
}
|
||||
KeyCode::Char('r') => {
|
||||
match self.focused_panel {
|
||||
PanelType::System => {
|
||||
// Simple tmux popup with SSH rebuild using configured user and alias
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
// Launch tmux popup with SSH using config values
|
||||
let ssh_command = format!(
|
||||
"ssh -tt {}@{} 'bash -ic {}'",
|
||||
self.config.ssh.rebuild_user,
|
||||
hostname,
|
||||
self.config.ssh.rebuild_alias
|
||||
);
|
||||
std::process::Command::new("tmux")
|
||||
.arg("display-popup")
|
||||
.arg(&ssh_command)
|
||||
.spawn()
|
||||
.ok(); // Ignore errors, tmux will handle them
|
||||
}
|
||||
}
|
||||
PanelType::Services => {
|
||||
// Service restart command
|
||||
if let (Some(service_name), Some(hostname)) = (self.get_selected_service(), self.current_host.clone()) {
|
||||
if self.start_command(&hostname, CommandType::ServiceRestart, service_name.clone()) {
|
||||
return Ok(Some(UiCommand::ServiceRestart { hostname, service_name }));
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
info!("Manual refresh requested");
|
||||
}
|
||||
// System rebuild command - works on any panel for current host
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
let connection_ip = self.get_connection_ip(&hostname);
|
||||
// Create command that shows logo, rebuilds, and waits for user input
|
||||
let logo_and_rebuild = format!(
|
||||
"echo 'Rebuilding system: {} ({})' && ssh -tt {}@{} \"bash -ic '{}'\"",
|
||||
hostname,
|
||||
connection_ip,
|
||||
self.config.ssh.rebuild_user,
|
||||
connection_ip,
|
||||
self.config.ssh.rebuild_cmd
|
||||
);
|
||||
|
||||
std::process::Command::new("tmux")
|
||||
.arg("split-window")
|
||||
.arg("-v")
|
||||
.arg("-p")
|
||||
.arg("30")
|
||||
.arg(&logo_and_rebuild)
|
||||
.spawn()
|
||||
.ok(); // Ignore errors, tmux will handle them
|
||||
}
|
||||
}
|
||||
KeyCode::Char('B') => {
|
||||
// Backup command - works on any panel for current host
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
let connection_ip = self.get_connection_ip(&hostname);
|
||||
// Create command that shows logo, runs backup, and waits for user input
|
||||
let logo_and_backup = format!(
|
||||
"echo 'Running backup: {} ({})' && ssh -tt {}@{} \"bash -ic '{}'\"",
|
||||
hostname,
|
||||
connection_ip,
|
||||
self.config.ssh.rebuild_user,
|
||||
connection_ip,
|
||||
format!("{} start borgbackup", self.config.ssh.service_manage_cmd)
|
||||
);
|
||||
|
||||
std::process::Command::new("tmux")
|
||||
.arg("split-window")
|
||||
.arg("-v")
|
||||
.arg("-p")
|
||||
.arg("30")
|
||||
.arg(&logo_and_backup)
|
||||
.spawn()
|
||||
.ok(); // Ignore errors, tmux will handle them
|
||||
}
|
||||
}
|
||||
KeyCode::Char('s') => {
|
||||
if self.focused_panel == PanelType::Services {
|
||||
// Service start command
|
||||
if let (Some(service_name), Some(hostname)) = (self.get_selected_service(), self.current_host.clone()) {
|
||||
if self.start_command(&hostname, CommandType::ServiceStart, service_name.clone()) {
|
||||
return Ok(Some(UiCommand::ServiceStart { hostname, service_name }));
|
||||
}
|
||||
}
|
||||
// Service start command via SSH with progress display
|
||||
if let (Some(service_name), Some(hostname)) = (self.get_selected_service(), self.current_host.clone()) {
|
||||
let connection_ip = self.get_connection_ip(&hostname);
|
||||
let service_start_command = format!(
|
||||
"echo 'Starting service: {} on {}' && ssh -tt {}@{} \"bash -ic '{} start {}'\"",
|
||||
service_name,
|
||||
hostname,
|
||||
self.config.ssh.rebuild_user,
|
||||
connection_ip,
|
||||
self.config.ssh.service_manage_cmd,
|
||||
service_name
|
||||
);
|
||||
|
||||
std::process::Command::new("tmux")
|
||||
.arg("split-window")
|
||||
.arg("-v")
|
||||
.arg("-p")
|
||||
.arg("30")
|
||||
.arg(&service_start_command)
|
||||
.spawn()
|
||||
.ok(); // Ignore errors, tmux will handle them
|
||||
}
|
||||
}
|
||||
KeyCode::Char('S') => {
|
||||
if self.focused_panel == PanelType::Services {
|
||||
// Service stop command
|
||||
if let (Some(service_name), Some(hostname)) = (self.get_selected_service(), self.current_host.clone()) {
|
||||
if self.start_command(&hostname, CommandType::ServiceStop, service_name.clone()) {
|
||||
return Ok(Some(UiCommand::ServiceStop { hostname, service_name }));
|
||||
// Service stop command via SSH with progress display
|
||||
if let (Some(service_name), Some(hostname)) = (self.get_selected_service(), self.current_host.clone()) {
|
||||
let connection_ip = self.get_connection_ip(&hostname);
|
||||
let service_stop_command = format!(
|
||||
"echo 'Stopping service: {} on {}' && ssh -tt {}@{} \"bash -ic '{} stop {}'\"",
|
||||
service_name,
|
||||
hostname,
|
||||
self.config.ssh.rebuild_user,
|
||||
connection_ip,
|
||||
self.config.ssh.service_manage_cmd,
|
||||
service_name
|
||||
);
|
||||
|
||||
std::process::Command::new("tmux")
|
||||
.arg("split-window")
|
||||
.arg("-v")
|
||||
.arg("-p")
|
||||
.arg("30")
|
||||
.arg(&service_stop_command)
|
||||
.spawn()
|
||||
.ok(); // Ignore errors, tmux will handle them
|
||||
}
|
||||
}
|
||||
KeyCode::Char('L') => {
|
||||
// Show service logs via service-manage script in tmux split window
|
||||
if let (Some(service_name), Some(hostname)) = (self.get_selected_service(), self.current_host.clone()) {
|
||||
let connection_ip = self.get_connection_ip(&hostname);
|
||||
let logs_command = format!(
|
||||
"ssh -tt {}@{} '{} logs {}'",
|
||||
self.config.ssh.rebuild_user,
|
||||
connection_ip,
|
||||
self.config.ssh.service_manage_cmd,
|
||||
service_name
|
||||
);
|
||||
|
||||
std::process::Command::new("tmux")
|
||||
.arg("split-window")
|
||||
.arg("-v")
|
||||
.arg("-p")
|
||||
.arg("30")
|
||||
.arg(&logs_command)
|
||||
.spawn()
|
||||
.ok(); // Ignore errors, tmux will handle them
|
||||
}
|
||||
}
|
||||
KeyCode::Char('w') => {
|
||||
// Wake on LAN for offline hosts
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
// Check if host has MAC address configured
|
||||
if let Some(host_details) = self.config.hosts.get(&hostname) {
|
||||
if let Some(mac_address) = &host_details.mac_address {
|
||||
// Parse MAC address and send WoL packet
|
||||
let mac_bytes = Self::parse_mac_address(mac_address);
|
||||
match mac_bytes {
|
||||
Ok(mac) => {
|
||||
match MagicPacket::new(&mac).send() {
|
||||
Ok(_) => {
|
||||
info!("WakeOnLAN packet sent successfully to {} ({})", hostname, mac_address);
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::error!("Failed to send WakeOnLAN packet to {}: {}", hostname, e);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(_) => {
|
||||
tracing::error!("Invalid MAC address format for {}: {}", hostname, mac_address);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
KeyCode::Char('b') => {
|
||||
if self.focused_panel == PanelType::Backup {
|
||||
// Trigger backup
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
self.start_command(&hostname, CommandType::BackupTrigger, hostname.clone());
|
||||
return Ok(Some(UiCommand::TriggerBackup { hostname }));
|
||||
}
|
||||
KeyCode::Char('t') => {
|
||||
// Open SSH terminal session in tmux window
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
let connection_ip = self.get_connection_ip(&hostname);
|
||||
let ssh_command = format!(
|
||||
"echo 'Opening SSH terminal to: {}' && ssh -tt {}@{}",
|
||||
hostname,
|
||||
self.config.ssh.rebuild_user,
|
||||
connection_ip
|
||||
);
|
||||
|
||||
std::process::Command::new("tmux")
|
||||
.arg("split-window")
|
||||
.arg("-v")
|
||||
.arg("-p")
|
||||
.arg("30") // Use 30% like other commands
|
||||
.arg(&ssh_command)
|
||||
.spawn()
|
||||
.ok(); // Ignore errors, tmux will handle them
|
||||
}
|
||||
}
|
||||
KeyCode::Tab => {
|
||||
if key.modifiers.contains(KeyModifiers::SHIFT) {
|
||||
// Shift+Tab cycles through panels
|
||||
self.next_panel();
|
||||
} else {
|
||||
// Tab cycles to next host
|
||||
self.navigate_host(1);
|
||||
// Tab cycles to next host
|
||||
self.navigate_host(1);
|
||||
}
|
||||
KeyCode::Up | KeyCode::Char('k') => {
|
||||
// Move service selection up
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
host_widgets.services_widget.select_previous();
|
||||
}
|
||||
}
|
||||
KeyCode::BackTab => {
|
||||
// BackTab (Shift+Tab on some terminals) also cycles panels
|
||||
self.next_panel();
|
||||
}
|
||||
KeyCode::Up => {
|
||||
// Scroll up in focused panel
|
||||
self.scroll_focused_panel(-1);
|
||||
}
|
||||
KeyCode::Down => {
|
||||
// Scroll down in focused panel
|
||||
self.scroll_focused_panel(1);
|
||||
KeyCode::Down | KeyCode::Char('j') => {
|
||||
// Move service selection down
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
let total_services = {
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
host_widgets.services_widget.get_total_services_count()
|
||||
};
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
host_widgets.services_widget.select_next(total_services);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
Ok(None)
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Navigate between hosts
|
||||
@@ -363,9 +428,8 @@ impl TuiApp {
|
||||
self.current_host = Some(self.available_hosts[self.host_index].clone());
|
||||
|
||||
// Check if user navigated away from localhost
|
||||
let localhost = gethostname::gethostname().to_string_lossy().to_string();
|
||||
if let Some(ref current) = self.current_host {
|
||||
if current != &localhost {
|
||||
if current != &self.localhost {
|
||||
self.user_navigated_away = true;
|
||||
} else {
|
||||
self.user_navigated_away = false; // User navigated back to localhost
|
||||
@@ -376,25 +440,6 @@ impl TuiApp {
|
||||
}
|
||||
|
||||
|
||||
/// Switch to next panel (Shift+Tab) - only cycles through visible panels
|
||||
pub fn next_panel(&mut self) {
|
||||
let visible_panels = self.get_visible_panels();
|
||||
if visible_panels.len() <= 1 {
|
||||
return; // Can't switch if only one or no panels visible
|
||||
}
|
||||
|
||||
// Find current panel index in visible panels
|
||||
if let Some(current_index) = visible_panels.iter().position(|&p| p == self.focused_panel) {
|
||||
// Move to next visible panel
|
||||
let next_index = (current_index + 1) % visible_panels.len();
|
||||
self.focused_panel = visible_panels[next_index];
|
||||
} else {
|
||||
// Current panel not visible, switch to first visible panel
|
||||
self.focused_panel = visible_panels[0];
|
||||
}
|
||||
|
||||
info!("Switched to panel: {:?}", self.focused_panel);
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -414,144 +459,11 @@ impl TuiApp {
|
||||
self.should_quit
|
||||
}
|
||||
|
||||
/// Get current service status for state-aware command validation
|
||||
fn get_current_service_status(&self, hostname: &str, service_name: &str) -> Option<String> {
|
||||
if let Some(host_widgets) = self.host_widgets.get(hostname) {
|
||||
return host_widgets.services_widget.get_service_status(service_name);
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Start command execution with immediate visual feedback
|
||||
pub fn start_command(&mut self, hostname: &str, command_type: CommandType, target: String) -> bool {
|
||||
// Get current service status to validate command
|
||||
let current_status = self.get_current_service_status(hostname, &target);
|
||||
|
||||
// Validate if command makes sense for current state
|
||||
let should_execute = match (&command_type, current_status.as_deref()) {
|
||||
(CommandType::ServiceStart, Some("inactive") | Some("failed") | Some("dead")) => true,
|
||||
(CommandType::ServiceStop, Some("active")) => true,
|
||||
(CommandType::ServiceRestart, Some("active") | Some("inactive") | Some("failed") | Some("dead")) => true,
|
||||
(CommandType::ServiceStart, Some("active")) => {
|
||||
// Already running - don't execute
|
||||
false
|
||||
},
|
||||
(CommandType::ServiceStop, Some("inactive") | Some("failed") | Some("dead")) => {
|
||||
// Already stopped - don't execute
|
||||
false
|
||||
},
|
||||
(_, None) => {
|
||||
// Unknown service state - allow command to proceed
|
||||
true
|
||||
},
|
||||
_ => true, // Default: allow other combinations
|
||||
};
|
||||
|
||||
// ALWAYS store the pending transition for immediate visual feedback, even if we don't execute
|
||||
if let Some(host_widgets) = self.host_widgets.get_mut(hostname) {
|
||||
host_widgets.pending_service_transitions.insert(
|
||||
target.clone(),
|
||||
(command_type, current_status.unwrap_or_else(|| "unknown".to_string()), Instant::now())
|
||||
);
|
||||
}
|
||||
|
||||
should_execute
|
||||
}
|
||||
|
||||
/// Clear pending transitions when real status updates arrive or timeout
|
||||
fn clear_completed_transitions(&mut self, hostname: &str, service_metrics: &[&Metric]) {
|
||||
if let Some(host_widgets) = self.host_widgets.get_mut(hostname) {
|
||||
let mut completed_services = Vec::new();
|
||||
|
||||
// Check each pending transition to see if real status has changed
|
||||
for (service_name, (command_type, original_status, _start_time)) in &host_widgets.pending_service_transitions {
|
||||
|
||||
// Look for status metric for this service
|
||||
for metric in service_metrics {
|
||||
if metric.name == format!("service_{}_status", service_name) {
|
||||
let new_status = metric.value.as_string();
|
||||
|
||||
// Check if status has changed from original (command completed)
|
||||
if &new_status != original_status {
|
||||
// Verify it changed in the expected direction
|
||||
let expected_change = match command_type {
|
||||
CommandType::ServiceStart => &new_status == "active",
|
||||
CommandType::ServiceStop => &new_status != "active",
|
||||
CommandType::ServiceRestart => true, // Any change indicates restart completed
|
||||
_ => false,
|
||||
};
|
||||
|
||||
if expected_change {
|
||||
completed_services.push(service_name.clone());
|
||||
}
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Remove completed transitions
|
||||
for service_name in completed_services {
|
||||
host_widgets.pending_service_transitions.remove(&service_name);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Scroll the focused panel up or down
|
||||
pub fn scroll_focused_panel(&mut self, direction: i32) {
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
let focused_panel = self.focused_panel; // Get the value before borrowing
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
|
||||
match focused_panel {
|
||||
PanelType::System => {
|
||||
if direction > 0 {
|
||||
host_widgets.system_scroll_offset = host_widgets.system_scroll_offset.saturating_add(1);
|
||||
} else {
|
||||
host_widgets.system_scroll_offset = host_widgets.system_scroll_offset.saturating_sub(1);
|
||||
}
|
||||
info!("System panel scroll offset: {}", host_widgets.system_scroll_offset);
|
||||
}
|
||||
PanelType::Services => {
|
||||
// For services panel, Up/Down moves selection cursor, not scroll
|
||||
let total_services = host_widgets.services_widget.get_total_services_count();
|
||||
|
||||
if direction > 0 {
|
||||
host_widgets.services_widget.select_next(total_services);
|
||||
info!("Services selection moved down");
|
||||
} else {
|
||||
host_widgets.services_widget.select_previous();
|
||||
info!("Services selection moved up");
|
||||
}
|
||||
}
|
||||
PanelType::Backup => {
|
||||
if direction > 0 {
|
||||
host_widgets.backup_scroll_offset = host_widgets.backup_scroll_offset.saturating_add(1);
|
||||
} else {
|
||||
host_widgets.backup_scroll_offset = host_widgets.backup_scroll_offset.saturating_sub(1);
|
||||
}
|
||||
info!("Backup panel scroll offset: {}", host_widgets.backup_scroll_offset);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/// Get list of currently visible panels
|
||||
fn get_visible_panels(&self) -> Vec<PanelType> {
|
||||
let mut visible_panels = vec![PanelType::System, PanelType::Services];
|
||||
|
||||
// Check if backup panel should be shown
|
||||
if let Some(hostname) = &self.current_host {
|
||||
if let Some(host_widgets) = self.host_widgets.get(hostname) {
|
||||
if host_widgets.backup_widget.has_data() {
|
||||
visible_panels.push(PanelType::Backup);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
visible_panels
|
||||
}
|
||||
|
||||
|
||||
/// Render the dashboard (real btop-style multi-panel layout)
|
||||
pub fn render(&mut self, frame: &mut Frame, metric_store: &MetricStore) {
|
||||
@@ -583,6 +495,21 @@ impl TuiApp {
|
||||
])
|
||||
.split(main_chunks[1]); // main_chunks[1] is now the content area (between title and statusbar)
|
||||
|
||||
// Check if current host is offline
|
||||
let current_host_offline = if let Some(hostname) = self.current_host.clone() {
|
||||
self.calculate_host_status(&hostname, metric_store) == Status::Offline
|
||||
} else {
|
||||
true // No host selected is considered offline
|
||||
};
|
||||
|
||||
// If host is offline, render wake-up message instead of panels
|
||||
if current_host_offline {
|
||||
self.render_offline_host_message(frame, main_chunks[1]);
|
||||
self.render_btop_title(frame, main_chunks[0], metric_store);
|
||||
self.render_statusbar(frame, main_chunks[2]);
|
||||
return;
|
||||
}
|
||||
|
||||
// Check if backup panel should be shown
|
||||
let show_backup = if let Some(hostname) = self.current_host.clone() {
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
@@ -620,15 +547,11 @@ impl TuiApp {
|
||||
|
||||
// Render services widget for current host
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
let is_focused = self.focused_panel == PanelType::Services;
|
||||
let (scroll_offset, pending_transitions) = {
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
(host_widgets.services_scroll_offset, host_widgets.pending_service_transitions.clone())
|
||||
};
|
||||
let is_focused = true; // Always show service selection
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
host_widgets
|
||||
.services_widget
|
||||
.render_with_transitions(frame, content_chunks[1], is_focused, scroll_offset, &pending_transitions); // Services takes full right side
|
||||
.render(frame, content_chunks[1], is_focused); // Services takes full right side
|
||||
}
|
||||
|
||||
// Render statusbar at the bottom
|
||||
@@ -644,48 +567,91 @@ impl TuiApp {
|
||||
|
||||
if self.available_hosts.is_empty() {
|
||||
let title_text = "cm-dashboard • no hosts discovered";
|
||||
let title = Paragraph::new(title_text).style(Typography::title());
|
||||
let title = Paragraph::new(title_text)
|
||||
.style(Style::default().fg(Theme::background()).bg(Theme::status_color(Status::Unknown)));
|
||||
frame.render_widget(title, area);
|
||||
return;
|
||||
}
|
||||
|
||||
// Create spans for each host with status indicators
|
||||
let mut spans = vec![Span::styled("cm-dashboard • ", Typography::title())];
|
||||
// Calculate worst-case status across all hosts (excluding offline)
|
||||
let mut worst_status = Status::Ok;
|
||||
for host in &self.available_hosts {
|
||||
let host_status = self.calculate_host_status(host, metric_store);
|
||||
// Don't include offline hosts in status aggregation
|
||||
if host_status != Status::Offline {
|
||||
worst_status = Status::aggregate(&[worst_status, host_status]);
|
||||
}
|
||||
}
|
||||
|
||||
// Use the worst status color as background
|
||||
let background_color = Theme::status_color(worst_status);
|
||||
|
||||
// Split the title bar into left and right sections
|
||||
let chunks = Layout::default()
|
||||
.direction(Direction::Horizontal)
|
||||
.constraints([Constraint::Length(22), Constraint::Min(0)])
|
||||
.split(area);
|
||||
|
||||
// Left side: "cm-dashboard" text with version
|
||||
let title_text = format!(" cm-dashboard v{}", env!("CARGO_PKG_VERSION"));
|
||||
let left_span = Span::styled(
|
||||
&title_text,
|
||||
Style::default().fg(Theme::background()).bg(background_color).add_modifier(Modifier::BOLD)
|
||||
);
|
||||
let left_title = Paragraph::new(Line::from(vec![left_span]))
|
||||
.style(Style::default().bg(background_color));
|
||||
frame.render_widget(left_title, chunks[0]);
|
||||
|
||||
// Right side: hosts with status indicators
|
||||
let mut host_spans = Vec::new();
|
||||
|
||||
for (i, host) in self.available_hosts.iter().enumerate() {
|
||||
if i > 0 {
|
||||
spans.push(Span::styled(" ", Typography::title()));
|
||||
host_spans.push(Span::styled(
|
||||
" ",
|
||||
Style::default().fg(Theme::background()).bg(background_color)
|
||||
));
|
||||
}
|
||||
|
||||
// Always show normal status icon based on metrics (no command status at host level)
|
||||
let host_status = self.calculate_host_status(host, metric_store);
|
||||
let (status_icon, status_color) = (StatusIcons::get_icon(host_status), Theme::status_color(host_status));
|
||||
let status_icon = StatusIcons::get_icon(host_status);
|
||||
|
||||
// Add status icon
|
||||
spans.push(Span::styled(
|
||||
// Add status icon with background color as foreground against status background
|
||||
host_spans.push(Span::styled(
|
||||
format!("{} ", status_icon),
|
||||
Style::default().fg(status_color),
|
||||
Style::default().fg(Theme::background()).bg(background_color),
|
||||
));
|
||||
|
||||
if Some(host) == self.current_host.as_ref() {
|
||||
// Selected host in bold bright white
|
||||
spans.push(Span::styled(
|
||||
// Selected host in bold background color against status background
|
||||
host_spans.push(Span::styled(
|
||||
host.clone(),
|
||||
Typography::title().add_modifier(Modifier::BOLD),
|
||||
Style::default()
|
||||
.fg(Theme::background())
|
||||
.bg(background_color)
|
||||
.add_modifier(Modifier::BOLD),
|
||||
));
|
||||
} else {
|
||||
// Other hosts in normal style with status color
|
||||
spans.push(Span::styled(
|
||||
// Other hosts in normal background color against status background
|
||||
host_spans.push(Span::styled(
|
||||
host.clone(),
|
||||
Style::default().fg(status_color),
|
||||
Style::default().fg(Theme::background()).bg(background_color),
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
let title_line = Line::from(spans);
|
||||
let title = Paragraph::new(vec![title_line]);
|
||||
// Add right padding
|
||||
host_spans.push(Span::styled(
|
||||
" ",
|
||||
Style::default().fg(Theme::background()).bg(background_color)
|
||||
));
|
||||
|
||||
frame.render_widget(title, area);
|
||||
let host_line = Line::from(host_spans);
|
||||
let host_title = Paragraph::new(vec![host_line])
|
||||
.style(Style::default().bg(background_color))
|
||||
.alignment(ratatui::layout::Alignment::Right);
|
||||
frame.render_widget(host_title, chunks[1]);
|
||||
}
|
||||
|
||||
/// Calculate overall status for a host based on its metrics
|
||||
@@ -693,7 +659,7 @@ impl TuiApp {
|
||||
let metrics = metric_store.get_metrics_for_host(hostname);
|
||||
|
||||
if metrics.is_empty() {
|
||||
return Status::Unknown;
|
||||
return Status::Offline;
|
||||
}
|
||||
|
||||
// First check if we have the aggregated host status summary from the agent
|
||||
@@ -701,33 +667,27 @@ impl TuiApp {
|
||||
return host_summary_metric.status;
|
||||
}
|
||||
|
||||
// Fallback to old aggregation logic with proper Pending handling
|
||||
// Rewritten status aggregation - only Critical, Warning, or OK for top bar
|
||||
let mut has_critical = false;
|
||||
let mut has_warning = false;
|
||||
let mut has_pending = false;
|
||||
let mut ok_count = 0;
|
||||
|
||||
for metric in &metrics {
|
||||
match metric.status {
|
||||
Status::Critical => has_critical = true,
|
||||
Status::Warning => has_warning = true,
|
||||
Status::Pending => has_pending = true,
|
||||
Status::Ok => ok_count += 1,
|
||||
Status::Unknown => {} // Ignore unknown for aggregation
|
||||
// Treat all other statuses as OK for top bar aggregation
|
||||
Status::Ok | Status::Pending | Status::Inactive | Status::Unknown => {},
|
||||
Status::Offline => {}, // Ignore offline
|
||||
}
|
||||
}
|
||||
|
||||
// Priority order: Critical > Warning > Pending > Ok > Unknown
|
||||
// Only return Critical, Warning, or OK - no other statuses
|
||||
if has_critical {
|
||||
Status::Critical
|
||||
} else if has_warning {
|
||||
Status::Warning
|
||||
} else if has_pending {
|
||||
Status::Pending
|
||||
} else if ok_count > 0 {
|
||||
Status::Ok
|
||||
} else {
|
||||
Status::Unknown
|
||||
Status::Ok
|
||||
}
|
||||
}
|
||||
|
||||
@@ -748,71 +708,140 @@ impl TuiApp {
|
||||
let mut shortcuts = Vec::new();
|
||||
|
||||
// Global shortcuts
|
||||
shortcuts.push("Tab: Switch Host".to_string());
|
||||
shortcuts.push("Shift+Tab: Switch Panel".to_string());
|
||||
|
||||
// Scroll shortcuts (always available)
|
||||
shortcuts.push("↑↓: Scroll".to_string());
|
||||
|
||||
// Panel-specific shortcuts
|
||||
match self.focused_panel {
|
||||
PanelType::System => {
|
||||
shortcuts.push("R: Rebuild".to_string());
|
||||
}
|
||||
PanelType::Services => {
|
||||
shortcuts.push("S: Start".to_string());
|
||||
shortcuts.push("Shift+S: Stop".to_string());
|
||||
shortcuts.push("R: Restart".to_string());
|
||||
}
|
||||
PanelType::Backup => {
|
||||
shortcuts.push("B: Trigger Backup".to_string());
|
||||
}
|
||||
}
|
||||
shortcuts.push("Tab: Host".to_string());
|
||||
shortcuts.push("↑↓/jk: Select".to_string());
|
||||
shortcuts.push("r: Rebuild".to_string());
|
||||
shortcuts.push("B: Backup".to_string());
|
||||
shortcuts.push("s/S: Start/Stop".to_string());
|
||||
shortcuts.push("L: Logs".to_string());
|
||||
shortcuts.push("t: Terminal".to_string());
|
||||
shortcuts.push("w: Wake".to_string());
|
||||
|
||||
// Always show quit
|
||||
shortcuts.push("Q: Quit".to_string());
|
||||
shortcuts.push("q: Quit".to_string());
|
||||
|
||||
shortcuts
|
||||
}
|
||||
|
||||
fn render_system_panel(&mut self, frame: &mut Frame, area: Rect, _metric_store: &MetricStore) {
|
||||
let system_block = if self.focused_panel == PanelType::System {
|
||||
Components::focused_widget_block("system")
|
||||
} else {
|
||||
Components::widget_block("system")
|
||||
};
|
||||
let system_block = Components::widget_block("system");
|
||||
let inner_area = system_block.inner(area);
|
||||
frame.render_widget(system_block, area);
|
||||
// Get current host widgets, create if none exist
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
let scroll_offset = {
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
host_widgets.system_scroll_offset
|
||||
};
|
||||
// Clone the config to avoid borrowing issues
|
||||
let config = self.config.clone();
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
host_widgets.system_widget.render_with_scroll(frame, inner_area, scroll_offset);
|
||||
host_widgets.system_widget.render(frame, inner_area, &hostname, Some(&config));
|
||||
}
|
||||
}
|
||||
|
||||
fn render_backup_panel(&mut self, frame: &mut Frame, area: Rect) {
|
||||
let backup_block = if self.focused_panel == PanelType::Backup {
|
||||
Components::focused_widget_block("backup")
|
||||
} else {
|
||||
Components::widget_block("backup")
|
||||
};
|
||||
let backup_block = Components::widget_block("backup");
|
||||
let inner_area = backup_block.inner(area);
|
||||
frame.render_widget(backup_block, area);
|
||||
|
||||
// Get current host widgets for backup widget
|
||||
if let Some(hostname) = self.current_host.clone() {
|
||||
let scroll_offset = {
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
host_widgets.backup_scroll_offset
|
||||
};
|
||||
let host_widgets = self.get_or_create_host_widgets(&hostname);
|
||||
host_widgets.backup_widget.render_with_scroll(frame, inner_area, scroll_offset);
|
||||
host_widgets.backup_widget.render(frame, inner_area);
|
||||
}
|
||||
}
|
||||
|
||||
/// Render offline host message with wake-up option
|
||||
fn render_offline_host_message(&self, frame: &mut Frame, area: Rect) {
|
||||
use ratatui::layout::Alignment;
|
||||
use ratatui::style::Modifier;
|
||||
use ratatui::text::{Line, Span};
|
||||
use ratatui::widgets::{Block, Borders, Paragraph};
|
||||
|
||||
// Get hostname for message
|
||||
let hostname = self.current_host.as_ref()
|
||||
.map(|h| h.as_str())
|
||||
.unwrap_or("Unknown");
|
||||
|
||||
// Check if host has MAC address for wake-on-LAN
|
||||
let has_mac = self.current_host.as_ref()
|
||||
.and_then(|hostname| self.config.hosts.get(hostname))
|
||||
.and_then(|details| details.mac_address.as_ref())
|
||||
.is_some();
|
||||
|
||||
// Create message content
|
||||
let mut lines = vec![
|
||||
Line::from(Span::styled(
|
||||
format!("Host '{}' is offline", hostname),
|
||||
Style::default().fg(Theme::muted_text()).add_modifier(Modifier::BOLD),
|
||||
)),
|
||||
Line::from(""),
|
||||
];
|
||||
|
||||
if has_mac {
|
||||
lines.push(Line::from(Span::styled(
|
||||
"Press 'w' to wake up host",
|
||||
Style::default().fg(Theme::primary_text()).add_modifier(Modifier::BOLD),
|
||||
)));
|
||||
} else {
|
||||
lines.push(Line::from(Span::styled(
|
||||
"No MAC address configured - cannot wake up",
|
||||
Style::default().fg(Theme::muted_text()),
|
||||
)));
|
||||
}
|
||||
|
||||
// Create centered message
|
||||
let message = Paragraph::new(lines)
|
||||
.block(Block::default()
|
||||
.borders(Borders::ALL)
|
||||
.border_style(Style::default().fg(Theme::muted_text()))
|
||||
.title(" Offline Host ")
|
||||
.title_style(Style::default().fg(Theme::muted_text()).add_modifier(Modifier::BOLD)))
|
||||
.style(Style::default().bg(Theme::background()).fg(Theme::primary_text()))
|
||||
.alignment(Alignment::Center);
|
||||
|
||||
// Center the message in the available area
|
||||
let popup_area = ratatui::layout::Layout::default()
|
||||
.direction(Direction::Vertical)
|
||||
.constraints([
|
||||
Constraint::Percentage(40),
|
||||
Constraint::Length(6),
|
||||
Constraint::Percentage(40),
|
||||
])
|
||||
.split(area)[1];
|
||||
|
||||
let popup_area = ratatui::layout::Layout::default()
|
||||
.direction(Direction::Horizontal)
|
||||
.constraints([
|
||||
Constraint::Percentage(25),
|
||||
Constraint::Percentage(50),
|
||||
Constraint::Percentage(25),
|
||||
])
|
||||
.split(popup_area)[1];
|
||||
|
||||
frame.render_widget(message, popup_area);
|
||||
}
|
||||
|
||||
/// Parse MAC address string (e.g., "AA:BB:CC:DD:EE:FF") to [u8; 6]
|
||||
/// Get the connection IP for a hostname based on host configuration
|
||||
fn get_connection_ip(&self, hostname: &str) -> String {
|
||||
if let Some(host_details) = self.config.hosts.get(hostname) {
|
||||
host_details.get_connection_ip(hostname)
|
||||
} else {
|
||||
hostname.to_string()
|
||||
}
|
||||
}
|
||||
|
||||
fn parse_mac_address(mac_str: &str) -> Result<[u8; 6], &'static str> {
|
||||
let parts: Vec<&str> = mac_str.split(':').collect();
|
||||
if parts.len() != 6 {
|
||||
return Err("MAC address must have 6 parts separated by colons");
|
||||
}
|
||||
|
||||
let mut mac = [0u8; 6];
|
||||
for (i, part) in parts.iter().enumerate() {
|
||||
match u8::from_str_radix(part, 16) {
|
||||
Ok(byte) => mac[i] = byte,
|
||||
Err(_) => return Err("Invalid hexadecimal byte in MAC address"),
|
||||
}
|
||||
}
|
||||
Ok(mac)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -143,10 +143,12 @@ impl Theme {
|
||||
pub fn status_color(status: Status) -> Color {
|
||||
match status {
|
||||
Status::Ok => Self::success(),
|
||||
Status::Inactive => Self::muted_text(), // Gray for inactive services in service list
|
||||
Status::Pending => Self::highlight(), // Blue for pending
|
||||
Status::Warning => Self::warning(),
|
||||
Status::Critical => Self::error(),
|
||||
Status::Unknown => Self::muted_text(),
|
||||
Status::Offline => Self::muted_text(), // Dark gray for offline
|
||||
}
|
||||
}
|
||||
|
||||
@@ -242,10 +244,12 @@ impl StatusIcons {
|
||||
pub fn get_icon(status: Status) -> &'static str {
|
||||
match status {
|
||||
Status::Ok => "●",
|
||||
Status::Inactive => "○", // Empty circle for inactive services
|
||||
Status::Pending => "◉", // Hollow circle for pending
|
||||
Status::Warning => "◐",
|
||||
Status::Critical => "◯",
|
||||
Status::Critical => "!",
|
||||
Status::Unknown => "?",
|
||||
Status::Offline => "○", // Empty circle for offline
|
||||
}
|
||||
}
|
||||
|
||||
@@ -254,10 +258,12 @@ impl StatusIcons {
|
||||
let icon = Self::get_icon(status);
|
||||
let status_color = match status {
|
||||
Status::Ok => Theme::success(), // Green
|
||||
Status::Inactive => Theme::muted_text(), // Gray for inactive services
|
||||
Status::Pending => Theme::highlight(), // Blue
|
||||
Status::Warning => Theme::warning(), // Yellow
|
||||
Status::Critical => Theme::error(), // Red
|
||||
Status::Unknown => Theme::muted_text(), // Gray
|
||||
Status::Offline => Theme::muted_text(), // Dark gray for offline
|
||||
};
|
||||
|
||||
vec![
|
||||
@@ -289,27 +295,9 @@ impl Components {
|
||||
)
|
||||
}
|
||||
|
||||
/// Widget block with focus indicator (blue border)
|
||||
pub fn focused_widget_block(title: &str) -> Block<'_> {
|
||||
Block::default()
|
||||
.title(title)
|
||||
.borders(Borders::ALL)
|
||||
.style(Style::default().fg(Theme::highlight()).bg(Theme::background())) // Blue border for focus
|
||||
.title_style(
|
||||
Style::default()
|
||||
.fg(Theme::highlight()) // Blue title for focus
|
||||
.bg(Theme::background()),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
impl Typography {
|
||||
/// Main title style (dashboard header)
|
||||
pub fn title() -> Style {
|
||||
Style::default()
|
||||
.fg(Theme::primary_text())
|
||||
.bg(Theme::background())
|
||||
}
|
||||
|
||||
/// Widget title style (panel headers) - bold bright white
|
||||
pub fn widget_title() -> Style {
|
||||
|
||||
@@ -30,6 +30,8 @@ pub struct BackupWidget {
|
||||
backup_disk_product_name: Option<String>,
|
||||
/// Backup disk serial number from SMART data
|
||||
backup_disk_serial_number: Option<String>,
|
||||
/// Backup disk wear percentage from SMART data
|
||||
backup_disk_wear_percent: Option<f32>,
|
||||
/// Backup disk filesystem label
|
||||
backup_disk_filesystem_label: Option<String>,
|
||||
/// Number of completed services
|
||||
@@ -65,6 +67,7 @@ impl BackupWidget {
|
||||
backup_disk_used_gb: None,
|
||||
backup_disk_product_name: None,
|
||||
backup_disk_serial_number: None,
|
||||
backup_disk_wear_percent: None,
|
||||
backup_disk_filesystem_label: None,
|
||||
services_completed_count: None,
|
||||
services_failed_count: None,
|
||||
@@ -197,6 +200,9 @@ impl Widget for BackupWidget {
|
||||
"backup_disk_serial_number" => {
|
||||
self.backup_disk_serial_number = Some(metric.value.as_string());
|
||||
}
|
||||
"backup_disk_wear_percent" => {
|
||||
self.backup_disk_wear_percent = metric.value.as_f32();
|
||||
}
|
||||
"backup_disk_filesystem_label" => {
|
||||
self.backup_disk_filesystem_label = Some(metric.value.as_string());
|
||||
}
|
||||
@@ -285,8 +291,8 @@ impl Widget for BackupWidget {
|
||||
}
|
||||
|
||||
impl BackupWidget {
|
||||
/// Render with scroll offset support
|
||||
pub fn render_with_scroll(&mut self, frame: &mut Frame, area: Rect, scroll_offset: usize) {
|
||||
/// Render backup widget
|
||||
pub fn render(&mut self, frame: &mut Frame, area: Rect) {
|
||||
let mut lines = Vec::new();
|
||||
|
||||
// Latest backup section
|
||||
@@ -328,21 +334,31 @@ impl BackupWidget {
|
||||
);
|
||||
lines.push(ratatui::text::Line::from(disk_spans));
|
||||
|
||||
// Serial number as sub-item
|
||||
// Collect sub-items to determine tree structure
|
||||
let mut sub_items = Vec::new();
|
||||
|
||||
if let Some(serial) = &self.backup_disk_serial_number {
|
||||
lines.push(ratatui::text::Line::from(vec![
|
||||
ratatui::text::Span::styled(" ├─ ", Typography::tree()),
|
||||
ratatui::text::Span::styled(format!("S/N: {}", serial), Typography::secondary())
|
||||
]));
|
||||
sub_items.push(format!("S/N: {}", serial));
|
||||
}
|
||||
|
||||
if let Some(wear) = self.backup_disk_wear_percent {
|
||||
sub_items.push(format!("Wear: {:.0}%", wear));
|
||||
}
|
||||
|
||||
// Usage as sub-item
|
||||
if let (Some(used), Some(total)) = (self.backup_disk_used_gb, self.backup_disk_total_gb) {
|
||||
let used_str = Self::format_size_with_proper_units(used);
|
||||
let total_str = Self::format_size_with_proper_units(total);
|
||||
sub_items.push(format!("Usage: {}/{}", used_str, total_str));
|
||||
}
|
||||
|
||||
// Render sub-items with proper tree structure
|
||||
let num_items = sub_items.len();
|
||||
for (i, item) in sub_items.into_iter().enumerate() {
|
||||
let is_last = i == num_items - 1;
|
||||
let tree_char = if is_last { " └─ " } else { " ├─ " };
|
||||
lines.push(ratatui::text::Line::from(vec![
|
||||
ratatui::text::Span::styled(" └─ ", Typography::tree()),
|
||||
ratatui::text::Span::styled(format!("Usage: {}/{}", used_str, total_str), Typography::secondary())
|
||||
ratatui::text::Span::styled(tree_char, Typography::tree()),
|
||||
ratatui::text::Span::styled(item, Typography::secondary())
|
||||
]));
|
||||
}
|
||||
}
|
||||
@@ -366,42 +382,20 @@ impl BackupWidget {
|
||||
let total_lines = lines.len();
|
||||
let available_height = area.height as usize;
|
||||
|
||||
// Calculate scroll boundaries
|
||||
let max_scroll = if total_lines > available_height {
|
||||
total_lines - available_height
|
||||
} else {
|
||||
total_lines.saturating_sub(1)
|
||||
};
|
||||
let effective_scroll = scroll_offset.min(max_scroll);
|
||||
|
||||
// Apply scrolling if needed
|
||||
if scroll_offset > 0 || total_lines > available_height {
|
||||
// Show only what fits, with "X more below" if needed
|
||||
if total_lines > available_height {
|
||||
let lines_for_content = available_height.saturating_sub(1); // Reserve one line for "more below"
|
||||
let mut visible_lines: Vec<_> = lines
|
||||
.into_iter()
|
||||
.skip(effective_scroll)
|
||||
.take(available_height)
|
||||
.take(lines_for_content)
|
||||
.collect();
|
||||
|
||||
// Add scroll indicator if there are hidden lines
|
||||
if total_lines > available_height {
|
||||
let hidden_above = effective_scroll;
|
||||
let hidden_below = total_lines.saturating_sub(effective_scroll + available_height);
|
||||
|
||||
if (hidden_above > 0 || hidden_below > 0) && !visible_lines.is_empty() {
|
||||
let scroll_text = if hidden_above > 0 && hidden_below > 0 {
|
||||
format!("... {} above, {} below", hidden_above, hidden_below)
|
||||
} else if hidden_above > 0 {
|
||||
format!("... {} more above", hidden_above)
|
||||
} else {
|
||||
format!("... {} more below", hidden_below)
|
||||
};
|
||||
|
||||
// Replace last line with scroll indicator
|
||||
visible_lines.pop();
|
||||
visible_lines.push(ratatui::text::Line::from(vec![
|
||||
ratatui::text::Span::styled(scroll_text, Typography::muted())
|
||||
]));
|
||||
}
|
||||
let hidden_below = total_lines.saturating_sub(lines_for_content);
|
||||
if hidden_below > 0 {
|
||||
let more_line = ratatui::text::Line::from(vec![
|
||||
ratatui::text::Span::styled(format!("... {} more below", hidden_below), Typography::muted())
|
||||
]);
|
||||
visible_lines.push(more_line);
|
||||
}
|
||||
|
||||
let paragraph = Paragraph::new(ratatui::text::Text::from(visible_lines));
|
||||
|
||||
@@ -9,7 +9,6 @@ use tracing::debug;
|
||||
|
||||
use super::Widget;
|
||||
use crate::ui::theme::{Components, StatusIcons, Theme, Typography};
|
||||
use crate::ui::CommandType;
|
||||
use ratatui::style::Style;
|
||||
|
||||
/// Services widget displaying hierarchical systemd service statuses
|
||||
@@ -113,13 +112,10 @@ impl ServicesWidget {
|
||||
name.to_string()
|
||||
};
|
||||
|
||||
// Parent services always show active/inactive status
|
||||
// Parent services always show actual systemctl status
|
||||
let status_str = match info.widget_status {
|
||||
Status::Ok => "active".to_string(),
|
||||
Status::Pending => "pending".to_string(),
|
||||
Status::Warning => "inactive".to_string(),
|
||||
Status::Critical => "failed".to_string(),
|
||||
Status::Unknown => "unknown".to_string(),
|
||||
_ => info.status.clone(), // Use actual status from agent (active/inactive/failed)
|
||||
};
|
||||
|
||||
format!(
|
||||
@@ -128,41 +124,14 @@ impl ServicesWidget {
|
||||
)
|
||||
}
|
||||
|
||||
/// Get status icon for service, considering pending transitions for visual feedback
|
||||
fn get_service_icon_and_status(&self, service_name: &str, info: &ServiceInfo, pending_transitions: &HashMap<String, (CommandType, String, std::time::Instant)>) -> (String, String, ratatui::prelude::Color) {
|
||||
// Check if this service has a pending transition
|
||||
if let Some((command_type, _original_status, _start_time)) = pending_transitions.get(service_name) {
|
||||
// Show transitional icons for pending commands
|
||||
let (icon, status_text) = match command_type {
|
||||
CommandType::ServiceRestart => ("↻", "restarting"),
|
||||
CommandType::ServiceStart => ("↑", "starting"),
|
||||
CommandType::ServiceStop => ("↓", "stopping"),
|
||||
_ => return (StatusIcons::get_icon(info.widget_status).to_string(), info.status.clone(), Theme::status_color(info.widget_status)), // Not a service command
|
||||
};
|
||||
return (icon.to_string(), status_text.to_string(), Theme::highlight());
|
||||
}
|
||||
|
||||
// Normal status display
|
||||
let icon = StatusIcons::get_icon(info.widget_status);
|
||||
let status_color = match info.widget_status {
|
||||
Status::Ok => Theme::success(),
|
||||
Status::Pending => Theme::highlight(),
|
||||
Status::Warning => Theme::warning(),
|
||||
Status::Critical => Theme::error(),
|
||||
Status::Unknown => Theme::muted_text(),
|
||||
};
|
||||
|
||||
(icon.to_string(), info.status.clone(), status_color)
|
||||
}
|
||||
|
||||
|
||||
/// Create spans for sub-service with icon next to name, considering pending transitions
|
||||
fn create_sub_service_spans_with_transitions(
|
||||
/// Create spans for sub-service with icon next to name
|
||||
fn create_sub_service_spans(
|
||||
&self,
|
||||
name: &str,
|
||||
info: &ServiceInfo,
|
||||
is_last: bool,
|
||||
pending_transitions: &HashMap<String, (CommandType, String, std::time::Instant)>,
|
||||
) -> Vec<ratatui::text::Span<'static>> {
|
||||
// Truncate long sub-service names to fit layout (accounting for indentation)
|
||||
let short_name = if name.len() > 18 {
|
||||
@@ -171,19 +140,28 @@ impl ServicesWidget {
|
||||
name.to_string()
|
||||
};
|
||||
|
||||
// Get status icon and text, considering pending transitions
|
||||
let (icon, mut status_str, status_color) = self.get_service_icon_and_status(name, info, pending_transitions);
|
||||
// Get status icon and text
|
||||
let icon = StatusIcons::get_icon(info.widget_status);
|
||||
let status_color = match info.widget_status {
|
||||
Status::Ok => Theme::success(),
|
||||
Status::Inactive => Theme::muted_text(),
|
||||
Status::Pending => Theme::highlight(),
|
||||
Status::Warning => Theme::warning(),
|
||||
Status::Critical => Theme::error(),
|
||||
Status::Unknown => Theme::muted_text(),
|
||||
Status::Offline => Theme::muted_text(),
|
||||
};
|
||||
|
||||
// For sub-services, prefer latency if available (unless transition is pending)
|
||||
if !pending_transitions.contains_key(name) {
|
||||
if let Some(latency) = info.latency_ms {
|
||||
status_str = if latency < 0.0 {
|
||||
"timeout".to_string()
|
||||
} else {
|
||||
format!("{:.0}ms", latency)
|
||||
};
|
||||
// For sub-services, prefer latency if available
|
||||
let status_str = if let Some(latency) = info.latency_ms {
|
||||
if latency < 0.0 {
|
||||
"timeout".to_string()
|
||||
} else {
|
||||
format!("{:.0}ms", latency)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
info.status.clone()
|
||||
};
|
||||
let tree_symbol = if is_last { "└─" } else { "├─" };
|
||||
|
||||
vec![
|
||||
@@ -231,36 +209,13 @@ impl ServicesWidget {
|
||||
}
|
||||
|
||||
/// Get currently selected service name (for actions)
|
||||
/// Only returns parent service names since only parent services can be selected
|
||||
pub fn get_selected_service(&self) -> Option<String> {
|
||||
// Build the same display list to find the selected service
|
||||
let mut display_lines: Vec<(String, Status, bool, Option<(ServiceInfo, bool)>, String)> = Vec::new();
|
||||
|
||||
// Only parent services can be selected, so just get the parent service at selected_index
|
||||
let mut parent_services: Vec<_> = self.parent_services.iter().collect();
|
||||
parent_services.sort_by(|(a, _), (b, _)| a.cmp(b));
|
||||
|
||||
for (parent_name, parent_info) in parent_services {
|
||||
let parent_line = self.format_parent_service_line(parent_name, parent_info);
|
||||
display_lines.push((parent_line, parent_info.widget_status, false, None, parent_name.clone()));
|
||||
|
||||
if let Some(sub_list) = self.sub_services.get(parent_name) {
|
||||
let mut sorted_subs = sub_list.clone();
|
||||
sorted_subs.sort_by(|(a, _), (b, _)| a.cmp(b));
|
||||
|
||||
for (i, (sub_name, sub_info)) in sorted_subs.iter().enumerate() {
|
||||
let is_last_sub = i == sorted_subs.len() - 1;
|
||||
let full_sub_name = format!("{}_{}", parent_name, sub_name);
|
||||
display_lines.push((
|
||||
sub_name.clone(),
|
||||
sub_info.widget_status,
|
||||
true,
|
||||
Some((sub_info.clone(), is_last_sub)),
|
||||
full_sub_name,
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
display_lines.get(self.selected_index).map(|(_, _, _, _, raw_name)| raw_name.clone())
|
||||
parent_services.get(self.selected_index).map(|(name, _)| name.to_string())
|
||||
}
|
||||
|
||||
/// Get total count of selectable services (parent services only, not sub-services)
|
||||
@@ -269,25 +224,6 @@ impl ServicesWidget {
|
||||
self.parent_services.len()
|
||||
}
|
||||
|
||||
/// Get current status of a specific service by name
|
||||
pub fn get_service_status(&self, service_name: &str) -> Option<String> {
|
||||
// Check if it's a parent service
|
||||
if let Some(parent_info) = self.parent_services.get(service_name) {
|
||||
return Some(parent_info.status.clone());
|
||||
}
|
||||
|
||||
// Check sub-services (format: parent_sub)
|
||||
for (parent_name, sub_list) in &self.sub_services {
|
||||
for (sub_name, sub_info) in sub_list {
|
||||
let full_sub_name = format!("{}_{}", parent_name, sub_name);
|
||||
if full_sub_name == service_name {
|
||||
return Some(sub_info.status.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
None
|
||||
}
|
||||
|
||||
/// Calculate which parent service index corresponds to a display line index
|
||||
fn calculate_parent_service_index(&self, display_line_index: &usize) -> usize {
|
||||
@@ -442,13 +378,9 @@ impl Widget for ServicesWidget {
|
||||
|
||||
impl ServicesWidget {
|
||||
|
||||
/// Render with focus, scroll, and pending transitions for visual feedback
|
||||
pub fn render_with_transitions(&mut self, frame: &mut Frame, area: Rect, is_focused: bool, scroll_offset: usize, pending_transitions: &HashMap<String, (CommandType, String, std::time::Instant)>) {
|
||||
let services_block = if is_focused {
|
||||
Components::focused_widget_block("services")
|
||||
} else {
|
||||
Components::widget_block("services")
|
||||
};
|
||||
/// Render with focus
|
||||
pub fn render(&mut self, frame: &mut Frame, area: Rect, is_focused: bool) {
|
||||
let services_block = Components::widget_block("services");
|
||||
let inner_area = services_block.inner(area);
|
||||
frame.render_widget(services_block, area);
|
||||
|
||||
@@ -472,14 +404,14 @@ impl ServicesWidget {
|
||||
return;
|
||||
}
|
||||
|
||||
// Use the existing render logic but with pending transitions
|
||||
self.render_services_with_transitions(frame, content_chunks[1], is_focused, scroll_offset, pending_transitions);
|
||||
// Render the services list
|
||||
self.render_services(frame, content_chunks[1], is_focused);
|
||||
}
|
||||
|
||||
/// Render services list with pending transitions awareness
|
||||
fn render_services_with_transitions(&mut self, frame: &mut Frame, area: Rect, is_focused: bool, scroll_offset: usize, pending_transitions: &HashMap<String, (CommandType, String, std::time::Instant)>) {
|
||||
// Build hierarchical service list for display - include raw service name for pending transition lookups
|
||||
let mut display_lines: Vec<(String, Status, bool, Option<(ServiceInfo, bool)>, String)> = Vec::new(); // Added raw service name
|
||||
/// Render services list
|
||||
fn render_services(&mut self, frame: &mut Frame, area: Rect, is_focused: bool) {
|
||||
// Build hierarchical service list for display
|
||||
let mut display_lines: Vec<(String, Status, bool, Option<(ServiceInfo, bool)>)> = Vec::new();
|
||||
|
||||
// Sort parent services alphabetically for consistent order
|
||||
let mut parent_services: Vec<_> = self.parent_services.iter().collect();
|
||||
@@ -488,7 +420,7 @@ impl ServicesWidget {
|
||||
for (parent_name, parent_info) in parent_services {
|
||||
// Add parent service line
|
||||
let parent_line = self.format_parent_service_line(parent_name, parent_info);
|
||||
display_lines.push((parent_line, parent_info.widget_status, false, None, parent_name.clone())); // Include raw name
|
||||
display_lines.push((parent_line, parent_info.widget_status, false, None));
|
||||
|
||||
// Add sub-services for this parent (if any)
|
||||
if let Some(sub_list) = self.sub_services.get(parent_name) {
|
||||
@@ -498,49 +430,48 @@ impl ServicesWidget {
|
||||
|
||||
for (i, (sub_name, sub_info)) in sorted_subs.iter().enumerate() {
|
||||
let is_last_sub = i == sorted_subs.len() - 1;
|
||||
let full_sub_name = format!("{}_{}", parent_name, sub_name);
|
||||
// Store sub-service info for custom span rendering
|
||||
display_lines.push((
|
||||
sub_name.clone(),
|
||||
sub_info.widget_status,
|
||||
true,
|
||||
Some((sub_info.clone(), is_last_sub)),
|
||||
full_sub_name, // Raw service name for pending transition lookup
|
||||
)); // true = sub-service, with is_last info
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Apply scroll offset and render visible lines (same as existing logic)
|
||||
// Show only what fits, with "X more below" if needed
|
||||
let available_lines = area.height as usize;
|
||||
let total_lines = display_lines.len();
|
||||
|
||||
// Calculate scroll boundaries
|
||||
let max_scroll = if total_lines > available_lines {
|
||||
total_lines - available_lines
|
||||
// Reserve one line for "X more below" if needed
|
||||
let lines_for_content = if total_lines > available_lines {
|
||||
available_lines.saturating_sub(1)
|
||||
} else {
|
||||
total_lines.saturating_sub(1)
|
||||
available_lines
|
||||
};
|
||||
let effective_scroll = scroll_offset.min(max_scroll);
|
||||
|
||||
// Get visible lines after scrolling
|
||||
let visible_lines: Vec<_> = display_lines
|
||||
.iter()
|
||||
.skip(effective_scroll)
|
||||
.take(available_lines)
|
||||
.take(lines_for_content)
|
||||
.collect();
|
||||
|
||||
let hidden_below = total_lines.saturating_sub(lines_for_content);
|
||||
|
||||
let lines_to_show = visible_lines.len();
|
||||
|
||||
if lines_to_show > 0 {
|
||||
// Add space for "X more below" message if needed
|
||||
let total_chunks_needed = if hidden_below > 0 { lines_to_show + 1 } else { lines_to_show };
|
||||
let service_chunks = Layout::default()
|
||||
.direction(Direction::Vertical)
|
||||
.constraints(vec![Constraint::Length(1); lines_to_show])
|
||||
.constraints(vec![Constraint::Length(1); total_chunks_needed])
|
||||
.split(area);
|
||||
|
||||
for (i, (line_text, line_status, is_sub, sub_info, raw_service_name)) in visible_lines.iter().enumerate()
|
||||
for (i, (line_text, line_status, is_sub, sub_info)) in visible_lines.iter().enumerate()
|
||||
{
|
||||
let actual_index = effective_scroll + i; // Real index in the full list
|
||||
let actual_index = i; // Simple index since we're not scrolling
|
||||
|
||||
// Only parent services can be selected - calculate parent service index
|
||||
let is_selected = if !*is_sub {
|
||||
@@ -552,46 +483,23 @@ impl ServicesWidget {
|
||||
};
|
||||
|
||||
let mut spans = if *is_sub && sub_info.is_some() {
|
||||
// Use custom sub-service span creation WITH pending transitions
|
||||
// Use custom sub-service span creation
|
||||
let (service_info, is_last) = sub_info.as_ref().unwrap();
|
||||
self.create_sub_service_spans_with_transitions(line_text, service_info, *is_last, pending_transitions)
|
||||
self.create_sub_service_spans(line_text, service_info, *is_last)
|
||||
} else {
|
||||
// Parent services - check if this parent service has a pending transition using RAW service name
|
||||
if pending_transitions.contains_key(raw_service_name) {
|
||||
// Create spans with transitional status
|
||||
let (icon, status_text, _) = self.get_service_icon_and_status(raw_service_name, &ServiceInfo {
|
||||
status: "".to_string(),
|
||||
memory_mb: None,
|
||||
disk_gb: None,
|
||||
latency_ms: None,
|
||||
widget_status: *line_status
|
||||
}, pending_transitions);
|
||||
|
||||
// Use blue for transitional icons when not selected, background color when selected
|
||||
let icon_color = if is_selected && !*is_sub && is_focused {
|
||||
Theme::background() // Dark background color for visibility against blue selection
|
||||
} else {
|
||||
Theme::highlight() // Blue for normal case
|
||||
};
|
||||
|
||||
vec![
|
||||
ratatui::text::Span::styled(format!("{} ", icon), Style::default().fg(icon_color)),
|
||||
ratatui::text::Span::styled(line_text.clone(), Style::default().fg(Theme::primary_text())),
|
||||
ratatui::text::Span::styled(format!(" {}", status_text), Style::default().fg(icon_color)),
|
||||
]
|
||||
} else {
|
||||
StatusIcons::create_status_spans(*line_status, line_text)
|
||||
}
|
||||
// Parent services - use normal status spans
|
||||
StatusIcons::create_status_spans(*line_status, line_text)
|
||||
};
|
||||
|
||||
// Apply selection highlighting to parent services only, preserving status icon color
|
||||
// Apply selection highlighting to parent services only
|
||||
// Only show selection when Services panel is focused
|
||||
// Show selection highlighting even when transitional icons are present
|
||||
if is_selected && !*is_sub && is_focused {
|
||||
for (i, span) in spans.iter_mut().enumerate() {
|
||||
if i == 0 {
|
||||
// First span is the status icon - preserve its color
|
||||
span.style = span.style.bg(Theme::highlight());
|
||||
// First span is the status icon - use background color for visibility against blue selection
|
||||
span.style = span.style
|
||||
.bg(Theme::highlight())
|
||||
.fg(Theme::background());
|
||||
} else {
|
||||
// Other spans (text) get full selection highlighting
|
||||
span.style = span.style
|
||||
@@ -605,33 +513,12 @@ impl ServicesWidget {
|
||||
|
||||
frame.render_widget(service_para, service_chunks[i]);
|
||||
}
|
||||
}
|
||||
|
||||
// Show scroll indicator if there are more services than we can display (same as existing)
|
||||
if total_lines > available_lines {
|
||||
let hidden_above = effective_scroll;
|
||||
let hidden_below = total_lines.saturating_sub(effective_scroll + available_lines);
|
||||
|
||||
if hidden_above > 0 || hidden_below > 0 {
|
||||
let scroll_text = if hidden_above > 0 && hidden_below > 0 {
|
||||
format!("... {} above, {} below", hidden_above, hidden_below)
|
||||
} else if hidden_above > 0 {
|
||||
format!("... {} more above", hidden_above)
|
||||
} else {
|
||||
format!("... {} more below", hidden_below)
|
||||
};
|
||||
|
||||
if available_lines > 0 && lines_to_show > 0 {
|
||||
let last_line_area = Rect {
|
||||
x: area.x,
|
||||
y: area.y + (lines_to_show - 1) as u16,
|
||||
width: area.width,
|
||||
height: 1,
|
||||
};
|
||||
|
||||
let scroll_para = Paragraph::new(scroll_text).style(Typography::muted());
|
||||
frame.render_widget(scroll_para, last_line_area);
|
||||
}
|
||||
// Show "X more below" message if content was truncated
|
||||
if hidden_below > 0 {
|
||||
let more_text = format!("... {} more below", hidden_below);
|
||||
let more_para = Paragraph::new(more_text).style(Typography::muted());
|
||||
frame.render_widget(more_para, service_chunks[lines_to_show]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -45,12 +45,15 @@ pub struct SystemWidget {
|
||||
struct StoragePool {
|
||||
name: String,
|
||||
mount_point: String,
|
||||
pool_type: String, // "Single", "Raid0", etc.
|
||||
pool_type: String, // "single", "mergerfs (2+1)", "RAID5 (3+1)", etc.
|
||||
pool_health: Option<String>, // "healthy", "degraded", "critical", "rebuilding"
|
||||
drives: Vec<StorageDrive>,
|
||||
filesystems: Vec<FileSystem>, // For physical drive pools: individual filesystem children
|
||||
usage_percent: Option<f32>,
|
||||
used_gb: Option<f32>,
|
||||
total_gb: Option<f32>,
|
||||
status: Status,
|
||||
health_status: Status, // Separate status for pool health vs usage
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
@@ -61,6 +64,16 @@ struct StorageDrive {
|
||||
status: Status,
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
struct FileSystem {
|
||||
mount_point: String,
|
||||
usage_percent: Option<f32>,
|
||||
used_gb: Option<f32>,
|
||||
total_gb: Option<f32>,
|
||||
available_gb: Option<f32>,
|
||||
status: Status,
|
||||
}
|
||||
|
||||
impl SystemWidget {
|
||||
pub fn new() -> Self {
|
||||
Self {
|
||||
@@ -133,14 +146,14 @@ impl SystemWidget {
|
||||
self.agent_hash.as_ref()
|
||||
}
|
||||
|
||||
/// Get mount point for a pool name
|
||||
/// Get default mount point for a pool name (fallback only - should use actual mount_point metrics)
|
||||
fn get_mount_point_for_pool(&self, pool_name: &str) -> String {
|
||||
match pool_name {
|
||||
"root" => "/".to_string(),
|
||||
"steampool" => "/mnt/steampool".to_string(),
|
||||
"steampool_1" => "/steampool_1".to_string(),
|
||||
"steampool_2" => "/steampool_2".to_string(),
|
||||
_ => format!("/{}", pool_name), // Default fallback
|
||||
// For device names, use the device name directly as display name
|
||||
if pool_name.starts_with("nvme") || pool_name.starts_with("sd") || pool_name.starts_with("hd") {
|
||||
pool_name.to_string()
|
||||
} else {
|
||||
// For other pools, use the pool name as-is (will be overridden by mount_point metric)
|
||||
pool_name.to_string()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -151,32 +164,57 @@ impl SystemWidget {
|
||||
for metric in metrics {
|
||||
if metric.name.starts_with("disk_") {
|
||||
if let Some(pool_name) = self.extract_pool_name(&metric.name) {
|
||||
let mount_point = self.get_mount_point_for_pool(&pool_name);
|
||||
let pool = pools.entry(pool_name.clone()).or_insert_with(|| StoragePool {
|
||||
name: pool_name.clone(),
|
||||
mount_point: mount_point.clone(),
|
||||
pool_type: "Single".to_string(), // Default, could be enhanced
|
||||
mount_point: self.get_mount_point_for_pool(&pool_name), // Default fallback
|
||||
pool_type: "single".to_string(), // Default, will be updated
|
||||
pool_health: None,
|
||||
drives: Vec::new(),
|
||||
filesystems: Vec::new(),
|
||||
usage_percent: None,
|
||||
used_gb: None,
|
||||
total_gb: None,
|
||||
status: Status::Unknown,
|
||||
health_status: Status::Unknown,
|
||||
});
|
||||
|
||||
// Parse different metric types
|
||||
if metric.name.contains("_usage_percent") {
|
||||
if metric.name.contains("_usage_percent") && !metric.name.contains("_fs_") {
|
||||
// Only use drive-level metrics for pool totals, not filesystem metrics
|
||||
if let MetricValue::Float(usage) = metric.value {
|
||||
pool.usage_percent = Some(usage);
|
||||
pool.status = metric.status.clone();
|
||||
}
|
||||
} else if metric.name.contains("_used_gb") {
|
||||
} else if metric.name.contains("_used_gb") && !metric.name.contains("_fs_") {
|
||||
// Only use drive-level metrics for pool totals, not filesystem metrics
|
||||
if let MetricValue::Float(used) = metric.value {
|
||||
pool.used_gb = Some(used);
|
||||
}
|
||||
} else if metric.name.contains("_total_gb") {
|
||||
} else if metric.name.contains("_total_gb") && !metric.name.contains("_fs_") {
|
||||
// Only use drive-level metrics for pool totals, not filesystem metrics
|
||||
if let MetricValue::Float(total) = metric.value {
|
||||
pool.total_gb = Some(total);
|
||||
}
|
||||
} else if metric.name.contains("_mount_point") {
|
||||
if let MetricValue::String(mount_point) = &metric.value {
|
||||
pool.mount_point = mount_point.clone();
|
||||
}
|
||||
} else if metric.name.contains("_pool_type") {
|
||||
if let MetricValue::String(pool_type) = &metric.value {
|
||||
pool.pool_type = pool_type.clone();
|
||||
}
|
||||
} else if metric.name.contains("_pool_health") {
|
||||
if let MetricValue::String(health) = &metric.value {
|
||||
pool.pool_health = Some(health.clone());
|
||||
pool.health_status = metric.status.clone();
|
||||
}
|
||||
} else if metric.name.contains("_health") && !metric.name.contains("_pool_health") {
|
||||
// Handle physical drive health metrics (disk_{drive}_health)
|
||||
if let MetricValue::String(health) = &metric.value {
|
||||
// For physical drives, use the drive health as pool health
|
||||
pool.pool_health = Some(health.clone());
|
||||
pool.health_status = metric.status.clone();
|
||||
}
|
||||
} else if metric.name.contains("_temperature") {
|
||||
if let Some(drive_name) = self.extract_drive_name(&metric.name) {
|
||||
// Find existing drive or create new one
|
||||
@@ -194,12 +232,16 @@ impl SystemWidget {
|
||||
if let MetricValue::Float(temp) = metric.value {
|
||||
drive.temperature = Some(temp);
|
||||
drive.status = metric.status.clone();
|
||||
// For physical drives, if this is the main drive, also update pool health
|
||||
if drive.name == pool.name && pool.health_status == Status::Unknown {
|
||||
pool.health_status = metric.status.clone();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if metric.name.contains("_wear_percent") {
|
||||
if let Some(drive_name) = self.extract_drive_name(&metric.name) {
|
||||
// Find existing drive or create new one
|
||||
// For physical drives, ensure we create the drive object
|
||||
let drive_exists = pool.drives.iter().any(|d| d.name == drive_name);
|
||||
if !drive_exists {
|
||||
pool.drives.push(StorageDrive {
|
||||
@@ -214,6 +256,95 @@ impl SystemWidget {
|
||||
if let MetricValue::Float(wear) = metric.value {
|
||||
drive.wear_percent = Some(wear);
|
||||
drive.status = metric.status.clone();
|
||||
// For physical drives, if this is the main drive, also update pool health
|
||||
if drive.name == pool.name && pool.health_status == Status::Unknown {
|
||||
pool.health_status = metric.status.clone();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if metric.name.contains("_fs_") {
|
||||
// Handle filesystem metrics for physical drive pools (disk_{pool}_fs_{fs_name}_{metric})
|
||||
if let (Some(fs_name), Some(metric_type)) = self.extract_filesystem_metric(&metric.name) {
|
||||
// Find or create filesystem entry
|
||||
let fs_exists = pool.filesystems.iter().any(|fs| {
|
||||
let fs_id = if fs.mount_point == "/" {
|
||||
"root".to_string()
|
||||
} else {
|
||||
fs.mount_point.trim_start_matches('/').replace('/', "_")
|
||||
};
|
||||
fs_id == fs_name
|
||||
});
|
||||
|
||||
if !fs_exists {
|
||||
// Create filesystem entry with correct mount point
|
||||
let mount_point = if metric_type == "mount_point" {
|
||||
if let MetricValue::String(mount) = &metric.value {
|
||||
mount.clone()
|
||||
} else {
|
||||
// Fallback: handle special cases
|
||||
if fs_name == "root" {
|
||||
"/".to_string()
|
||||
} else {
|
||||
format!("/{}", fs_name.replace('_', "/"))
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Fallback for non-mount_point metrics: generate mount point from fs_name
|
||||
if fs_name == "root" {
|
||||
"/".to_string()
|
||||
} else {
|
||||
format!("/{}", fs_name.replace('_', "/"))
|
||||
}
|
||||
};
|
||||
|
||||
pool.filesystems.push(FileSystem {
|
||||
mount_point,
|
||||
usage_percent: None,
|
||||
used_gb: None,
|
||||
total_gb: None,
|
||||
available_gb: None,
|
||||
status: Status::Unknown,
|
||||
});
|
||||
}
|
||||
|
||||
// Update the filesystem with the metric value
|
||||
if let Some(filesystem) = pool.filesystems.iter_mut().find(|fs| {
|
||||
let fs_id = if fs.mount_point == "/" {
|
||||
"root".to_string()
|
||||
} else {
|
||||
fs.mount_point.trim_start_matches('/').replace('/', "_")
|
||||
};
|
||||
fs_id == fs_name
|
||||
}) {
|
||||
match metric_type.as_str() {
|
||||
"usage_percent" => {
|
||||
if let MetricValue::Float(usage) = metric.value {
|
||||
filesystem.usage_percent = Some(usage);
|
||||
filesystem.status = metric.status.clone();
|
||||
}
|
||||
}
|
||||
"used_gb" => {
|
||||
if let MetricValue::Float(used) = metric.value {
|
||||
filesystem.used_gb = Some(used);
|
||||
}
|
||||
}
|
||||
"total_gb" => {
|
||||
if let MetricValue::Float(total) = metric.value {
|
||||
filesystem.total_gb = Some(total);
|
||||
}
|
||||
}
|
||||
"available_gb" => {
|
||||
if let MetricValue::Float(available) = metric.value {
|
||||
filesystem.available_gb = Some(available);
|
||||
}
|
||||
}
|
||||
"mount_point" => {
|
||||
if let MetricValue::String(mount) = &metric.value {
|
||||
filesystem.mount_point = mount.clone();
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -230,79 +361,161 @@ impl SystemWidget {
|
||||
|
||||
/// Extract pool name from disk metric name
|
||||
fn extract_pool_name(&self, metric_name: &str) -> Option<String> {
|
||||
if let Some(captures) = metric_name.strip_prefix("disk_") {
|
||||
if let Some(pos) = captures.find('_') {
|
||||
return Some(captures[..pos].to_string());
|
||||
// Pattern: disk_{pool_name}_{various suffixes}
|
||||
// Since pool_name can contain underscores, work backwards from known metric suffixes
|
||||
if metric_name.starts_with("disk_") {
|
||||
// Handle filesystem metrics: disk_{pool}_fs_{filesystem}_{metric}
|
||||
if metric_name.contains("_fs_") {
|
||||
if let Some(fs_pos) = metric_name.find("_fs_") {
|
||||
return Some(metric_name[5..fs_pos].to_string()); // Skip "disk_", extract pool name before "_fs_"
|
||||
}
|
||||
}
|
||||
|
||||
// Handle pool-level metrics (usage_percent, used_gb, total_gb, mount_point, pool_type, pool_health)
|
||||
// Use rfind to get the last occurrence of these suffixes
|
||||
let pool_suffixes = ["_usage_percent", "_used_gb", "_total_gb", "_available_gb", "_mount_point", "_pool_type", "_pool_health"];
|
||||
for suffix in pool_suffixes {
|
||||
if let Some(suffix_pos) = metric_name.rfind(suffix) {
|
||||
return Some(metric_name[5..suffix_pos].to_string()); // Skip "disk_"
|
||||
}
|
||||
}
|
||||
|
||||
// Handle physical drive metrics: disk_{drive}_health, disk_{drive}_wear_percent, and disk_{drive}_temperature
|
||||
if (metric_name.ends_with("_health") && !metric_name.contains("_pool_health"))
|
||||
|| metric_name.ends_with("_wear_percent")
|
||||
|| metric_name.ends_with("_temperature") {
|
||||
// Count underscores to distinguish physical drive metrics (disk_{drive}_metric)
|
||||
// from pool drive metrics (disk_{pool}_{drive}_metric)
|
||||
let underscore_count = metric_name.matches('_').count();
|
||||
// disk_nvme0n1_wear_percent has 3 underscores: disk_nvme0n1_wear_percent
|
||||
if underscore_count == 3 { // disk_{drive}_metric (where drive has underscores)
|
||||
if let Some(suffix_pos) = metric_name.rfind("_health")
|
||||
.or_else(|| metric_name.rfind("_wear_percent"))
|
||||
.or_else(|| metric_name.rfind("_temperature")) {
|
||||
return Some(metric_name[5..suffix_pos].to_string()); // Skip "disk_"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Handle drive-specific metrics: disk_{pool}_{drive}_{metric}
|
||||
let drive_suffixes = ["_temperature", "_health"];
|
||||
for suffix in drive_suffixes {
|
||||
if let Some(suffix_pos) = metric_name.rfind(suffix) {
|
||||
// Extract pool name by finding the second-to-last underscore
|
||||
let before_suffix = &metric_name[..suffix_pos];
|
||||
if let Some(drive_start) = before_suffix.rfind('_') {
|
||||
if drive_start > 5 {
|
||||
return Some(metric_name[5..drive_start].to_string()); // Skip "disk_"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Extract filesystem name and metric type from filesystem metric names
|
||||
/// Pattern: disk_{pool}_fs_{filesystem_name}_{metric_type}
|
||||
fn extract_filesystem_metric(&self, metric_name: &str) -> (Option<String>, Option<String>) {
|
||||
if metric_name.starts_with("disk_") && metric_name.contains("_fs_") {
|
||||
// Find the _fs_ part
|
||||
if let Some(fs_start) = metric_name.find("_fs_") {
|
||||
let after_fs = &metric_name[fs_start + 4..]; // Skip "_fs_"
|
||||
|
||||
// Look for known metric suffixes (these can contain underscores)
|
||||
let known_suffixes = ["usage_percent", "used_gb", "total_gb", "available_gb", "mount_point"];
|
||||
|
||||
for suffix in known_suffixes {
|
||||
if after_fs.ends_with(suffix) {
|
||||
// Extract filesystem name by removing suffix and underscore
|
||||
if let Some(underscore_pos) = after_fs.rfind(&format!("_{}", suffix)) {
|
||||
let fs_name = after_fs[..underscore_pos].to_string();
|
||||
return (Some(fs_name), Some(suffix.to_string()));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
(None, None)
|
||||
}
|
||||
|
||||
/// Extract drive name from disk metric name
|
||||
fn extract_drive_name(&self, metric_name: &str) -> Option<String> {
|
||||
// Pattern: disk_pool_drive_metric
|
||||
let parts: Vec<&str> = metric_name.split('_').collect();
|
||||
if parts.len() >= 3 && parts[0] == "disk" {
|
||||
return Some(parts[2].to_string());
|
||||
// Pattern: disk_{pool_name}_{drive_name}_{metric_type} OR disk_{drive_name}_{metric_type}
|
||||
// Pool drives: disk_srv_media_sdb_temperature
|
||||
// Physical drives: disk_nvme0n1_temperature
|
||||
if metric_name.starts_with("disk_") {
|
||||
if let Some(suffix_pos) = metric_name.rfind("_temperature")
|
||||
.or_else(|| metric_name.rfind("_wear_percent"))
|
||||
.or_else(|| metric_name.rfind("_health")) {
|
||||
let before_suffix = &metric_name[..suffix_pos];
|
||||
|
||||
// Extract the last component as drive name (e.g., "sdb", "sdc", "nvme0n1")
|
||||
if let Some(drive_start) = before_suffix.rfind('_') {
|
||||
return Some(before_suffix[drive_start + 1..].to_string());
|
||||
} else {
|
||||
// Handle physical drive metrics: disk_{drive}_metric (no pool)
|
||||
// Extract everything after "disk_" as the drive name
|
||||
return Some(before_suffix[5..].to_string()); // Skip "disk_"
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// Render storage section with tree structure
|
||||
/// Render storage section with enhanced tree structure
|
||||
fn render_storage(&self) -> Vec<Line<'_>> {
|
||||
let mut lines = Vec::new();
|
||||
|
||||
for pool in &self.storage_pools {
|
||||
// Pool header line
|
||||
let usage_text = match (pool.usage_percent, pool.used_gb, pool.total_gb) {
|
||||
(Some(pct), Some(used), Some(total)) => {
|
||||
format!("{:.0}% {:.1}GB/{:.1}GB", pct, used, total)
|
||||
}
|
||||
_ => "—% —GB/—GB".to_string(),
|
||||
};
|
||||
// Pool header line with type and health
|
||||
let pool_label = if pool.pool_type.starts_with("drive (") {
|
||||
// For physical drives, show the drive name with temperature and wear percentage if available
|
||||
// Look for any drive with temp/wear data (physical drives may have drives named after the pool)
|
||||
let temp_opt = pool.drives.iter()
|
||||
.find_map(|d| d.temperature);
|
||||
let wear_opt = pool.drives.iter()
|
||||
.find_map(|d| d.wear_percent);
|
||||
|
||||
let pool_label = if pool.pool_type.to_lowercase() == "single" {
|
||||
let mut drive_info = Vec::new();
|
||||
if let Some(temp) = temp_opt {
|
||||
drive_info.push(format!("T: {:.0}°C", temp));
|
||||
}
|
||||
if let Some(wear) = wear_opt {
|
||||
drive_info.push(format!("W: {:.0}%", wear));
|
||||
}
|
||||
|
||||
if drive_info.is_empty() {
|
||||
format!("{}:", pool.name)
|
||||
} else {
|
||||
format!("{} {}:", pool.name, drive_info.join(" "))
|
||||
}
|
||||
} else if pool.pool_type == "single" {
|
||||
format!("{}:", pool.mount_point)
|
||||
} else {
|
||||
format!("{} ({}):", pool.mount_point, pool.pool_type)
|
||||
};
|
||||
let pool_spans = StatusIcons::create_status_spans(
|
||||
pool.status.clone(),
|
||||
pool.health_status.clone(),
|
||||
&pool_label
|
||||
);
|
||||
lines.push(Line::from(pool_spans));
|
||||
|
||||
// Drive lines with tree structure
|
||||
let has_usage_line = pool.usage_percent.is_some();
|
||||
for (i, drive) in pool.drives.iter().enumerate() {
|
||||
let is_last_drive = i == pool.drives.len() - 1;
|
||||
let tree_symbol = if is_last_drive && !has_usage_line { "└─" } else { "├─" };
|
||||
// Skip pool health line as discussed - removed
|
||||
|
||||
let mut drive_info = Vec::new();
|
||||
if let Some(temp) = drive.temperature {
|
||||
drive_info.push(format!("T: {:.0}C", temp));
|
||||
}
|
||||
if let Some(wear) = drive.wear_percent {
|
||||
drive_info.push(format!("W: {:.0}%", wear));
|
||||
}
|
||||
let drive_text = if drive_info.is_empty() {
|
||||
drive.name.clone()
|
||||
} else {
|
||||
format!("{} {}", drive.name, drive_info.join(" • "))
|
||||
// Total usage line (only show for multi-drive pools, skip for single physical drives)
|
||||
if !pool.pool_type.starts_with("drive (") {
|
||||
let usage_text = match (pool.usage_percent, pool.used_gb, pool.total_gb) {
|
||||
(Some(pct), Some(used), Some(total)) => {
|
||||
format!("Total: {:.0}% {:.1}GB/{:.1}GB", pct, used, total)
|
||||
}
|
||||
_ => "Total: —% —GB/—GB".to_string(),
|
||||
};
|
||||
|
||||
let mut drive_spans = vec![
|
||||
Span::raw(" "),
|
||||
Span::styled(tree_symbol, Typography::tree()),
|
||||
Span::raw(" "),
|
||||
];
|
||||
drive_spans.extend(StatusIcons::create_status_spans(drive.status.clone(), &drive_text));
|
||||
lines.push(Line::from(drive_spans));
|
||||
}
|
||||
|
||||
// Usage line
|
||||
if pool.usage_percent.is_some() {
|
||||
let tree_symbol = "└─";
|
||||
let has_drives = !pool.drives.is_empty();
|
||||
let has_filesystems = !pool.filesystems.is_empty();
|
||||
let has_children = has_drives || has_filesystems;
|
||||
let tree_symbol = if has_children { "├─" } else { "└─" };
|
||||
let mut usage_spans = vec![
|
||||
Span::raw(" "),
|
||||
Span::styled(tree_symbol, Typography::tree()),
|
||||
@@ -311,10 +524,126 @@ impl SystemWidget {
|
||||
usage_spans.extend(StatusIcons::create_status_spans(pool.status.clone(), &usage_text));
|
||||
lines.push(Line::from(usage_spans));
|
||||
}
|
||||
|
||||
// Drive lines with enhanced grouping
|
||||
if pool.pool_type.contains("mergerfs") && pool.drives.len() > 1 {
|
||||
// Group drives by type for mergerfs pools
|
||||
let (data_drives, parity_drives): (Vec<_>, Vec<_>) = pool.drives.iter().enumerate()
|
||||
.partition(|(_, drive)| {
|
||||
// Simple heuristic: drives with 'parity' in name or sdc (common parity drive)
|
||||
!drive.name.to_lowercase().contains("parity") && drive.name != "sdc"
|
||||
});
|
||||
|
||||
// Show data drives
|
||||
if !data_drives.is_empty() {
|
||||
lines.push(Line::from(vec![
|
||||
Span::raw(" "),
|
||||
Span::styled("├─ ", Typography::tree()),
|
||||
Span::styled("Data Disks:", Typography::secondary()),
|
||||
]));
|
||||
|
||||
for (i, (_, drive)) in data_drives.iter().enumerate() {
|
||||
let is_last = i == data_drives.len() - 1;
|
||||
if is_last && parity_drives.is_empty() {
|
||||
self.render_drive_line(&mut lines, drive, "│ └─");
|
||||
} else {
|
||||
self.render_drive_line(&mut lines, drive, "│ ├─");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Show parity drives
|
||||
if !parity_drives.is_empty() {
|
||||
lines.push(Line::from(vec![
|
||||
Span::raw(" "),
|
||||
Span::styled("└─ ", Typography::tree()),
|
||||
Span::styled("Parity:", Typography::secondary()),
|
||||
]));
|
||||
|
||||
for (i, (_, drive)) in parity_drives.iter().enumerate() {
|
||||
let is_last = i == parity_drives.len() - 1;
|
||||
if is_last {
|
||||
self.render_drive_line(&mut lines, drive, " └─");
|
||||
} else {
|
||||
self.render_drive_line(&mut lines, drive, " ├─");
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if pool.pool_type != "single" && pool.drives.len() > 1 {
|
||||
// Regular drive listing for non-mergerfs multi-drive pools
|
||||
for (i, drive) in pool.drives.iter().enumerate() {
|
||||
let is_last = i == pool.drives.len() - 1;
|
||||
let tree_symbol = if is_last { "└─" } else { "├─" };
|
||||
self.render_drive_line(&mut lines, drive, tree_symbol);
|
||||
}
|
||||
} else if pool.pool_type.starts_with("drive (") {
|
||||
// Physical drive pools: wear data shown in header, skip drive lines, show filesystems directly
|
||||
for (i, filesystem) in pool.filesystems.iter().enumerate() {
|
||||
let is_last = i == pool.filesystems.len() - 1;
|
||||
let tree_symbol = if is_last { "└─" } else { "├─" };
|
||||
|
||||
let fs_text = match (filesystem.usage_percent, filesystem.used_gb, filesystem.total_gb) {
|
||||
(Some(pct), Some(used), Some(total)) => {
|
||||
format!("{}: {:.0}% {:.1}GB/{:.1}GB", filesystem.mount_point, pct, used, total)
|
||||
}
|
||||
(Some(pct), _, Some(total)) => {
|
||||
format!("{}: {:.0}% —GB/{:.1}GB", filesystem.mount_point, pct, total)
|
||||
}
|
||||
(Some(pct), _, _) => {
|
||||
format!("{}: {:.0}% —GB/—GB", filesystem.mount_point, pct)
|
||||
}
|
||||
(_, Some(used), Some(total)) => {
|
||||
format!("{}: —% {:.1}GB/{:.1}GB", filesystem.mount_point, used, total)
|
||||
}
|
||||
_ => format!("{}: —% —GB/—GB", filesystem.mount_point),
|
||||
};
|
||||
|
||||
let mut fs_spans = vec![
|
||||
Span::raw(" "),
|
||||
Span::styled(tree_symbol, Typography::tree()),
|
||||
Span::raw(" "),
|
||||
];
|
||||
fs_spans.extend(StatusIcons::create_status_spans(filesystem.status.clone(), &fs_text));
|
||||
lines.push(Line::from(fs_spans));
|
||||
}
|
||||
} else {
|
||||
// Single drive or simple pools
|
||||
for (i, drive) in pool.drives.iter().enumerate() {
|
||||
let is_last = i == pool.drives.len() - 1;
|
||||
let tree_symbol = if is_last { "└─" } else { "├─" };
|
||||
self.render_drive_line(&mut lines, drive, tree_symbol);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
lines
|
||||
}
|
||||
|
||||
/// Helper to render a single drive line
|
||||
fn render_drive_line<'a>(&self, lines: &mut Vec<Line<'a>>, drive: &StorageDrive, tree_symbol: &'a str) {
|
||||
let mut drive_info = Vec::new();
|
||||
if let Some(temp) = drive.temperature {
|
||||
drive_info.push(format!("T: {:.0}°C", temp));
|
||||
}
|
||||
if let Some(wear) = drive.wear_percent {
|
||||
drive_info.push(format!("W: {:.0}%", wear));
|
||||
}
|
||||
|
||||
// Always show drive name with info, or just name if no info available
|
||||
let drive_text = if drive_info.is_empty() {
|
||||
drive.name.clone()
|
||||
} else {
|
||||
format!("{} {}", drive.name, drive_info.join(" "))
|
||||
};
|
||||
|
||||
let mut drive_spans = vec![
|
||||
Span::raw(" "),
|
||||
Span::styled(tree_symbol, Typography::tree()),
|
||||
Span::raw(" "),
|
||||
];
|
||||
drive_spans.extend(StatusIcons::create_status_spans(drive.status.clone(), &drive_text));
|
||||
lines.push(Line::from(drive_spans));
|
||||
}
|
||||
}
|
||||
|
||||
impl Widget for SystemWidget {
|
||||
@@ -409,13 +738,13 @@ impl Widget for SystemWidget {
|
||||
}
|
||||
|
||||
impl SystemWidget {
|
||||
/// Render with scroll offset support
|
||||
pub fn render_with_scroll(&mut self, frame: &mut Frame, area: Rect, scroll_offset: usize) {
|
||||
/// Render system widget
|
||||
pub fn render(&mut self, frame: &mut Frame, area: Rect, hostname: &str, config: Option<&crate::config::DashboardConfig>) {
|
||||
let mut lines = Vec::new();
|
||||
|
||||
// NixOS section
|
||||
lines.push(Line::from(vec![
|
||||
Span::styled("NixOS:", Typography::widget_title())
|
||||
Span::styled(format!("NixOS {}:", hostname), Typography::widget_title())
|
||||
]));
|
||||
|
||||
let build_text = self.nixos_build.as_deref().unwrap_or("unknown");
|
||||
@@ -428,6 +757,16 @@ impl SystemWidget {
|
||||
Span::styled(format!("Agent: {}", agent_version_text), Typography::secondary())
|
||||
]));
|
||||
|
||||
// Display detected connection IP
|
||||
if let Some(config) = config {
|
||||
if let Some(host_details) = config.hosts.get(hostname) {
|
||||
let detected_ip = host_details.get_connection_ip(hostname);
|
||||
lines.push(Line::from(vec![
|
||||
Span::styled(format!("IP: {}", detected_ip), Typography::secondary())
|
||||
]));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// CPU section
|
||||
lines.push(Line::from(vec![
|
||||
@@ -474,69 +813,30 @@ impl SystemWidget {
|
||||
Span::styled("Storage:", Typography::widget_title())
|
||||
]));
|
||||
|
||||
// Storage items with overflow handling
|
||||
// Storage items - let main overflow logic handle truncation
|
||||
let storage_lines = self.render_storage();
|
||||
let remaining_space = area.height.saturating_sub(lines.len() as u16);
|
||||
|
||||
if storage_lines.len() <= remaining_space as usize {
|
||||
// All storage lines fit
|
||||
lines.extend(storage_lines);
|
||||
} else if remaining_space >= 2 {
|
||||
// Show what we can and add overflow indicator
|
||||
let lines_to_show = (remaining_space - 1) as usize; // Reserve 1 line for overflow
|
||||
lines.extend(storage_lines.iter().take(lines_to_show).cloned());
|
||||
|
||||
// Count hidden pools
|
||||
let mut hidden_pools = 0;
|
||||
let mut current_pool = String::new();
|
||||
for (i, line) in storage_lines.iter().enumerate() {
|
||||
if i >= lines_to_show {
|
||||
// Check if this line represents a new pool (no indentation)
|
||||
if let Some(first_span) = line.spans.first() {
|
||||
let text = first_span.content.as_ref();
|
||||
if !text.starts_with(" ") && text.contains(':') {
|
||||
let pool_name = text.split(':').next().unwrap_or("").trim();
|
||||
if pool_name != current_pool {
|
||||
hidden_pools += 1;
|
||||
current_pool = pool_name.to_string();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if hidden_pools > 0 {
|
||||
let overflow_text = format!(
|
||||
"... and {} more pool{}",
|
||||
hidden_pools,
|
||||
if hidden_pools == 1 { "" } else { "s" }
|
||||
);
|
||||
lines.push(Line::from(vec![
|
||||
Span::styled(overflow_text, Typography::muted())
|
||||
]));
|
||||
}
|
||||
}
|
||||
lines.extend(storage_lines);
|
||||
|
||||
// Apply scroll offset
|
||||
let total_lines = lines.len();
|
||||
let available_height = area.height as usize;
|
||||
|
||||
// Always apply scrolling if scroll_offset > 0, even if content fits
|
||||
if scroll_offset > 0 || total_lines > available_height {
|
||||
let max_scroll = if total_lines > available_height {
|
||||
total_lines - available_height
|
||||
} else {
|
||||
total_lines.saturating_sub(1)
|
||||
};
|
||||
let effective_scroll = scroll_offset.min(max_scroll);
|
||||
|
||||
// Take only the visible portion after scrolling
|
||||
let visible_lines: Vec<Line> = lines
|
||||
// Show only what fits, with "X more below" if needed
|
||||
if total_lines > available_height {
|
||||
let lines_for_content = available_height.saturating_sub(1); // Reserve one line for "more below"
|
||||
let mut visible_lines: Vec<Line> = lines
|
||||
.into_iter()
|
||||
.skip(effective_scroll)
|
||||
.take(available_height)
|
||||
.take(lines_for_content)
|
||||
.collect();
|
||||
|
||||
let hidden_below = total_lines.saturating_sub(lines_for_content);
|
||||
if hidden_below > 0 {
|
||||
let more_line = Line::from(vec![
|
||||
Span::styled(format!("... {} more below", hidden_below), Typography::muted())
|
||||
]);
|
||||
visible_lines.push(more_line);
|
||||
}
|
||||
|
||||
let paragraph = Paragraph::new(Text::from(visible_lines));
|
||||
frame.render_widget(paragraph, area);
|
||||
} else {
|
||||
|
||||
@@ -1,88 +0,0 @@
|
||||
# Hardcoded Values Removed - Configuration Summary
|
||||
|
||||
## ✅ All Hardcoded Values Converted to Configuration
|
||||
|
||||
### **1. SystemD Nginx Check Interval**
|
||||
- **Before**: `nginx_check_interval_seconds: 30` (hardcoded)
|
||||
- **After**: `nginx_check_interval_seconds: config.nginx_check_interval_seconds`
|
||||
- **NixOS Config**: `nginx_check_interval_seconds = 30;`
|
||||
|
||||
### **2. ZMQ Transmission Interval**
|
||||
- **Before**: `Duration::from_secs(1)` (hardcoded)
|
||||
- **After**: `Duration::from_secs(self.config.zmq.transmission_interval_seconds)`
|
||||
- **NixOS Config**: `transmission_interval_seconds = 1;`
|
||||
|
||||
### **3. HTTP Timeouts in SystemD Collector**
|
||||
- **Before**:
|
||||
```rust
|
||||
.timeout(Duration::from_secs(10))
|
||||
.connect_timeout(Duration::from_secs(10))
|
||||
```
|
||||
- **After**:
|
||||
```rust
|
||||
.timeout(Duration::from_secs(self.config.http_timeout_seconds))
|
||||
.connect_timeout(Duration::from_secs(self.config.http_connect_timeout_seconds))
|
||||
```
|
||||
- **NixOS Config**:
|
||||
```nix
|
||||
http_timeout_seconds = 10;
|
||||
http_connect_timeout_seconds = 10;
|
||||
```
|
||||
|
||||
## **Configuration Structure Changes**
|
||||
|
||||
### **SystemdConfig** (agent/src/config/mod.rs)
|
||||
```rust
|
||||
pub struct SystemdConfig {
|
||||
// ... existing fields ...
|
||||
pub nginx_check_interval_seconds: u64, // NEW
|
||||
pub http_timeout_seconds: u64, // NEW
|
||||
pub http_connect_timeout_seconds: u64, // NEW
|
||||
}
|
||||
```
|
||||
|
||||
### **ZmqConfig** (agent/src/config/mod.rs)
|
||||
```rust
|
||||
pub struct ZmqConfig {
|
||||
// ... existing fields ...
|
||||
pub transmission_interval_seconds: u64, // NEW
|
||||
}
|
||||
```
|
||||
|
||||
## **NixOS Configuration Updates**
|
||||
|
||||
### **ZMQ Section** (hosts/common/cm-dashboard.nix)
|
||||
```nix
|
||||
zmq = {
|
||||
# ... existing fields ...
|
||||
transmission_interval_seconds = 1; # NEW
|
||||
};
|
||||
```
|
||||
|
||||
### **SystemD Section** (hosts/common/cm-dashboard.nix)
|
||||
```nix
|
||||
systemd = {
|
||||
# ... existing fields ...
|
||||
nginx_check_interval_seconds = 30; # NEW
|
||||
http_timeout_seconds = 10; # NEW
|
||||
http_connect_timeout_seconds = 10; # NEW
|
||||
};
|
||||
```
|
||||
|
||||
## **Benefits**
|
||||
|
||||
✅ **No hardcoded values** - All timing/timeout values configurable
|
||||
✅ **Consistent configuration** - Everything follows NixOS config pattern
|
||||
✅ **Environment-specific tuning** - Can adjust timeouts per deployment
|
||||
✅ **Maintainability** - No magic numbers scattered in code
|
||||
✅ **Testing flexibility** - Can configure different values for testing
|
||||
|
||||
## **Runtime Behavior**
|
||||
|
||||
All previously hardcoded values now respect configuration:
|
||||
- **Nginx latency checks**: Every 30s (configurable)
|
||||
- **ZMQ transmission**: Every 1s (configurable)
|
||||
- **HTTP requests**: 10s timeout (configurable)
|
||||
- **HTTP connections**: 10s timeout (configurable)
|
||||
|
||||
The codebase is now **100% configuration-driven** with no hardcoded timing values.
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "cm-dashboard-shared"
|
||||
version = "0.1.29"
|
||||
version = "0.1.131"
|
||||
edition = "2021"
|
||||
|
||||
[dependencies]
|
||||
|
||||
161
shared/src/agent_data.rs
Normal file
161
shared/src/agent_data.rs
Normal file
@@ -0,0 +1,161 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Complete structured data from an agent
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct AgentData {
|
||||
pub hostname: String,
|
||||
pub agent_version: String,
|
||||
pub timestamp: u64,
|
||||
pub system: SystemData,
|
||||
pub services: Vec<ServiceData>,
|
||||
pub backup: BackupData,
|
||||
}
|
||||
|
||||
/// System-level monitoring data
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct SystemData {
|
||||
pub cpu: CpuData,
|
||||
pub memory: MemoryData,
|
||||
pub storage: StorageData,
|
||||
}
|
||||
|
||||
/// CPU monitoring data
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct CpuData {
|
||||
pub load_1min: f32,
|
||||
pub load_5min: f32,
|
||||
pub load_15min: f32,
|
||||
pub frequency_mhz: f32,
|
||||
pub temperature_celsius: Option<f32>,
|
||||
}
|
||||
|
||||
/// Memory monitoring data
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct MemoryData {
|
||||
pub usage_percent: f32,
|
||||
pub total_gb: f32,
|
||||
pub used_gb: f32,
|
||||
pub available_gb: f32,
|
||||
pub swap_total_gb: f32,
|
||||
pub swap_used_gb: f32,
|
||||
pub tmpfs: Vec<TmpfsData>,
|
||||
}
|
||||
|
||||
/// Tmpfs filesystem data
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct TmpfsData {
|
||||
pub mount: String,
|
||||
pub usage_percent: f32,
|
||||
pub used_gb: f32,
|
||||
pub total_gb: f32,
|
||||
}
|
||||
|
||||
/// Storage monitoring data
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct StorageData {
|
||||
pub drives: Vec<DriveData>,
|
||||
pub pools: Vec<PoolData>,
|
||||
}
|
||||
|
||||
/// Individual drive data
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct DriveData {
|
||||
pub name: String,
|
||||
pub health: String,
|
||||
pub temperature_celsius: Option<f32>,
|
||||
pub wear_percent: Option<f32>,
|
||||
pub filesystems: Vec<FilesystemData>,
|
||||
}
|
||||
|
||||
/// Filesystem on a drive
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct FilesystemData {
|
||||
pub mount: String,
|
||||
pub usage_percent: f32,
|
||||
pub used_gb: f32,
|
||||
pub total_gb: f32,
|
||||
}
|
||||
|
||||
/// Storage pool (MergerFS, RAID, etc.)
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct PoolData {
|
||||
pub name: String,
|
||||
pub mount: String,
|
||||
pub pool_type: String, // "mergerfs", "raid", etc.
|
||||
pub health: String,
|
||||
pub usage_percent: f32,
|
||||
pub used_gb: f32,
|
||||
pub total_gb: f32,
|
||||
pub data_drives: Vec<PoolDriveData>,
|
||||
pub parity_drives: Vec<PoolDriveData>,
|
||||
}
|
||||
|
||||
/// Drive in a storage pool
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct PoolDriveData {
|
||||
pub name: String,
|
||||
pub temperature_celsius: Option<f32>,
|
||||
pub wear_percent: Option<f32>,
|
||||
pub health: String,
|
||||
}
|
||||
|
||||
/// Service monitoring data
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ServiceData {
|
||||
pub name: String,
|
||||
pub status: String, // "active", "inactive", "failed"
|
||||
pub memory_mb: f32,
|
||||
pub disk_gb: f32,
|
||||
pub user_stopped: bool,
|
||||
}
|
||||
|
||||
/// Backup system data
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct BackupData {
|
||||
pub status: String,
|
||||
pub last_run: Option<u64>,
|
||||
pub next_scheduled: Option<u64>,
|
||||
pub total_size_gb: Option<f32>,
|
||||
pub repository_health: Option<String>,
|
||||
}
|
||||
|
||||
impl AgentData {
|
||||
/// Create new agent data with current timestamp
|
||||
pub fn new(hostname: String, agent_version: String) -> Self {
|
||||
Self {
|
||||
hostname,
|
||||
agent_version,
|
||||
timestamp: chrono::Utc::now().timestamp() as u64,
|
||||
system: SystemData {
|
||||
cpu: CpuData {
|
||||
load_1min: 0.0,
|
||||
load_5min: 0.0,
|
||||
load_15min: 0.0,
|
||||
frequency_mhz: 0.0,
|
||||
temperature_celsius: None,
|
||||
},
|
||||
memory: MemoryData {
|
||||
usage_percent: 0.0,
|
||||
total_gb: 0.0,
|
||||
used_gb: 0.0,
|
||||
available_gb: 0.0,
|
||||
swap_total_gb: 0.0,
|
||||
swap_used_gb: 0.0,
|
||||
tmpfs: Vec::new(),
|
||||
},
|
||||
storage: StorageData {
|
||||
drives: Vec::new(),
|
||||
pools: Vec::new(),
|
||||
},
|
||||
},
|
||||
services: Vec::new(),
|
||||
backup: BackupData {
|
||||
status: "unknown".to_string(),
|
||||
last_run: None,
|
||||
next_scheduled: None,
|
||||
total_size_gb: None,
|
||||
repository_health: None,
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,8 +1,10 @@
|
||||
pub mod agent_data;
|
||||
pub mod cache;
|
||||
pub mod error;
|
||||
pub mod metrics;
|
||||
pub mod protocol;
|
||||
|
||||
pub use agent_data::*;
|
||||
pub use cache::*;
|
||||
pub use error::*;
|
||||
pub use metrics::*;
|
||||
|
||||
@@ -82,11 +82,13 @@ impl MetricValue {
|
||||
/// Health status for metrics
|
||||
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord)]
|
||||
pub enum Status {
|
||||
Ok,
|
||||
Pending,
|
||||
Warning,
|
||||
Critical,
|
||||
Unknown,
|
||||
Inactive, // Lowest priority
|
||||
Unknown, //
|
||||
Offline, //
|
||||
Pending, //
|
||||
Ok, // 5th place - good status has higher priority than unknown states
|
||||
Warning, //
|
||||
Critical, // Highest priority
|
||||
}
|
||||
|
||||
impl Status {
|
||||
@@ -180,6 +182,16 @@ impl HysteresisThresholds {
|
||||
Status::Ok
|
||||
}
|
||||
}
|
||||
Status::Inactive => {
|
||||
// Inactive services use normal thresholds like first measurement
|
||||
if value >= self.critical_high {
|
||||
Status::Critical
|
||||
} else if value >= self.warning_high {
|
||||
Status::Warning
|
||||
} else {
|
||||
Status::Ok
|
||||
}
|
||||
}
|
||||
Status::Pending => {
|
||||
// Service transitioning, use normal thresholds like first measurement
|
||||
if value >= self.critical_high {
|
||||
@@ -190,6 +202,16 @@ impl HysteresisThresholds {
|
||||
Status::Ok
|
||||
}
|
||||
}
|
||||
Status::Offline => {
|
||||
// Host coming back online, use normal thresholds like first measurement
|
||||
if value >= self.critical_high {
|
||||
Status::Critical
|
||||
} else if value >= self.warning_high {
|
||||
Status::Warning
|
||||
} else {
|
||||
Status::Ok
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,13 +1,9 @@
|
||||
use crate::metrics::Metric;
|
||||
use crate::agent_data::AgentData;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
/// Message sent from agent to dashboard via ZMQ
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct MetricMessage {
|
||||
pub hostname: String,
|
||||
pub timestamp: u64,
|
||||
pub metrics: Vec<Metric>,
|
||||
}
|
||||
/// Always structured data - no legacy metrics support
|
||||
pub type AgentMessage = AgentData;
|
||||
|
||||
/// Command output streaming message
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
@@ -20,15 +16,6 @@ pub struct CommandOutputMessage {
|
||||
pub timestamp: u64,
|
||||
}
|
||||
|
||||
impl MetricMessage {
|
||||
pub fn new(hostname: String, metrics: Vec<Metric>) -> Self {
|
||||
Self {
|
||||
hostname,
|
||||
timestamp: chrono::Utc::now().timestamp() as u64,
|
||||
metrics,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl CommandOutputMessage {
|
||||
pub fn new(hostname: String, command_id: String, command_type: String, output_line: String, is_complete: bool) -> Self {
|
||||
@@ -59,8 +46,8 @@ pub enum Command {
|
||||
pub enum CommandResponse {
|
||||
/// Acknowledgment of command
|
||||
Ack,
|
||||
/// Metrics response
|
||||
Metrics(Vec<Metric>),
|
||||
/// Agent data response
|
||||
AgentData(AgentData),
|
||||
/// Pong response to ping
|
||||
Pong,
|
||||
/// Error response
|
||||
@@ -76,7 +63,7 @@ pub struct MessageEnvelope {
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
pub enum MessageType {
|
||||
Metrics,
|
||||
AgentData,
|
||||
Command,
|
||||
CommandResponse,
|
||||
CommandOutput,
|
||||
@@ -84,10 +71,10 @@ pub enum MessageType {
|
||||
}
|
||||
|
||||
impl MessageEnvelope {
|
||||
pub fn metrics(message: MetricMessage) -> Result<Self, crate::SharedError> {
|
||||
pub fn agent_data(data: AgentData) -> Result<Self, crate::SharedError> {
|
||||
Ok(Self {
|
||||
message_type: MessageType::Metrics,
|
||||
payload: serde_json::to_vec(&message)?,
|
||||
message_type: MessageType::AgentData,
|
||||
payload: serde_json::to_vec(&data)?,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -119,11 +106,11 @@ impl MessageEnvelope {
|
||||
})
|
||||
}
|
||||
|
||||
pub fn decode_metrics(&self) -> Result<MetricMessage, crate::SharedError> {
|
||||
pub fn decode_agent_data(&self) -> Result<AgentData, crate::SharedError> {
|
||||
match self.message_type {
|
||||
MessageType::Metrics => Ok(serde_json::from_slice(&self.payload)?),
|
||||
MessageType::AgentData => Ok(serde_json::from_slice(&self.payload)?),
|
||||
_ => Err(crate::SharedError::Protocol {
|
||||
message: "Expected metrics message".to_string(),
|
||||
message: "Expected agent data message".to_string(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,42 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Test script to verify collector intervals are working correctly
|
||||
# Expected behavior:
|
||||
# - CPU/Memory: Every 2 seconds
|
||||
# - Systemd/Network: Every 10 seconds
|
||||
# - Backup/NixOS: Every 60 seconds
|
||||
# - Disk: Every 300 seconds (5 minutes)
|
||||
|
||||
echo "=== Testing Collector Interval Implementation ==="
|
||||
echo "Expected intervals from NixOS config:"
|
||||
echo " CPU: 2s, Memory: 2s"
|
||||
echo " Systemd: 10s, Network: 10s"
|
||||
echo " Backup: 60s, NixOS: 60s"
|
||||
echo " Disk: 300s (5m)"
|
||||
echo ""
|
||||
|
||||
# Note: Cannot run actual agent without proper config, but we can verify the code logic
|
||||
echo "✅ Code Implementation Status:"
|
||||
echo " - TimedCollector struct with interval tracking: IMPLEMENTED"
|
||||
echo " - Individual collector intervals from config: IMPLEMENTED"
|
||||
echo " - collect_metrics_timed() respects intervals: IMPLEMENTED"
|
||||
echo " - Debug logging shows interval compliance: IMPLEMENTED"
|
||||
echo ""
|
||||
|
||||
echo "🔍 Key Implementation Details:"
|
||||
echo " - MetricCollectionManager now tracks last_collection time per collector"
|
||||
echo " - Each collector gets Duration::from_secs(config.{collector}.interval_seconds)"
|
||||
echo " - Only collectors with elapsed >= interval are called"
|
||||
echo " - Debug logs show actual collection with interval info"
|
||||
echo ""
|
||||
|
||||
echo "📊 Expected Runtime Behavior:"
|
||||
echo " At 0s: All collectors run (startup)"
|
||||
echo " At 2s: CPU, Memory run"
|
||||
echo " At 4s: CPU, Memory run"
|
||||
echo " At 10s: CPU, Memory, Systemd, Network run"
|
||||
echo " At 60s: CPU, Memory, Systemd, Network, Backup, NixOS run"
|
||||
echo " At 300s: All collectors run including Disk"
|
||||
echo ""
|
||||
|
||||
echo "✅ CONCLUSION: Codebase now follows NixOS configuration intervals correctly!"
|
||||
@@ -1,32 +0,0 @@
|
||||
#!/usr/bin/env rust-script
|
||||
|
||||
use std::process;
|
||||
|
||||
/// Check if running inside tmux session
|
||||
fn check_tmux_session() {
|
||||
// Check for TMUX environment variable which is set when inside a tmux session
|
||||
if std::env::var("TMUX").is_err() {
|
||||
eprintln!("╭─────────────────────────────────────────────────────────────╮");
|
||||
eprintln!("│ ⚠️ TMUX REQUIRED │");
|
||||
eprintln!("├─────────────────────────────────────────────────────────────┤");
|
||||
eprintln!("│ CM Dashboard must be run inside a tmux session for proper │");
|
||||
eprintln!("│ terminal handling and remote operation functionality. │");
|
||||
eprintln!("│ │");
|
||||
eprintln!("│ Please start a tmux session first: │");
|
||||
eprintln!("│ tmux new-session -d -s dashboard cm-dashboard │");
|
||||
eprintln!("│ tmux attach-session -t dashboard │");
|
||||
eprintln!("│ │");
|
||||
eprintln!("│ Or simply: │");
|
||||
eprintln!("│ tmux │");
|
||||
eprintln!("│ cm-dashboard │");
|
||||
eprintln!("╰─────────────────────────────────────────────────────────────╯");
|
||||
process::exit(1);
|
||||
} else {
|
||||
println!("✅ Running inside tmux session - OK");
|
||||
}
|
||||
}
|
||||
|
||||
fn main() {
|
||||
println!("Testing tmux check function...");
|
||||
check_tmux_session();
|
||||
}
|
||||
@@ -1,53 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
echo "=== TMUX Check Implementation Test ==="
|
||||
echo ""
|
||||
|
||||
echo "📋 Testing tmux check logic:"
|
||||
echo ""
|
||||
|
||||
echo "1. Current environment:"
|
||||
if [ -n "$TMUX" ]; then
|
||||
echo " ✅ Running inside tmux session"
|
||||
echo " TMUX variable: $TMUX"
|
||||
else
|
||||
echo " ❌ NOT running inside tmux session"
|
||||
echo " TMUX variable: (not set)"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "2. Simulating dashboard tmux check logic:"
|
||||
echo ""
|
||||
|
||||
# Simulate the Rust check logic
|
||||
if [ -z "$TMUX" ]; then
|
||||
echo " Dashboard would show:"
|
||||
echo " ╭─────────────────────────────────────────────────────────────╮"
|
||||
echo " │ ⚠️ TMUX REQUIRED │"
|
||||
echo " ├─────────────────────────────────────────────────────────────┤"
|
||||
echo " │ CM Dashboard must be run inside a tmux session for proper │"
|
||||
echo " │ terminal handling and remote operation functionality. │"
|
||||
echo " │ │"
|
||||
echo " │ Please start a tmux session first: │"
|
||||
echo " │ tmux new-session -d -s dashboard cm-dashboard │"
|
||||
echo " │ tmux attach-session -t dashboard │"
|
||||
echo " │ │"
|
||||
echo " │ Or simply: │"
|
||||
echo " │ tmux │"
|
||||
echo " │ cm-dashboard │"
|
||||
echo " ╰─────────────────────────────────────────────────────────────╯"
|
||||
echo " Then exit with code 1"
|
||||
else
|
||||
echo " ✅ Dashboard tmux check would PASS - continuing normally"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "3. Implementation status:"
|
||||
echo " ✅ check_tmux_session() function added to dashboard/src/main.rs"
|
||||
echo " ✅ Called early in main() but only for TUI mode (not headless)"
|
||||
echo " ✅ Uses std::env::var(\"TMUX\") to detect tmux session"
|
||||
echo " ✅ Shows helpful error message with usage instructions"
|
||||
echo " ✅ Exits with code 1 if not in tmux"
|
||||
echo ""
|
||||
|
||||
echo "✅ TMUX check implementation complete!"
|
||||
Reference in New Issue
Block a user