Add comprehensive timeouts to all blocking system commands
Fixes random host disconnections caused by blocking operations preventing timely ZMQ packet transmission. Changes: - Add run_command_with_timeout() wrapper using tokio for async command execution - Apply 10s timeout to smartctl (prevents 30+ second hangs on failing drives) - Apply 5s timeout to du, lsblk, systemctl list commands - Apply 3s timeout to systemctl show/is-active, df, ip commands - Apply 2s timeout to hostname command - Use system 'timeout' command for sync operations where async not needed Critical fixes: - smartctl: Failing drives could block for 30+ seconds per drive - du: Large directories (Docker, PostgreSQL) could block 10-30+ seconds - systemctl/docker: Commands could block indefinitely during system issues With 1-second collection interval and 10-second heartbeat timeout, any blocking operation >10s causes false "host offline" alerts. These timeouts ensure collection completes quickly even during system degradation.
This commit is contained in:
@@ -43,8 +43,8 @@ impl NixOSCollector {
|
||||
match fs::read_to_string("/etc/hostname") {
|
||||
Ok(hostname) => Some(hostname.trim().to_string()),
|
||||
Err(_) => {
|
||||
// Fallback to hostname command
|
||||
match Command::new("hostname").output() {
|
||||
// Fallback to hostname command (with 2 second timeout)
|
||||
match Command::new("timeout").args(["2", "hostname"]).output() {
|
||||
Ok(output) => Some(String::from_utf8_lossy(&output.stdout).trim().to_string()),
|
||||
Err(_) => None,
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user