Logging and Diagnostics: Making Field Problems Solvable
A fault that happens once a week on a customer machine, a factory floor or inside a connected device cannot be debugged with a breakpoint. What the team has is whatever the software recorded at the time. Good diagnostics are designed, not accumulated: consistent levels, useful context, bounded storage, crash capture and a simple way for a customer or a device to hand the evidence over. This article covers how to build that in.
At a glance
- 8 levels
- Syslog defines eight severities, from emergency down to debug, a useful reference scale.
- UTC
- Timestamps in UTC with stated precision make logs from different machines comparable.
- Rotation
- Log files are capped and rotated so diagnostics never fill the disk.
- Symbols
- Debug symbols archived for each release turn crash dumps into readable stack traces.
The problem
Logs written ad hoc fail in two directions. Some code logs nothing useful, so the record stops just before the interesting moment. Other code logs everything on every loop, which buries the one relevant line and wears out flash storage on embedded devices.
Even good logs are useless if they cannot be retrieved. Asking a customer to find a hidden folder, or a field technician to attach a debugger, means most problems are never properly reported.
Write each log line for the engineer who will read it months from now with no other information. That is usually who reads it.
What a useful log line contains
Record a timestamp, a severity, the component and the event with its key values: the device serial number, the request identifier, the error code and the state the code was in. Structured formats such as JSON key-value pairs make logs searchable by tools rather than only by eye.
Keep secrets and personal data out. Passwords, tokens and keys must never be logged, and personal data should be minimised or masked, both for security and to meet privacy law such as GDPR.
What good looks like
Agree severity levels and their meaning across the codebase, and decide what is logged by default and what only when verbose logging is switched on.
Add crash capture, such as minidumps on Windows and core dumps or a crash handler on Linux and macOS, and archive the matching symbols for every released build.
Provide a one-click diagnostics export, or an opt-in upload, that bundles logs, version information and configuration into a single file.
| Level | Use for | Logged by default |
|---|---|---|
| Error | A failed operation the user or system will notice | Yes |
| Warning | Something unexpected that was handled | Yes |
| Info | Significant state changes and lifecycle events | Yes, sparingly |
| Debug | Detail for diagnosing a specific problem | No, enabled on demand |
| Trace | Step-by-step flow and raw data | No, development only |
From fault to fix
- Event: Logged with context and identifiers
- Store: Rotated, bounded, crash dumps kept
- Collect: Export bundle or opt-in upload
- Analyse: Symbols, search and correlation
- Fix: Reproduce, test, release
What it changes
Faster Diagnosis
Engineers start from evidence instead of a description of what the user thinks happened.
Fewer Repeat Visits
Intermittent faults are captured the first time they occur rather than reproduced at length.
Bounded Storage
Rotation and levels keep logs useful without filling disks or wearing out flash.
Privacy
Masked and minimised logs can be shared with support without exposing customer data.
Where to go next
For connected products, correlate device, cloud and app logs by carrying a shared identifier through each request. A fault often spans all three.
Allow verbose logging to be switched on by configuration or remotely for a limited time, so detail is available when needed without running at full volume permanently.
Problems this prevents
- Logs that stop before the fault
- Disks or flash filled by diagnostics
- Crash dumps without matching symbols
- Passwords or personal data in log files
Discuss your project
Email support@aavico.com or use Get in Touch when the website finishes loading.