Deployment Considerations
Deploying Sysmon in ICS/OT environments requires careful consideration of system performance, operational impact, and a phased approach. This page addresses the questions OT administrators commonly ask before deploying endpoint monitoring.
Recommended Viewing
Before deploying Sysmon in an ICS/OT environment, these resources provide essential context:
- Windows SYSMON - SANS ICS Concepts - Discussion between Don C. Weber and Aaron Boyd covering Sysmon fundamentals for ICS/OT environments.
- Gaining Endpoint Log Visibility in ICS Environments - SANS ICS Security Summit 2019 presentation by Mike Hoffman. Covers why endpoint log visibility matters in ICS, methods to achieve it, and architectures and technology constraints for centralizing logs to IT/OT SIEM from an oil and gas perspective. Essential viewing for understanding central event logging deployment and testing for network/system latency.
Performance Impact
The most common concern from OT administrators is: "Will Sysmon slow down my systems or impact operations?"
Sysmon is designed to be lightweight. It runs as a kernel-level driver and Windows service with minimal overhead. However, the performance impact depends on:
- Configuration complexity - More rules and event types mean more processing. Baseline configs are designed for minimal impact.
- System activity level - Busy systems with many process launches, network connections, and file operations generate more events.
- Event volume - The number of events logged affects disk I/O and event log size.
- Hardware age - Legacy systems with limited CPU and RAM are more sensitive to any additional overhead.
General Guidance
| Configuration | Expected Impact | Notes |
|---|---|---|
| IT Workstation Baseline | Low | Conservative exclude-based rules, disabled high-volume events (ImageLoad, RawAccessRead) |
| IT Server Baseline | Low | Server-appropriate exclusions, fewer processes to exclude than workstations |
| Server: AD/DC | Low-Moderate | Enables RawAccessRead; LSASS monitoring tuned for DC noise. DCs are high-value and warrant the overhead. |
| Server: Services | Low-Moderate | Enables ImageLoad for web workers; database/web process noise excluded. Rules for absent services have zero cost. |
| OT Baseline | Low | Same base as IT Workstation with OT file type additions; OT systems typically have less process activity |
| OT Enhanced | Low-Moderate | Additional network connection rules for industrial ports |
| OT Advanced | Moderate | Adds Event IDs 27-29 (schema 4.90); designed for systems running current Sysmon versions |
| Jump Host | Moderate | More events enabled (ImageLoad, ClipboardChange, all network); acceptable for chokepoint systems |
Network Latency
Sysmon processes events locally and writes to the Windows Event Log. It does not introduce network latency because:
- Sysmon does not intercept or delay network traffic
- Network connection logging (Event ID 3) records connections after they are established, not before
- Event log forwarding (WEF, SIEM agents) is a separate process and can be tuned independently
If you are forwarding Sysmon events to a central collector, the forwarding agent (not Sysmon) is what generates network traffic. Monitor the forwarding agent's bandwidth usage during initial deployment.
Event Log Size
Sysmon writes to its own event log channel: Applications and Services Logs > Microsoft > Windows > Sysmon > Operational. By default, Windows limits this log to 64 MB. For OT systems, consider:
- Increasing the log size - Set to 256 MB or 1 GB for systems without centralized log forwarding to retain more history
- Monitoring log rollover - When the log is full, oldest events are overwritten. Ensure your retention period meets incident response needs.
- Forwarding events - If using WEF or a SIEM, events are stored centrally and the local log size is less critical
To increase the Sysmon log size:
wevtutil sl Microsoft-Windows-Sysmon/Operational /ms:268435456(Sets the log to 256 MB. Adjust the value as needed.)
Phased Deployment Approach
Do not deploy Sysmon to all OT systems at once. Use a phased approach:
Phase 1: Lab/Test Environment
- Install Sysmon with the chosen config on a test system that mirrors production
- Run for at least one week to establish a baseline of normal event volume
- Monitor CPU and memory usage during this period
- Review events to identify noisy rules that need tuning
Phase 2: Non-Critical Systems First
- Deploy to systems whose temporary loss would not impact operations (engineering workstations, development systems)
- Monitor for one to two weeks
- Tune the config based on observed event patterns
Phase 3: Jump Hosts and Infrastructure
- Deploy to jump hosts, bastion hosts, and OT infrastructure systems (domain controllers, file servers)
- Use the Jump Host config for bastion systems
- These systems are high-value monitoring targets and typically handle the additional load well
Phase 4: Operational Systems
- Deploy to HMI stations, historian servers, and other operational systems
- Start with the IT Workstation Baseline, OT Baseline, or Server: Services as appropriate (lowest impact)
- Monitor closely during the first week for any performance impact
- Have a rollback plan ready (Sysmon uninstall:
sysmon.exe -u)
Critical systems: For safety-critical systems (e.g., systems directly controlling physical processes), evaluate the risk of any additional software carefully. It is acceptable to exclude the most critical real-time control systems from Sysmon deployment and monitor them via network-level tools instead. The goal is to maximize visibility without impacting safety.
Criticality-Based Decisions
Not every system needs the same level of monitoring. Consider this framework:
| System Type | Recommended Config | Priority |
|---|---|---|
| Jump hosts / bastion hosts | Jump Host (comprehensive) | High - deploy first |
| Domain controllers (IT or OT) | Server: AD/DC | High - authentication infrastructure, highest-value target |
| Engineering workstations | OT Baseline or OT Enhanced | High - common attack target |
| Database / web servers | Server: Services | High - covers SQL injection, webshells, xp_cmdshell |
| Historians (PI, Ignition, AVEVA) | Server: Services (covers database + web engines used by historians) | Medium - add OT vendor directory rules from OT configs |
| HMI / operator stations | OT Baseline | Medium - balance monitoring vs. performance |
| SCADA servers | Server: Services + OT vendor tuning | Medium - after testing in lab |
| IT workstations | IT Workstation Baseline | Medium - standard endpoint monitoring |
| General file/print servers | IT Server Baseline | Medium - standard server monitoring |
| Safety-critical controllers | None (use network monitoring) | Do not install without thorough evaluation |
Tuning Your Configuration
All ICS Watch Dog configs are starting points. Administrators must tune them for their specific environment. Common tuning actions:
Reducing Noise
- Identify noisy processes - After initial deployment, review Event ID 1 (ProcessCreate) for processes that generate excessive events. Add exclude rules for legitimate, frequently-launching processes specific to your environment.
- Exclude approved software - Enterprise management agents, backup software, and approved monitoring tools may need exclusions to reduce noise.
- Tune DNS rules - If DNS monitoring is too noisy, add exclusions for known-good internal domains.
Remote Access Tool Tuning
All configs detect known remote access tools by default. After deployment:
- Identify your organization's approved remote access solution
- Review the events generated by the approved tool
- Optionally add an exclude rule for the approved tool's process name and path
- Leave all unapproved remote access tools detected - any execution of TeamViewer, AnyDesk, etc. that is not sanctioned should generate an alert
Adding Vendor-Specific Rules
The OT Baseline includes monitoring for common ICS vendor installation directories. To add your site-specific software:
- Identify the installation path of your ICS/OT software
- Add a FileCreate rule following the existing pattern:
<TargetFilename name="ICS: Your Vendor" condition="contains">\Your Vendor Path\</TargetFilename> - Validate the rule by testing on a non-production system
Adding Vendor-Specific Process Exclusions
If an approved ICS vendor process generates excessive noise in ProcessCreate events:
- Confirm the process is legitimate by verifying its path, hash, and digital signature
- Add a targeted exclude rule using the full path (never exclude by filename alone):
<Image condition="is">C:\Program Files\Vendor\process.exe</Image> - Document why the exclusion was added
Rollback Plan
Always have a rollback plan before deploying Sysmon to production OT systems:
- Uninstall:
sysmon.exe -u(removes service and driver, no reboot required) - Update config:
sysmon.exe -c new-config.xml(apply a less aggressive config without uninstalling) - Disable without uninstall: Stop the Sysmon service via
sc stop Sysmon64(temporary, restarts on reboot)
Sysmon installation and uninstallation do not require a system reboot.