Coverage Assessment

The ICS Watch Dog coverage toolchain helps you measure how well your Sysmon configuration fits your actual system, identify gaps, and track changes over time. It consists of three scripts that work together: capture, measure, compare.

The Three Tools

Script	Purpose	PS Version
`Export-SystemInventory.ps1`	Captures what is running on a system (processes, software, ports, services) and writes it to a JSON file.	2.0+ (Console/Markdown), 3.0+ (JSON)
`Get-SysmonCoverage.ps1`	Measures how much of a system's inventory is covered by a Sysmon configuration. Reports four metrics: process, software, port, and ATT&CK technique coverage.	2.0+ (Console/Markdown), 3.0+ (JSON)
`Compare-SystemInventory.ps1`	Diffs two inventory JSON files and reports what was added or removed. Useful after patches, software installs, or for comparing two hosts.	3.0+ (requires JSON input)

All three scripts are read-only. They do not modify the system, probe the network, or require external dependencies.

Quick Start

Live system coverage (interactive)

.\Get-SysmonCoverage.ps1 -ConfigPath sysmonconfig-baseline-ot.xml

What-if with additional modules

.\Get-SysmonCoverage.ps1 `
    -ConfigPath sysmonconfig-baseline-ot.xml `
    -AdditionalModules @(
        'modules\vendor-ot\siemens-tia-portal.xml',
        'modules\protocol\modbus-tcp.xml') `
    -OutputFormat Markdown -OutputPath coverage-report.md

Capture inventory for offline analysis

.\Export-SystemInventory.ps1 -OutputPath my-system.json

Analyze a captured inventory on another machine

.\Get-SysmonCoverage.ps1 -ConfigPath sysmonconfig-baseline-ot.xml -InventoryPath my-system.json

The Workflow

Baseline -- Run Get-SysmonCoverage.ps1 against your current config. Record the coverage percentages.
Identify gaps -- Review the gap report: unmonitored processes, uncovered OT software, unmatched industrial ports.
Try modules -- Use -AdditionalModules to project what coverage would look like with specific modules added. No config changes needed.
Merge -- When satisfied, use Merge-SysmonModules.ps1 to build your deployment config.
Re-measure -- Run coverage again against the merged config to confirm gaps closed.

For locked-down OT hosts, use the offline workflow: capture inventory with Export-SystemInventory.ps1 on the target host, transfer the JSON file, and run Get-SysmonCoverage.ps1 -InventoryPath on your analysis workstation.

Reading the Coverage Report

Process Coverage

Fraction of running processes matched by at least one rule in the config. A low percentage on baseline-ot is expected -- the baseline intentionally avoids vendor-specific rules to minimize false positives. Adding vendor modules (Siemens, Rockwell, etc.) increases process coverage for systems running that software.

OT Software Coverage

Fraction of installed OT-relevant software (matched by vendor name) that has a corresponding module. The gap report suggests which module to add for each uncovered vendor.

Industrial Port Coverage

Fraction of listening industrial protocol ports (Modbus 502, OPC-UA 4840, etc.) matched by a protocol detection rule. Adding protocol modules closes these gaps.

ATT&CK Technique Coverage

Count of distinct MITRE ATT&CK technique IDs found in the deployed rules. Not a percentage -- there is no universal denominator. More techniques means broader detection.

Honest Interpretation

Coverage percentages are fit indicators, not grades.

25% process coverage on baseline-ot is normal. The baseline monitors broadly via exclusion rules, not by naming every binary.
100% port coverage does not mean you will detect Modbus attacks. It means you have a rule that watches that port.
0% software coverage means no OT vendor modules are merged -- not that the config is broken.
The coverage tool measures config-to-system fit, not detection efficacy. For efficacy testing, see the Efficacy Testing guide.

Output Formats

All three scripts support -OutputFormat Console (default), JSON, and Markdown.

Format	Use Case	PS 2.0
Console	Interactive review in a terminal	Yes
JSON	Automation, pipelines, SIEM ingest	PS 3.0+ only
Markdown	AI analysis, tickets, documentation	Yes

Use -OutputPath to write to a file instead of the screen.

Comparing Inventories

Compare-SystemInventory.ps1 diffs two inventory JSON files and reports only the changes (added and removed items per category).

.\Compare-SystemInventory.ps1 -ReferencePath before.json -DifferencePath after.json

Use cases: tracking changes after a patch, comparing a lab host to a production host, reviewing a community-submitted inventory against your reference.

Contributing Inventories

Sanitized system inventories help the project build better modules and configs for real OT environments. To submit an inventory:

Export with redaction: .\Export-SystemInventory.ps1 -OutputPath my-system.json -Redact
Review the JSON manually. The -Redact flag strips usernames from paths and omits the hostname, but cannot catch everything (project names, custom service names, license keys in display names).
Submit via GitHub issue as a feature request, or include with a pull request for a new module.

For building new detection modules based on your coverage gaps, see the Build Your Own Module guide (coming soon).

Limitations

Read-only inventory only. No active scanning, no EDR/AV correlation.
Does not verify Sysmon is actually running or that rules fire. Use Efficacy Testing for that.
Process inventory is a point-in-time snapshot. Processes that start later are not captured.
Software detection relies on the Windows registry uninstall key. Portable or xcopy-deployed software is not detected.
Industrial port matching is based on a built-in catalog of known OT protocol ports. Custom or non-standard ports are not automatically recognized.

Troubleshooting

Issue	Cause	Fix
"JSON output requires PowerShell 3.0 or later"	Running on Windows 7 / PS 2.0	Use `-OutputFormat Console` or `Markdown`
0 processes found	Insufficient privileges to enumerate processes	Run as Administrator or a user with process-read permissions
0 listening ports	`Get-NetTCPConnection` unavailable and netstat fallback failed	Ensure the script can run `netstat -ano`
ExecutionPolicy error	Script execution is restricted	Run with `-ExecutionPolicy Bypass` or set policy to `RemoteSigned`

Next: Found gaps? Build your own module to close them, or browse the Module Library for existing modules. Validate your detection with Efficacy Testing.