Efficacy Testing

ICS Watch Dog Efficacy Test Script

The ICS Watch Dog project includes a PowerShell validation script that tests Sysmon config efficacy by triggering safe, benign actions and verifying the events appear in the Sysmon log.

Requirements

• Windows with Sysmon installed, configured, and running
• Administrator privileges (required to read the Sysmon event log)
• PowerShell 3.0 or later
• No external modules or dependencies

Download

Download the script from the ICS Watch Dog repository:

tools/Test-SysmonConfig.ps1

Usage

Run from an elevated (Administrator) PowerShell prompt:

# Run default observation-only tests (9 Event IDs)
.\Test-SysmonConfig.ps1

# Run all tests including registry and WMI changes (14 Event IDs)
.\Test-SysmonConfig.ps1 -AllowSystemChanges

# Increase wait time for slow systems (default: 10 seconds)
.\Test-SysmonConfig.ps1 -WaitSeconds 20

# Skip the confirmation prompt (for automated testing)
.\Test-SysmonConfig.ps1 -SkipConfirmation

Test Groups

Tests are divided into two groups based on system impact:

How It Works

1. Pre-flight checks -- Verifies administrator privileges, Sysmon service running, event log accessible, PowerShell version.
2. Planned changes display -- Shows every action the script will perform, with full paths and warnings for system-modifying tests.
3. Confirmation prompt -- Requires you to type Y before proceeding (override with -SkipConfirmation).
4. Execute triggers -- Performs safe actions that should generate Sysmon events.
5. Wait for propagation -- Pauses to allow Sysmon to write events to the log (configurable via -WaitSeconds).
6. Verify events -- Queries the Sysmon event log for each expected Event ID within the test time window.
7. Cleanup -- Removes all test artifacts with per-item success/failure reporting.
8. Summary report -- Pass/fail per Event ID, cleanup status, and guidance for any failures.

Interpreting Results

A [PASS] result means the Sysmon event appeared in the log within the wait period. A [FAIL] result may indicate:

• The Sysmon config does not monitor this event type (check your config's RuleGroup for the Event ID)
• The event type is disabled in the config (e.g., ImageLoad and FileDelete are disabled by default in most ICS Watch Dog configs)
• Event log propagation was slower than the wait period -- re-run with -WaitSeconds 30
• The config uses include rules that did not match the specific test action (review the rule conditions)

Not all Event IDs will pass with every config. For example, the community file-create-only config only monitors Event ID 11 -- all other tests will correctly show [FAIL]. The test validates what your config detects.

Event ID Test Details

The following table describes each test action and what it validates:

Observation-Only Tests (Default)

System Change Tests (Requires -AllowSystemChanges)

Event IDs Not Tested

The following Event IDs are not included in the script because they cannot be safely triggered from PowerShell without risking system stability or requiring specialized tools:

To test these Event IDs, use dedicated security testing tools in a lab environment (see Advanced Testing Tools below).

Manual Cleanup

If the script reports cleanup failures, use these commands to manually remove test artifacts. All artifacts use the prefix ICSWatchDog-EfficacyTest for easy identification.

Files

Test files are created in the user's Temp directory:

# Check for remaining test files
Get-ChildItem -Path $env:TEMP -Filter "ICSWatchDog-EfficacyTest*"

# Remove them
Remove-Item -Path "$env:TEMP\ICSWatchDog-EfficacyTest*" -Force

Registry

The system change test creates a single value in the HKCU Run key:

# Check for the test registry value
Get-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Run" -Name "ICSWatchDog-EfficacyTest" -ErrorAction SilentlyContinue

# Remove it
Remove-ItemProperty -Path "HKCU:\Software\Microsoft\Windows\CurrentVersion\Run" -Name "ICSWatchDog-EfficacyTest" -Force

WMI Subscriptions

The system change test creates three WMI objects. Remove them in this order (binding first, then consumer, then filter):

# Remove binding
Get-WmiObject -Namespace "root/subscription" -Class "__FilterToConsumerBinding" |
  Where-Object { $_.Filter -like "*ICSWatchDog-EfficacyTest*" } | Remove-WmiObject

# Remove consumer
Get-WmiObject -Namespace "root/subscription" -Class "CommandLineEventConsumer" |
  Where-Object { $_.Name -eq "ICSWatchDog-EfficacyTest-Consumer" } | Remove-WmiObject

# Remove filter
Get-WmiObject -Namespace "root/subscription" -Class "__EventFilter" |
  Where-Object { $_.Name -eq "ICSWatchDog-EfficacyTest-Filter" } | Remove-WmiObject

To verify all WMI test objects are removed:

# Should return no results
Get-WmiObject -Namespace "root/subscription" -Class "__EventFilter" |
  Where-Object { $_.Name -like "*ICSWatchDog*" }
Get-WmiObject -Namespace "root/subscription" -Class "CommandLineEventConsumer" |
  Where-Object { $_.Name -like "*ICSWatchDog*" }
Get-WmiObject -Namespace "root/subscription" -Class "__FilterToConsumerBinding" |
  Where-Object { $_.Filter -like "*ICSWatchDog*" }

Advanced Testing Tools

The ICS Watch Dog efficacy test script is a focused smoke test. For organizations wanting deeper adversary emulation and detection validation, these tools provide broader coverage:

When to Test

Run efficacy tests:

• After initial Sysmon deployment -- Confirm the config is working before relying on it for detection.
• After config changes -- Verify that rule modifications did not break existing detections.
• After Sysmon version upgrades -- Confirm compatibility with the new version.
• Periodically -- Quarterly or semi-annually to catch configuration drift or accidental changes.
• During incident response preparation -- Validate detection capability as part of IR readiness exercises.