EventSentry v4.1

EventSentry v4.1 builds on v4.0 released earlier this year and offers a lot of exciting new & improved features that enhance a variety of different monitoring scenarios. In this release we improved:

  • ADMonitor
  • Laptop / Mobile Monitoring
  • Performance Monitoring
  • NetFlow Security
  • MSP Support
  • Security Features
  • Web Reporting


ADMonitor

Expiring passwords can often be an issue in larger networks, especially for mobile users whose passwords expire while they are out of the office. The new “Password Reminder” feature in ADMonitor alleviates this issue by giving you the ability to automatically send out password expiration reminders to your users before the password expires. The only requirement is that the ‘mail’ attribute is set for your users, or that it’s possible to dynamically determine the end user’s email address from one or more of its AD properties (e.g. first name, last name).

ADMonitor Password Reminder Email Configuration

We also added new ADMonitor-related tiles that provide an overview of recent AD changes.

Active Directory Statistics


Laptop / Mobile Monitoring

With an increasing number of employees working remotely, ensuring that laptops are properly monitored and secure should remain a priority for any company that manages laptops. Starting with v4.1, EventSentry detects the BitLocker status of any host, allowing you to run reports to identify all laptops that pose a security risk due to their hard drive not being encrypted.

Battery Health Monitoring

We also improved operational monitoring by tracking the health of laptop batteries; EventSentry can now tell you how healthy a laptop battery is based on the current capacity and the charge cycle count.


Performance Monitoring

Up until now performance data could “only” be retrieved from Windows performance counters and SNMP-based counters, but obtaining data from other sources like web pages or utilities was not supported. This limitation is a thing of the past as you can now use the output of any executable or script as a data source – with practically unlimited possibilities. For example, numerical data from system tools, web pages and log files can now be visualized and alerted upon – all with the same familiar interface. An example of this new functionality can be seen on our live demo, where we’re displaying air pollution stats from 4 major cities in the US along with the global PPM (courtesy of the EPA).


NetFlow Security

EventSentry’s NetFlow implementation already includes two important security-related features: The ability to detect port scans and identifying traffic going to / coming from potentially malicious IP addresses (with support for AbuseIPDb).

One potential short-coming with the malicious IP detection is that any of your public-facing IP addresses will – sooner or later – be contacted by a remote IP address deemed malicious. These alerts often result in unwanted and unnecessary noise, especially if the port / service that the remote IP address tried to contact is blocked anyways.

To make these alerts more actionable, NetFlow v4.1 keeps track of the amount of traffic sent/received from a malicious IP, resulting in intelligent notifications that are only triggered if the amount of traffic exceeds a (configurable) limit. This means that you will only get an alert if a meaningful amount of data (e.g. 1Mb) was transferred to or from a malicious IP – for example if a APT is active on your network. Once identified, you can either get an alert and/or take corrective action by blocking the offending IP address.


MSP Support

EventSentry’s architecture already supports MSP-style scenarios well: Granular permissions and multi-tenant support in the web reports allow for multiple clients, and remote agents running on a customer network can securely transmit all data encrypted over the Internet to the central EventSentry collector.

Starting with V4.1, both the heartbeat and network services components also integrate with the collector and can transmit all collected data from the customer’s network directly to the collector – instead of requiring a direct database connection.


Security

Helping you keep your network as secure as Fort Knox and assisting you with your compliance requirements remains a top priority for EventSentry.

Do you know how many servers and workstations on your network require a reboot to finish installing Windows updates or software? EventSentry now detects pending reboots as part of its inventory functionality – simply schedule a report on this new flag and you’ll never forget to reboot critical systems again.

BitLocker and “Needs Reboot” indicators

BitLocker detection mentioned earlier also helps you secure your mobile workforce by quickly identifying laptops that do not use full disk encryption.

To aid troubleshooting and forensic analysis we added a “Changes” view that shows all permanent changes that occurred on a selected host – for example services being added/removed, critical system files, software installations and more. This is available on the Host Inventory page as well as the IP Activity page.

Changes made to a monitor host at-a-glance

For users who need to be compliant with CJIS we also added CJIS reports to list of compliance reports.


Web Reports

Besides a UI refresh and easier access to event log data, Syslog messages can now also be acknowledged – just like event log records.

Since accumulating too much data is a common issue for our users, we improved the Database Summary dashboard tile which actively monitors the database size and detects failed purge jobs.

The IP Activity page has been improved and now offers more actionable information about IP and host activity.

Finally, for those concerned about Java(c)’s licensing, EventSentry now utilizes the OpenJDK.

We hope you find this additions useful, as always please don’t hesitate to send us feedback. I also encourage you to check out our system32.eventsentry.com site which has a ton of information about Windows events and more. It’s under constant development and tells you how events are related, whether you should monitor them, which audit settings are associated with the event and much more.

Happy Monitoring!

RDProtector: Automatically blocking malicious IPs from RDP with EventSentry

RDProtector: Automatically blocking malicious IPs from RDP with EventSentry

The recently discovered BlueKeep RDP vulnerability reminds us yet again (as if we needed to be reminded) that monitoring RDP is not a luxury but an absolute necessity.

Many organizations still expose RDP ports to the Internet, making it a prime target for attacks. But even when RDP is only available internally it can still pose a threat – especially for large networks.

So let’s start this off with some very basic best practices:

  • Make sure that RDP access is blocked from the Internet (e.g. only accessible via VPN)
  • RDP should be disabled on hosts where it’s not needed
  • All RDP access should be monitored (see below)

In this post you will see how EventSentry (and EventSentry Light) can be configured to automatically block remote hosts that have failed to log on via RDP after a certain number of times. Utilizing EventSentry offers a number of benefits over other approaches:

  1. It works with any version of Windows, from Windows 2008 to Windows 2019
  2. It works regardless of account lockout policies
  3. The threshold and time period are fully configurable
  4. The default action (block Windows firewall) can be substituted and/or supplemented with other actions

Before we delve into the nitty gritty details I need to level the playing field and explain why blocking remote RDP connection attempts is not as simple as linking event id 4625 with type 10 (failed RDP logon attempt) with an action. See, in the good old days security events logged by Windows mostly meant what they said. Failed logon events logged by Windows always included the correct logon type – all the way back to Windows Server 2003 (back then it was event 529). Having an event that included both the username, IP address and logon type made it straightforward to setup a rule:

If # of failed logons with type 10 of a [user] and/or from [IP address] > [threshold] then do [ABC].

All that changed with the introduction of NLA (Network Level Authentication), where the initial authentication of a RDP session is offloaded to another Windows subsystem, resulting in key information being lost in translation inside Windows.

The result is that starting with Windows 2008 and NLA enabled, event id 4625 always classify failed RDP logon attempts as logon type 3 instead of logon type 10. As a reminder, logon type indicates a network logon – not a RDP logon. It’s consequently impossible to use 4625 events as the sole indicator for a failed RDP logon.

Security Event 4625 with Logon Type 3 (network logon)

How do you know if NLA is enabled? It’s usually pretty simple: If you are prompted for credentials when initiating a RDP connection before you see the Windows logon screen then NLA is enabled.

In an effort to better audit RDP connectivity events, Windows 2008 and later include a new event log, the Microsoft-Windows-RemoteDesktopServices-RdpCoreTS/Operational log, which logs some RDP activity. I say some because it cannot be used to solely detect failed RDP logins. While we have been able to consistently generate events when a remote client connects (event id 131), we have been unable to consistently generate the more important event id 140, which indicates a failed login (which could be used in place of the 4625 event to trigger an action).

Microsoft-Windows-RemoteDesktopServices-RdpCoreTS event 131

So what are we to do? On the one hand we have an event telling us that a RDP connection has been initiated (although not fully logged on yet), and on the other hand we have a failed logon event that is virtually identical to hundreds or thousands of other failed logon events.

Thankfully there is an easy solution with EventSentry’s filter chaining feature, which allows us to correlate events from the security event log with the new RdpCoreTS event log. This allows us to correlate audit failure event 4625 from the Security event log with information event 131 that is logged in the Microsoft-Windows-RemoteDesktopServices-RdpCoreTS/Operational event log

Filter chaining is activated on the package level, and can trigger an action (e.g. email, process, …) when all filters in a package match events in a certain time period. To make sure that the correct types of events are chained (correlated) together, insertion strings sharing the same data can be specified. And since both events include the IP address of the remote host connecting, they can be linked (chained) together if they occur within a certain time frame (e.g. 10 seconds).

When linking events via insertion strings it’s important that the strings match exactly, any deviation will break the chain. This turns out to be a potential issue since event id 131 doesn’t just log the remote IP address but also the remote source port in a single string (e.g. 192.168.1.1:33544). Event 4625 also logs the remote IP and source port, but in different fields.

To address this, EventSentry includes a feature that can override existing insertion strings (or create new ones if none already exist) which comes in handy in this scenario. In the case of the event 131 we can use a RegEx pattern to simply remove the remote port from the string so that we only end up with the IP address – as the only insertion string.

Transforming or creating insertion strings (meta data) using RegEx expressions

The diagram below visualizes how the failed RDP login detection works with EventSentry. When an unsuccessful login via RDP occurs (1), whether or not NLA is enabled (2) determines which type of 4625 event will be logged by Windows. The RDP subsystem logs event 131 either way (3), but we utilize it when NLA is active. Without NLA we simply utilize event 4625 (4) as the trigger for one or more actions, whereas with NLA being active we need to evaluate two different events.

With NLA enabled, event id 131 is evaluated first (5). Since event 131 is logged regardless of whether the subsequent authentication is successful or not, it needs to be correlated with a potential subsequent 4625 security event (8). In order to correlate those two events based on the IP address however, the remote port needs to be removed from event 131 so that only the IP address remains (6). Once event 131 is registered and reformatted, EventSentry will look for subsequent 4625 events (8) with a matching IP address (7).


Note: Since blocking every failed RDP-based authentication could lock out legitimate users that enter an incorrect password by accident, it’s highly recommend to add a threshold for event 4625 (8). When downloaded from EventSentry, our 4625 filter has a default threshold of 3 in 1 minute per IP address. This means that hosts will be blocked if an incorrect password is specified 4 times within 1 minute (from the same IP address, that’s what insertion string 20 is for).

Filter threshold configuration

Correlating multiple events is the nature of a filter chaining package (9), which requires that all events listed in the package match during a specified time interval. Once all filters (131 + 4625 in this case) match, EventSentry will log event id 10650 to the application event log, specifying the name of the filter chaining package along with the time span and insertion string(s), the ip address in this case (10). That event is then used as the trigger for one or more actions (11), such as blocking the remote IP using the Windows firewall and/or for sending an email alert.

Blocking an IP address with Windows Firewall is easy and can be done with the netsh.exe command, for example:

%SYSTEMROOT%\system32\netsh.exe advfirewall firewall add rule name="$STR3 $YEAR-$MONTH-$DAY -- automatic block by EventSentry" dir=in interface=any action=block remoteip=$STR3/32

$YEAR, $MONTH and $DAY are variables that are generally always available in EventSentry, and $STR3 is the third insertion string from whichever event triggered the action. In our example we trigger netsh from event id 10650, which specifies the IP address in its insertion string %3:

The filter chain for event log package %1 is complete.

Duration: %2 second(s)
Insertion Strings (if any): %3

Below is the actual event as it would be found in the EventSentry event viewer. You can view the insertions strings with the EventSentry management console under Tools -> Utilities -> Event Message Browser or with the EventSentry SysAdmin Tools.

After we put everything together in EventSentry we end up with the following:

1. A Filter Chaining Package (“RDProtector”) which logs the above event when it detects failed RDP logons
2. A filter that triggers the firewall blocking from event 10650 (“Block Failed RDP IP”)
3. An action (“Block IP with Windows Firewall”) that calls netsh.exe to block an IP address

Newer EventSentry installations include the RDProtector package out of the box, but the package can also be downloaded through the Tools -> Download Packages feature. Keep in mind that both the “Corrective Actions” package and the “Block IP with Windows Firewall” action need to be created manually, their respective configuration is shown below.

The process command line (“Arguments”) should be: advfirewall firewall add rule name=”$STR3 $YEAR-$MONTH-$DAY — automatic block by EventSentry” dir=in interface=any action=block remoteip=$STR3/32

Rules added to the Windows firewall are perpetual of course, which – depending on the number of blocks – may result in a large number of Windows firewall rules. A somewhat easy work-around would be to launch a script that:

1. Creates the firewall rule
2. Waits a certain amount of time (e.g. 5 min)
3. Deletes the firewall rule again

A script with a 3-minute timeout would look slimilar to this:

advfirewall firewall add rule name="$STR3 $YEAR-$MONTH-$DAY -- automatic block by EventSentry" dir=in interface=any action=block remoteip=$STR3/32

timeout /t 180

advfirewall firewall delete rule name="$STR3 $YEAR-$MONTH-$DAY -- automatic block by EventSentry" dir=in

Stay safe out there.

EventSentry v4.0 – Introducing ADMonitor

Since Active Directory is the foundation of all Windows networks, monitoring Active Directory needs to be part of any comprehensive security strategy. Up to version 3.5, EventSentry utilized Windows auditing and the security event log to provide reports on:

  • User Account Changes
  • Group Changes
  • Computer Account Changes

While this functionality provides a good basis for monitoring the most relevant changes to Active Directory, we felt that a more comprehensive approach to monitoring Active Directory was needed – without the need to install & maintain yet another product!

ADMonitor is new (optional) component included in EventSentry that vastly improves Active Directory monitoring with these additional features:

  • Monitors changes to all objects (e.g. OUs) – not just users/groups/computers
  • Captures every attribute change made to an object, not just high level changes
  • Provides before & after values for all changes
  • Monitors Group Policy changes
  • User status reports (show idle users, users with non-expiring passwords, …)
  • Monitoring does not require auditing

We’re excited that we can now offer EventSentry ADMonitor to our users who are looking for a more in-depth Active Directory monitoring solution.

Active Directory is essentially a representation of the employees and their roles in your organization. But employees come and go, roles/responsibilities change, contractors get temporary access and so forth. But while adding users and additional access is usually reflected properly in Active Directory (otherwise IT would get a call because somebody presumably can’t do their job), removing access is often forgotten. As a result, users that should have been removed from AD a long time ago continue to exist. With ADMonitor it’s easy to identify orphaned user accounts (and many others) and keep your Active Directory lean and clean.

Discover weak links in Active Directory
Discover weak links in Active Directory

Since a significant development effort stands behind ADMonitor, it will be offered as an optional component that is licensed on a per-user basis. Pricing is very competitive with other solutions and we also offer bundle discounts to customers who already own or will purchase agent licenses; please request a quote here.

But enough theory, let’s look into the installation, configuration and reporting of ADMonitor.

Installation

Since ADMonitor is a component of EventSentry, it’s easily activated as part of the main EventSentry setup. Just like with other components of EventSentry (Heartbeat Monitor, Collector, …), users have the option to enable ADMonitor during the post installation setup procedure.

ADMonitor can be installed on any host that is part of the domain that needs to be monitored, it does not need to be installed on a domain controller.

Enabling ADMonitor
Enabling ADMonitor

Immediately following the initial installation, ADMonitor will initialize itself by creating an offline copy of all Active Directory objects. This process can take from a few seconds to a few minutes, depending on the number of objects in AD, connection speed to the domain controller as well as the overall performance of the host running ADMonitor.

Configuration

The initial configuration of ADMonitor is simple and only requires you to pick a password for the ADMonitor service account. If you’re adding ADMonitor to an existing installation you may also need to select the appropriate EventSentry database action to which ADMonitor reports changes. Otherwise, ADMonitor is ready from the get go and will monitor all Active Directory changes.

Reporting

ADMonitor provides three types of reports:

  • Object Changes
  • Group Policy Changes
  • User Status

Object Changes
Shows any change made to an AD objects. Reports can be filtered on the type of action performed (added, removed, modified), on the object type (user, group, organizationalUnit, …) and on the user who performed the action.

ADMonitor Object Change
ADMonitor Object Changes

Note that the detailed changes to group policies are available in the “Group Policy Changes” report below. Of course you can expect the same type of summary view you’re already used to from most other EventSentry features and create reports like:

  • Show all changes to organizational units
  • Show all new objects created
  • Show all users that were changed
  • and more

Group Policy Changes
When a group policy is changed, it is first indicated on the “Object Changes” report, since the versionNumber attribute of the AD object changes. The actual group policy settings themselves are available on the “Group Policy Changes” report however, since group policy settings are not stored in AD.

The screen shot below shows that the Default Domain Policy was changed, with the Specify traps for the public community setting being enabled.

ADMonitor Group Policy Change
ADMonitor Group Policy Change

Users
The users report helps you identify potentially problematic user accounts such as idle users, users who haven’t change their passwords in years and others.

This report contains a list of all user objects in Active Directory including the following details:

  • Name, Full Name, SAM Account Name, Path, UPN
  • Administrative Account (yes/no)
  • Disabled (yes/no)
  • Password Never Expires (yes/no)
  • Password Expired (yes/no)
  • Password must change (yes/no)
  • Locked Out (yes/no)
  • Last Logon
  • Password Last Set
  • Account Expiration Date
  • Creation Date

ADMonitor User Overview
ADMonitor User Overview

With ADMonitor you can now get detailed user stats with just a few clicks and quickly identify user accounts that need to be reviewed, changed or deleted. Of course you can also schedule all reports directly from the web reports and get daily/weekly AD status reports directly in your inbox, e.g.:

  • List of all Group Policy changes
  • List of all idle user accounts
  • List of all newly created users and/or groups

You can also create your own reports for just about anything that involves a change to an Active Directory object, for example all organizational units created in the last 24 hours.

ADMonitor also includes a number of stand-alone utilities that support advanced features such as filtering and email notifications that I will cover in a future post.

With ADMonitor, EventSentry users can now gain the additional visibility needed to fully audit all Active Directory & Group Policy changes. As a result, EventSentry users can more easily enhance compliance, security and accountability in their network without the need to install additional software – saving both time and money.

EventSentry v3.5 Released: Windows Process Monitoring to the Max, Registry Tracking, Tags & More

EventSentry v3.5 continues to increase visibility into networks with additional vantage points, making it easier for EventSentry users to reduce their attack surface as well as discover anomalies.

Process & Network Activity Tracking

One major focus of this release is process network activity, an important component in any monitoring strategy. Do you know which applications listen for incoming connections on your monitored machines – or when a new process suddenly starts accepting incoming traffic? Do you know which processes perform outgoing network connections, and to where? How much data are they transferring?

Process Tracking with Sysmon
Figure 1: View process network activity from Sysmon

To help you (and possibly your overzealous auditors) answer these questions, EventSentry v3.5 takes the existing process tracking functionality to the next level by integrating with Sysmon and showing processes with active or listening network connections. With EventSentry deployed you can now see:

  • Complete Process Details (start & stop times, duration, caller, PID)
  • Process checksum
  • Process command line
  • All processes listening for incoming connections
  • All active processes
  • Network activity initiated by a process (Figure 1, requires Sysmon)
  • Correlation with EventSentry NetFlow (Figure 2, requires NetFlow component)

This means that you can easily see which network connections a host establishes if you have Sysmon installed, and can even correlate that information with the EventSentry NetFlow component with just a click (see below). This information is invaluable for forensics and troubleshooting alike.

Process Network Activity
Figure 2: Detailed process network activity from NetFlow data

But even without Sysmon, EventSentry can now show you every open TCP port on a monitored machine (optionally all active connections as well), making it  easy to discover rogue services on a network – even if they are blocked by the firewall. Figure 3 below shows all active processes which are listening for incoming connections, grouped by host.

All Active Listening Processes
Figure 3: All active processes which are listening for incoming network connections

Registry Tracking

A new member in the compliance tracking features family is registry tracking. Similar to file access tracking, it normalizes all registry audit events on a monitored machine, making it much easier to report on the registry activity and changes. Configuring registry tracking to work with existing registry auditing is incredibly easy and can be enabled in 60 seconds if the proper audit settings are already in place. The screen shots below show a list of recent registry activity as well as the details of changes:

Registry Tracking Overview
Registry activity on monitored machines

 

Registry Tracking Details
List of changes made to critical registry values

Tags

Users managing a large number of hosts will appreciate the new “Tags” feature which addresses a shortcoming with the existing flat group structure. Tags allow groups or hosts to be tagged with keywords (e.g. production, staging, development). The created tags can then be used in the web reports (e.g. Show me disk space from all hosts tagged with “development”) and for dynamic package assignments.

FIM

The file checksum monitoring component received a few enhancements to help reduce noise while also adding new functionality.

FIM can now verify the digital signature of executable files and optionally suppress alerts if a file is digitally signed – think Windows updates. This can reduce the number of alerts you get significantly and thus make the remaining alerts more meaningful. The digital signature status can also be displayed in the web reports as a new column.

FIM can also calculate the entropy (essentially a measure of randomness) of files with a scale from 0 to 10, with 10 being the maximum of a completely random file. This is useful for Ransomware detection, since encrypted (and compressed files as well) files have a higher entropy than regular files. Combined with a threshold filter this can detect when a large number of encrypted files are suddenly being processed in a given directory and thus indicate a Ransomware infection.

File Monitoring Alert
File monitoring alerts now include signature details and entropy

We also replaced the existing SHA-256 checksum algorithm with a faster version in 3.5 which should result in a lower CPU utilization on systems which need to calculate a large number of checksums.

Disk Space Monitoring

A common annoyance with disk space monitoring are large volumes where an otherwise useful limit of, say 5%, is just not useful. For example, 5% of a 2 Tb drive is still 100 Gb, and in most cases there is probably no reason to sound an alarm. Dynamic thresholds (a new feature) addresses this issue by automatically adjusting the limit based on the drive size. The result: Fewer alerts!

EventSentry will log an event to the application event log when dynamic thresholds are enabled AND the current settings warrant a change. An event will look something like this:

The percentage-based threshold on drive F:\ has been dynamically adjusted from 5 percent to 0.5 percent based on the total drive size of 999 GB. A low disk space alert will be triggered when the available space on this volume falls below 19 GB.

Other Improvements

The software inventory page (detailed tab) will now show which hosts do NOT have a particular software installed when the search is restricted to a specific software product. Also related to processes, process tracking can now generate the checksum of all execute files, which can then be searched for at Malware databases like virustotal.com. If you utilize the maintenance mode feature in EventSentry then you can now see whether a host is in maintenance mode or not in the web reports. And last but not least, event logs can now be sent to a remote Syslog receiver via TLS.

Under the Hood

We always tweak and improve EventSentry to ensure it runs as efficiently as possible. In this release we replaced the SHA 256 algorithm with a more efficient version, resulting in less CPU usage by the agent when calculating SHA 256 checksums. As we gradually move to a full 64-bit monitoring suite, the Heartbeat Agent is the next component now also available as a 64-bit process so that all EventSentry services are now available in 64-bit. We plan on porting all executables over to 64-bit within the next 6-12 months. At that point you will not be able to run EventSentry on 32-bit platforms anymore; monitoring 32-bit hosts will of course still be supported for the foreseeable future.

To help with the stability of all EventSentry and simply troubleshooting, all server-side components will now automatically generate crash dumps if they encounters a problem. Finally, the management console includes additional context and ribbon buttons.

EventSentry SysAdmin Tools: Digital Signature Verification with checksum.exe

Windows supports a code-signing feature called Authenticode, which allows a software publisher to digitally sign executable files (e.g. .exe, .msi, …) so that users can verify their autenticity. The digital signature of a file can be viewed in the file properties in Windows explorer on the “Digital Signature” tab.

Viewing the digital signature of the Opera browser

Digital signature verification has been added to the checksum utility, which already calculates the checksum and entropy of a file. When using the new /s switch, checksum.exe will tell you whether:

  • the file is digitally signed
  • a counter signature exists
  • the digital signature is valid
  • the algorithm used (e.g. SHA 256)
  • who signed the file
  • who issued the certificate
  • when the file was signed

The utility also sets the ERRORLEVEL variable accordingly; if a signature check is requested with the /s switch but the file is unsigned, then checksum.exe will return %ERRORLEVEL% 2. Below is a sample output of the utility in action:

Viewing the digital signature of the Windows ping utility

Digital signature verification will be added to EventSentry’s FIM monitoring component (“File Checksum Monitoring”) in the upcoming v3.4.3 release, which will give you the option to only get notified when unsigned files are changed, thus reducing overall noise.

You can download the latest version from here – enjoy!