File Integrity Monitoring, aka as FIM, is a must-have feature for anyone in charge of security. With FIM, one can detect when a critical file, such as a file that belongs to the Operating System, or a key configuration file, is changed. In most cases, configuring FIM is straightforward: If the file changes then generate an alert.
This binary approach (no change = good, change = bad) doesn’t work very well with transaction log files however – files that are allowed to change only in that new data can be appended to the log file. A good example are payment transaction logs: These files record transaction data and are supposed to continuously increase in size, but existing data is never supposed to change.
Regular FIM checksum monitoring doesn’t work well in this case since new data being appended to the files on a regular basis would trigger constant checksum alerts. To solve this problem and prevent tampering of log files, we introduced a new feature to our FIM monitoring capabilities in EventSentry v5.0.1.98: Only verify incremental checksums.
How does that work? Every time the file changes (in most cases that would be data being appended to the file), all data up to the previously known file size is verified. After the new data has been successfully written, EventSentry will re-calculate the checksum and wait for more changes. This means that existing data is not allowed to change and will trigger a checksum alert, whereas new data can be added without issue.
As the file grows, only the part of the file that was previously checksummed is verified
If someone where to change an existing part of the log, even if it was at the same time new data is appended, EventSentry would detect the change and trigger a checksum alert.
File Monitoring still works the same exact way in EventSentry, and all that is needed to activate this new feature is to check the “Only verify incremental checksum” box in the file monitoring dialog. This setting will affect all directories monitored by the package.
EventSentry alert after a file checksum changed
Other EventSentry features can be leveraged as well for monitoring (transaction log) files to ensure that critical files are not tampered with.
Log File Monitoring: Content of text-based logs can be transferred to the EventSentry database in real time, essentially creating a searchable backup copy of all content.
File Access Tracking: Utilizing NTFS permissions, all (write) access to the log folder can be tracked, providing an access log of every user who made changes to one or more log files.
Monitoring files that record transactions is useful even outside of PCI compliance, but if your organization needs to be PCI compliant then EventSentry also includes PCI compliance reports out of the box:
Whether it’s for compliance or improving security in general, FIM is an important security tool. In EventSentry, file integrity monitoring can also be used for automation, for example performing certain tasks when a file is added to or removed from a folder.
I think we’ve all been there before – you log on to a server remotely via RDP, and do the needful – but don’t immediately log off. But then you get distracted by a phone call, an email, a chat, or a good old-fashioned physical interaction with another human being. So when it comes time clock out for the night, you shut down your computer or log off. Or maybe you’ve been working on a laptop and your VPN got interrupted. The result is the same – previously active RDP sessions are automatically disconnected. Or maybe one (not you, of course) purposely disconnects their session so that the next login process is faster.
Well whatever may be the reason, a key issue with disconnected RDP sessions is that, by default, they remain connected indefinitely until the server gets rebooted, you login again (because you forgot to do something?), or a coworker notices the offense and politely requests you log off.
At this point you’re probably thinking “OK, yes, but who really cares“? With all the serious stuff that’s going on around you, you’re probably wondering why the heck you suddenly need to worry about disconnected RDP sessions? Well, disconnected RDP sessions can be problematic for a number of reasons:
Potential Issues
Confusion
It can be confusing when you’re logging on to a server and notice that another user is logged on, even if it’s just a disconnected session. Are they doing important stuff? Can the other user by logged off? Or you’re trying to reboot a server, but the infamous “Other users are currently logged on … ” message pops up. Assuming you’re polite, you’ll reach out to the disconnected fellow to ensure that the user isn’t running any important apps. In short, it’s a pain. And we all have enough pain as it is, do we not?
Security
So this is the fun one and, I admit, the real reason for this blog post (no, it was not the confusion). Turns out that any local administrator has the ability to hijack any disconnected session on the server he or she is an administrator on. And while this doesn’t sound like a huge problem at first (if the user is already an admin, then he can do anything blah blah), there is a very specific scenario where this is a significant security risk. Being an admin on a machine is a great privilege, but hardly the key to the kingdom. If a domain admin (or even worse, an Enterprise Admin!) however logs on to the same host and disconnects their session, then the local admin can promote himself/herself to domain admin at no cost (EventSentry would of course detect this) by simply taking over the disconnected session. The process involves only a handful of commands, one of which is creating a temporary service (EventSentry would detect this also).
Resource Consumption
Processes from disconnected sessions do continue to run and consume some processing power, which can be an issue in environments with limited computing power or cloud environments where you pay for CPU time. It’s unlikely to have a large impact but it could add up for larger deployments.
I’m hoping that we can (almost) all agree now that disconnected sessions have a number of drawbacks, so let’s move on to ways to address this.
Solutions
User Education
While there are ways to forcefully log off idle users (see below), I think it’s important to make your admins aware that disconnected sessions pose a security risk, but also communicate to your users in general that disconnected sessions should be avoided when necessary. While this won’t solve the problem entirely, it will hopefully get some of your users to be more aware.
Notifications
EventSentry users can utilize filter timers to get notified when a user has been logged on to a server too long. Those not familiar with filter timers can learn more about them here. In short, filter timers work with events that are usually logged in pairs – such as a process start/process end, logon/logoff, service stop/service start etc – and can notify you when such an event pair is incomplete. For example, a logon is recorded but not followed by a logoff within a certain amount of time (the time period is configurable).
So in this scenario, a user’s logon event would start a filter timer that essentially puts the original event on hold. It won’t forward it to a notification just yet – only if the timer expires and no subsequent event has deleted the timer. A later logoff event (that is linked to the logon event via some property of the event like a login ID) would end the filter timer. If the logoff event does not happen (or happen too late), EventSentry will release the pending (logon) event and can send a notification to let either the team or the user know (if the login pattern matches email addresses) that the filter time has expired.
BONUS: If the Windows username can be matched to an email address (e.g. username is john.doe and the email address is john.doe@mycorp.com), then EventSentry can even send an email directly to the user (opposed to a generic email) reminding them to log off. Even better, the email can be customized and provide specific instructions. See KB 465 for instructions on how to configure this in EventSentry. The screenshot below shows such a customized email.
Disclaimer: This method doesn’t necessarily distinguish between users who are logged on and users who have disconnected sessions. It detects that a user has had a Windows session active for an extended time period. In most cases this should work reasonably well to detect disconnected sessions as well. especially with longer timer periods (since most users don’t need to be logged on to a server for many hours).
Forcibly logging off disconnected sessions
When you tried your best the nice way using education and emails and it’s just not working, well, then it turns out that Windows allows you to use the nuclear button – logging off disconnected user sessions after a certain time. Since this can be configured via group policy, implementing this is fairly easy. Simply create a new group policy and assign it to appropriate OU(s) and configure the following setting:
Local Computer Policy
|-- Computer Configuration
|-- Administrative Templates
|-- Windows Components
|-- Remote Desktop Services
|-- Remote Desktop Session Host
|-- Session Time Limits
Then simply enable the option and configure an appropriate time limit. The available timeout options are actually quite useful and range from 1 minute (which pretty much converts a disconnect with a logoff) all the way up to 5 days (which is better than nothing I suppose, but why even bother at that point?). What option you select here largely depends on how concerned you are about resource usage and sessions being hijacked. I personally would recommend a lower end of 30 minutes up to 8 hours at the maximum.
Validating RDP settings with EventSentry Validation Scripts
EventSentry includes two validation scripts that validate whether RDP session timeouts are enabled:
Disconnected RDP sessions aren’t an immediate security risk, since they require an intruder to somehow gain admin rights on a domain member machine first. Sophisticated attacks however rarely involve just one step, but usually take advantage of multiple vulnerabilities and exploits. Since there are few to no downsides to notifying or even logging off users after a certain amount of time, I would recommend to follow the recommendations outlined in this blog post.
Just when the Microsoft Exchange exploit CVE-2021-26855 thought it would win the “Exploit of the year” award, it got unseated by the – still evolving – Log4J exploit just weeks before the end of the year!
Had somebody asked Sysadmins in November what Log4J was then I suspect that the majority would have had no idea. It seems that the biggest challenge the Log4J exploit poses for Sysadmins is simply the fact that nobody knows all the places where Log4J is being used. Most exploits affect a specific piece of software or hardware equipment, but Log4J is not a separate product, it’s just sitting somewhere on the file system of some Java application – likely among a myriad of other open source libraries.
So how do you find vulnerable instances of Log4J on a network without spending hours running scripts and manually going through software inventory? And, how do you ensure that it stays that way, and a vulnerable version of Log4J doesn’t get installed at a later point?
The answer to that question are EventSentry’s validation scripts. If you’re running EventSentry 4.2 or later, then you can utilize its validation scripts engine to discover vulnerable instances of Log4J on your monitored Windows-based systems with the latest Log4j discovery script.
As a reminder, validation scripts validate best practices across your infrastructure, and are regularly updated by EventSentry to detect insecure settings, outdated Windows systems and potential exposure to exploits.
The Log4j discovery script can be assigned to any Windows-based host, and will scan the hard drive to discover all vulnerable Log4j libraries on a system. In most cases the script takes fewer than 5 minutes to run, and the results can be viewed in the Web Reports under Health -> Validation Scripts -> Status as soon as the script has finished executing. Simply filter on the script name on the Summary page or specify this query to filter the validation script results
script:"*Log4j*" AND passed:No
Please note that the speed with which the script executes may vary by host since the speed depends on the number files scanned, disk speed etc.
Since we’ve accumulated a lot of resources around EventSentry that are updated frequently, we’ve decided to launch a GitHub page where anyone can access and download scripts, configuration templates, screen backgrounds and our brand-new PowerShell module that is still under development.
We currently have 4 repositories available:
Scripts: Collection of scripts that can either be used in conjunction with EventSentry to enhance its monitoring capabilities or used independently to enhance security and automate tasks.
Configuration: Configuration templates for Ransomware and general security as well as a recommended Sysmon template.
PowerShell Module: The recently launched EventSentry PowerShell module supports the automation of a small number of EventSentry configuration tasks, such as managing hosts and groups, adding maintenance schedules and more. Note that the PowerShell module only supports a small number of tasks at this point. Feel free to request additional cmdlets via support.
Screen Backgrounds: 6 different desktop backgrounds that you should immediately apply to the desktop of your EventSentry server.
Of course we encourage collaboration, especially in the scripts and configuration repositories. Please contact us if you have any questions.
Anybody who’s looked for answers on the Internet has likely stumbled across a “TOP X LISTS”: The “10 things famous people do every day”, “Top 10 stocks to buy”, the “20 books you have to read” are just some examples of the myriad of lists that are out there offering answers. You may have even stumbled upon a few “Top 10 (or 12) Events To Monitor” articles too.
Why We Love Top 10 Lists and 10 Reasons Why We Love Making Lists provide some insight as to why these types of articles keep popping up all over the place for just about any topic. And it makes sense when you think about it! You’re facing a new problem/challenge you presumably know little about, get a “Hey, just do these 10 things!” list back and: Done.
But while “Top 10” lists are surely useful for a variety of topics (“Top 10 Causes of House Fires“), they are less useful when it comes to identifying event IDs to monitor. Why? Because auditing 10, 20 or even 30 events is just not enough to detect suspicious activity or help with forensics. Just consider that Windows 2019 potentially logs over 400 different events to the event logs – almost 3 x as many as Windows Server 2003 did. Sure, in practice Windows 2019 likely only logs some of these 400 events, but even a minimalist would probably agree that monitoring fewer than 10% of all events is probably not going to give you a whole lot of visibility into your network.
But before we go any further, let’s distinguish between auditing and monitoring. Enabling auditing tells a system to constantly create a trail of activity that can later be analyzed – either manually or by software. Monitoring on the other hand means that you’re actually doing something with those events – whether that’s storing them in a different location, analyzing them or getting email alerts.
But enabling auditing (correctly) is always the first step that any subsequent process builds on. And, enabling auditing is not only free but generally doesn’t impact system performance either (the only exception are large event logs that can affect memory usage).
Yet the sad reality is that many organizations out there are still not properly auditing their Windows servers. A system that’s not auditing its activity gives you neither the ability to respond to important events, nor does it let anyone retrace the steps of attackers after an intrusion has occurred (forensics). So let’s repeat: Regardless of whether you have a monitoring solution in place or not, or are planning on getting one, auditing should always be on and needs to be the first thing you do.
Auditing alone is, of course, no longer sufficient to maintain a secure network, and not only because clearing your event logs is one of the first thing intruders do after they attack. As the developers of EventSentry we’re obviously a little biased, but with proper event log monitoring in place, you can:
Store events in a secure location, safe from tampering & deletion
Correlate events across multiple hosts
Receive real-time alerts for critical events
Detect suspicious behavior
and more. So, whether you’re in charge of a grocery store in Idaho, a government contractor in Virginia or in charge of distributing oil for the Eastern U.S., the events below should always be monitored:
Now, going all out and monitoring 85 events as the baseline may seem crazy and overkill – after all you’ve never seen a “Eat these 85 foods to be healthy” list – but let’s remember the 4 reasons you should monitor these events:
You can never retroactively enable auditing. More is better.
Most of these 85 events log events infrequently.
Attackers don’t want you to enable auditing.
The Internet is crazy.
And just when you think you’re good, one needs to point out that even auditing these 85 events is not sufficient if you have to be compliant with regulations like CMMC, PCI and others (if you need to be compliant then I recommend our free validator here). And here are 3 great reasons those events are a good baseline:
They document changes made to the OS (e.g. scheduled task added)
They report a security issue (e.g. group membership changed)
They are logged infrequently and thus won’t spam your event log(s)
To activate these audit settings, either run the auditpol commands at the bottom of the list on all hosts or, a much better option, setup a group policy that will ensure these settings are always enforced across the entire domain/forest. The linked page includes instructions on how to import the necessary audit settings into a GPO, but here they are just in case:
Open the “Group Policy Management” application
Navigate to the “Group Policy Objects” container of the applicable domain
Right-click the container and add a new GPO object with a descriptive name (e.g. “Mandatory Auditing”)
Right-click the newly created GPO object and select “Import Settings”
Proceed with the wizard and point the “Backup Folder” path to the folder where the zip file was extracted to
The GPO object will now contain all audit policies for all events listed above
Link the GPO to the domain or select OUs
Larger networks may require different audit settings depending on server role, location and security level which may result in more complicated group policies. Remember that EventSentry can keep track of your audit policies to make sure your policies are accurate.
