Author: ingmar.koecher

Can the EventSentry Agents cause the same outage & disruption like the CrowdStrike Falcon sensor did?

ingmar.koecher 0 Comment EventSentry, Pure Knowledge, Security ,

The faulty Rapid Response Content CrowdStrike update that disabled millions of Windows machines across the globe on 7/19/2024 was any IT professional’s nightmare. Having to manually visit and restore each affected machine (further complicated by BitLocker) severely limited the recovery speed, especially for businesses with remote locations, TVs, kiosks, etc.

Of course, we’re all used to seeing bugs in the OS, our phones, and third-party software—but the impact is usually much less severe than what was observed on that fateful Black Friday. Prior to this incident, the majority of IT staff probably never imagined that an outage like this was even possible (and it appears as if CrowdStrike’s management team didn’t either).

It doesn’t come as a surprise, then, that many who were directly affected by this bug are seriously concerned about the software running on their networks—especially monitoring and security software like EventSentry that sits on most endpoints. How many more ticking time bombs are there that can take out everything within minutes? Many customers who use CrowdStrike also use EventSentry and are naturally wondering whether the EventSentry agents also have the “capability” to cause a BSOD.

The good news is that, since EventSentry is a user-mode service that does not directly run any code inside the Windows Kernel, it cannot cause a system crash like the CrowdStrike Falcon sensor did. A similar bug in the EventSentry agent would “merely” cause the EventSentry agent to terminate (“crash”) and normally restart automatically.

All EventSentry updates (including patches) are tested on our supported operating systems with a variety of configurations prior to release. This ensures that potential bugs affecting OS stability are identified before they are released to customers.

However, EventSentry is a highly configurable monitoring suite that provides users with myriad ways to customize it—for example, through the ability to call custom monitoring scripts for monitoring and remediation. As such, it’s vital that access to the EventSentry management console and configuration is monitored and restricted via Windows permissions. By default, EventSentry:

* only gives local Administrators access to the configuration
* logs every launching of the management console
* logs every time the configuration is changed
* agents log an event when a new configuration is applied

Nevertheless, bugs in user-mode software can still negatively impact a monitored system. For example, a user-mode process can use up all CPU time or exhaust all available memory, slowly causing a system crash. User-mode processes can also impact the kernel indirectly, for example, by opening (and not closing) extremely large numbers of thread handles that are allocated in the kernel nonpaged pool.

Exhausting the nonpaged pool (which can only be stored in physical RAM) can also cause a BSOD. For example, the C++ code below, when executed on a system, will slowly bring it to its knees until it’s either unresponsive or crashed. Windows Server has no built-in protections to prevent this from happening.

unsigned int ThreadTest(void *dummy)
{
    while (1)
        { Sleep(1000); }
}

int main(int argc, char **argv)
{
    std::vector<HANDLE> threadHandles;

    while (true)
    {
        HANDLE hThread = CreateThread(NULL, 10240, (LPTHREAD_START_ROUTINE)ThreadTest, NULL, 0, NULL);
        if (hThread != NULL)
            threadHandles.push_back(hThread);
    }
}

These type of “subtle” bug takes time, however, and EventSentry’s extensive performance monitoring features would detect such abnormal resource usage on a monitored system in various ways.

EventSentry also doesn’t use content files like CrowdStrike does. Instead, rules are shipped through package updates which:

* Do not get shipped automatically and require the user to open the management console.
* Do not have the ability to crash the agent or the system.

It’s important to understand that developing software products—especially when used within the increasingly complicated Windows ecosystem—is a complex and intricate process. Supporting multiple platforms, languages, options, and configurations further complicates it. While it’s unfortunately impossible to write perfect, bug-free software, working only with experienced developers and utilizing both automated and manual testing procedures can minimize the risk of disruptive bugs without affecting the evolution of the software product.

CrowdStrike’s Falcon sensor, due to its operation in kernel mode, is both a powerful and high-risk software product—similar to hardware drivers. It appears that CrowdStrike’s QA efforts for its “Rapid Response Content” were not proportional to the risk posed by the Falcon sensor, with extensive QA being applied only to Falcon sensor software updates. It is surprising and disappointing that a large software corporation like CrowdStrike, which sells an expensive software product, did not anticipate this risk ahead of time and instead offloaded the risk to its users. Software vendors can all learn from this disconnect and expand their QA efforts beyond just software code.

Adding insult to injury is the fact that the current CEO of CrowdStrike, George Kurtz, was CTO of McAfee when that software also crippled millions of computers back in 2010. It did this by marking a core Windows executable – svchost.exe – as being infected with a virus and deleting it.

EventSentry 5.1.1.104: Security, Security, Security!

ingmar.koecher 0 Comment Uncategorized , ,

Everybody wants to have a more secure network – and everybody has various tools at their disposal to at least improve the security of their network. But which tool is the best for the job, and where do you start? The answer to this question is somewhat easier (and more structured) for organizations that have to adhere to compliance frameworks (ISO, CMMC, PCI, SOC, …), but a little harder for business that have no such requirements.

EventSentry has long included many tools to increase the security of your network, and in the latest 5.1.1.104 update we made it significantly easier to increase the security of your infrastructure with 3 new dashboards:

The security dashboards will guide you through the process of:

  • Ensuring your audit settings are correct
  • Identifying insecure settings on your network
  • Illustrating significant changes that should be reviewed

So how do you get started? After applying the latest patch, head over to the web reports and load any dashboard and hit the SPACE bar – the new security dashboards will show up in the list and you can import them there.

After the dashboards are imported, click on the Dashboard in the menu and select the first one: “Dashboard [1] Foundation”. This dashboard simply evaluates all of your audit policies to make sure they are adequate. Proper auditing is crucial, and is the foundation for many other monitoring initiatives.

Important: You can click on the header of each imported dashboard to navigate to a KB article, which explains the purpose of the dashboard in detail and how to resolve any identified issues.

It’s recommended to configure audit settings via group policy, to ensure that settings are always enforced and apply network-wide. You will know that your audit settings are correct when all tiles on this dashboard are “OK”. Please keep in mind that it may take an hour before new audit settings are pushed by GPO to your end points and subsequently picked up by EventSentry. Detailed information about this dashboard is available here.

Once your audit settings are in the clear you can graduate to dashboard #2 – Attack Surface. This dashboard utilizes various validation scripts to ensure that your monitored hosts follow best practices and/or pass security and compliance requirements. The checks are generally divided into different categories like

  • Best Practices
  • Security
  • Privacy
  • CIS Critical Security Controls

Most findings on this dashboard can usually be resolved via group policy and/or registry settings. The end result is a reduced attack surface of your network since your monitored hosts will now adhere to recommended best practices and security recommendations. More information about this dashboard can be found here.

The last dashboard, “Security [3] Critical Changes” illustrates critical changes that have occurred on your network (and domain) recently. This dashboard should be reviewed daily and will reveal important changes like

  • Services/Drives/Scheduled Tasks added
  • Software Installed
  • Recent logon failures
  • Important AD changes

Patch 104 also includes improvements to the names of built-in event log packages, which have been consolidated and given better, more descriptive names. As such, expect a lot of changes if you do a package update in the management console.

Another useful improvement is that you can now easily access information about security events with a new “info” button in the management console.

If you’re running EventSentry 5.1 then download the latest patch and make sure to install & review the new security dashboards!

It’s Not Black Magic: Malware & Ransomware in Plain English

ingmar.koecher 0 Comment RansomWare, Security , ,
It’s Not Black Magic: Malware & Ransomware in Plain English

It was almost exactly 10 years ago in December 2013 that we wrote our first blog post about detecting CryptoLocker, which was the first sophisticated Ransomware attack of its kind back then. BTW, 2013 was the year of the Boston Marathon bombing, Edward Snowden leaking secret NSA information, Syrians fleeing their home country and Nelson Mandela passing away.

While it was somewhat unclear at the time whether Ransomware was here to stay, 10 years later we know that – unfortunately – CryptoLocker was just a prelude of many more bad things to come. Cryptolocker was a pilot episode, and we’ve been enjoying a never-ending Malware soap opera since then.

With the help of cryptocurrency among other things, Ransomware has turned into a lucrative business model, generating some 1 billion in ransomware payments in 2023. In other words, it pays to encrypt! Cha-Ching! So we can all agree that Ransomware will be here to stay as long as networks remain insecure and companies are willing to pay.

But despite its bad reputation, Ransomware does have one positive aspect – it will let you know that it’s there! And that’s in stark contrast to other Malware that often remains undetected within a network for long periods. Malware often lurks in the shadows and either waits for commands from its shadow master or slowly exfiltrates classified data – e.g. from a defense contractor. Nation states seem to utilize Malware both to plant time bombs as well as exfiltrate trade secrets. After all, why waste precious time with R&D when you can just download everything?

Many still don’t fully understand how Malware and Ransomware work. How does it get in? What does it do once it gets in? Can I still detect it after it established itself? Is my company a potential target? But whether it’s Ransomware or Malware (I will refer to them collectively as Malsomware here), it needs to be blocked or – at the very least detected as quickly as possible.

Over the next few weeks, we’ll be posting a number of articles that explain Malsomware and how EventSentry can help detect and uncover these intruders. We’ll start with a simple diagram that simplifies the journey of your average infection. The good news for EventSentry (v5.1) users, is that you can reduce the risk of Malsomware infections in every stage listed below, starting with the exploitation stage.

Reconnaissance is often the first step when preparing an attack on a target, but since it’s a manual process, it’s usually only done for high-value, targeted attacks. And even though these attacks are likely rare, it’s still a good idea to educate key staff on the risk and limit the public exposure of personal data. Facebook, LinkedIn, WhatsApp, and other social media sites & apps are all extremely valuable resources for attackers and help with social engineering attacks. Don’t underestimate the risk – doing research on a company and its individuals doesn’t take as many resources as one might think, and let’s not forget that our adversaries likely outnumber us significantly. Bottom Line: If your organization has valuable data or a lot of funds, then you might end up on a radar sooner than you’d like.

But when discussing risk – one important factor is often overlooked: Our own risk assessment for malsomware infections. How we (and collectively our organization) assess our own risk strongly affects how seriously we take threats, and how many resources we subsequently invest in defenses – say cyber security measures. This is human nature and makes perfect sense: If you don’t have the funds for a security system for your house (alarm, new locks, cameras, …) then you are likely to convince yourself that you don’t need this, and you live in a safe neighborhood – even if the data says otherwise.

Human psychology aside, after ten years of Ransomware one would think that all organizations should be aware of the elevated risk of cyber attacks, right? What’s missing in this equation, however, is that people and organizations usually don’t like to invest in projects that don’t yield an immediate ROI. As such, risk mitigation that is both expensive and complicated might backfire – and complicated it is. Do you get a SIEM? An XDR? MDR? EDR? Network Monitoring? Do you get multiple ones? Which one is the best? Which one fits your budget? It can take organizations months of meetings, quotes, evaluations & finally an implementation before an effective mitigation system is finally in place. Unfortunately for them, Malware actors don’t usually wait until you are ready & prepared to take them on.

Malware actors are casting a wide net – and most organizations today are at risk. If you’re managing public infrastructure or any government office, you’re at risk. But you’re also at risk if you’re working for a boring government agency, or just working *with* a government agency. Because attackers might target your “boring” or “insignificant” organization so they can later infiltrate the more exciting government network. Defense contractors are the prime example and the reason why CMMC has been created. Yes, your organization might create bolts, lights, or metal parts. But if you have a relationship or connection with a more valuable contractor, then hacking your network might help the attacker get into that more valuable target. Is your company generating significant revenue and profits? You’re obviously at risk for Ransomware. And so the list goes on – even if you work for a 10-person marketing company, you’re at risk for identity theft, botnets, crypto miners, and so forth.

In summary, while the risk level (and appropriate countermeasures) differs between different companies, every Internet-facing company is exposed to some level of risk. Stay tuned to learn more about how Malsomware attacks and how you can protect your network. Oh, and if you’re in charge of a Windows network, then head to system32.eventsentry.com right now and compare the recommended audit settings with the ones in your network. It’s easy, fast, and free and lays the groundwork for a comprehensive security solution down the road. You’re welcome.

Revealing Suspicious VPN Activity with Anomaly Detection

ingmar.koecher 0 Comment EventSentry, Monitoring, Security ,

Anybody who monitors logs of any kinds, knows that the extracting useful information from the gigabytes of data being collected remains one of the biggest challenges. One of the more important metrics to keep an eye on are all sorts of logons that occur in your network – especially if they originate on the Internet – such as VPN logins.

With the introduction of Anomaly Detection in EventSentry v5.1, filtering out suspicious activity – such as logons from previously unknown users or IP addresses – is now possible and can mean the difference between missing or detecting a malicious logon that could wreak a lot of havoc on your network.

What’s unique about EventSentry’s anomaly detection is that it works any type of data – whether it’s from an event (e.g. Windows logon), a log file (e.g. HTTP log) or a Syslog message (e.g. VPN login). As long as the data you are capturing follows a pattern that can be mapped to a regular expression (RegEx), anomaly detection should be able to analyze and report on it. And the best part EventSentry’s anomaly detection is that it works in real-time – alerts about suspicious activity are usually generated within seconds of the event occurring.

Since pfSense firewalls are popular and free, we’ll look at VPN logins in this example. Again, this technique can be applied to any other product, as long as it logs or sends activity to EventSentry and includes both a username and IP address in a single message. Now, let’s cut to the chase and start with the message the pfSense firewall sends when a user logs on:

Jan 13 20:36:00 openvpn[53530]: openvpn server 'ovpns1' user 'domain\username' address '25.22.29.248:12377' - connected

The goal of this exercise is to determine when a user (successfully) logs on from a different IP address, e.g. when their credentials have been compromised. By establishing a baseline and linking usernames to IP addresses, we can flag any previously unseen IP address as suspicious and investigate. This is of course the same concept we apply to other activity on your network – previously unseen processes, logons and so forth.

Back to our VPN message from pfSense. EventSentry supports writing select Syslog messages to the event log, something we’ll have to do in order for the anomaly detection to work. This is configured in the Syslog component of EventSentry’s Network Services, as shown in the screenshot below. You may already have this enabled on your network in which case you would want to add the highlighted line, but if you don’t then simply enable this:

Once enabled, EventSentry will log any Syslog message that matches any of the listed patterns to the event log with event id 500, similar to what’s shown below:

The message logged to the event log is almost identical to the message sent by pfSense, except that EventSentry adds some metadata to indicate where the message was sent from along with the associated facility/severity. Using the regular expression

syslog@(.*)\[kern\/user.notice\].*\suser\s'(.*)'\saddress\s'(.*)\:\d+'.*

we can extract the sender, username and IP address. As a reminder, you can experiment with regular expressions on regex101.com as shown below:

Advanced users can also test the regular expression in the management console, but the regex101 web site provides more feedback that can be helpful when troubleshooting. By creating an event log filter which looks for these Syslog events and applies the regex, the event can then be used as input for the anomaly engine, since it relies solely on insertion strings which are not present in Syslog events by default (technically they are, but not in the format that would be needed).

Note: Even though the filter is an Anomaly filter, the filter type will have to be temporarily set to Include so that the regular expression can be configured in the Advanced dialog. It’s a good idea to test the filter with live data before configuring the Anomaly settings – to ensure that the event you are trying to process matches your filter rule.

With the regular expression override configured, the filter can now be configured for anomalies, using insertion string #2 (the username) as the key, and insertion string #3 (the IP address) as the value. The configuration in the management console should look similar to what is shown below:

As shown in the screenshot above, the filter matches Syslog events from the openvpn server and creates anomaly patterns based on the username and IP address, with a learning period of 1 week.

This means that when a new key (=username) is encountered, the value (=IP address) is recorded and the learning period (1 week) starts. During the learning period, events from the same user are not marked as anomalies and the first and future IP address during the learning period are associated with the username. After the learning period, any new IP address reported will be considered an anomaly.

To summarize, the overall flow of events is shown in the diagram below again:

The final step is to create another filter which will perform the desired notification action when an anomaly is found. This step can be skipped if a method for reporting on anomalies is already present, for example:

  • Require an acknowledgement in the web reports (see below)
  • Send an email notification
  • Open a ticket

The screenshot below shows the advanced filter settings necessary to require an acknowledgement from anomaly events:

This approach can be used to detect anomalies with a variety of inputs, whether the data comes directly from the event log or from an auxiliary data source like Syslog or a log file. As long as the source can be normalized into data pairs that can be fed into the anomaly engine, suspicious network activity can be detected in real time with EventSentry.

EventSentry v5.1: Anomaly Detection / Permission Inventory / Training Courses & More!

ingmar.koecher 0 Comment Announcements, EventSentry, Monitoring , , , ,

We’re extremely excited to announce the availability of the EventSentry v5.1, which will detect threats and suspicious behavior more effectively – while also providing users with additional reports and dashboards for CMMC and TISAX compliance. The usability of EventSentry was also improved across the board, making it easier to use, manage and maintain EventSentry on a day-by-day basis.

We also released 60+ training videos to help you get started and take EventSentry to the next level. The videos will be available to everyone for free both on YouTube and in your account area.

Anomaly Detection

One of the best real-time event log monitoring engines got a lot smarter in v5.1 and can now detect unusual behavior like the following:

  • A user who never logged on to a server/workstation
  • A user connects from a remote IP from which he/she usually doesn’t normally connect from
  • A previously unknown process starts
  • A process loads a DLL it has never loaded before (requires Sysmon)

But the best part about the new Anomaly detection is that it’s customizable – so anybody can create their own anomaly rules based on their needs.

Anomaly filters utilize insertion strings of events, so they work best with well-formatted events from the security or Sysmon event logs.

EventSentry ships with a number of built-in anomaly filters, but watch the official training video to understand how anomaly filters work to create your own.

Database Maintenance & Web-Based DB Maintenance Jobs

Also new is the ability to schedule database maintenance jobs directly from the web reports – so it’s no longer necessary to create or edit (embedded) scripts, application or task schedulers just to keep your database size in check. Simply schedule a job in the web reports and review the results after the job completed. The maintenance jobs themselves have also been improved and will attempt to free up disk space after each purge cycle when using the built-in database.

We’ve also improved the database usage page, which now shows significantly more details about the database health, including trends to help identify features that are growing in size.

Dynamic Package Updates

Unlike validation scripts, which (can) automagically update in the background so you can enjoy the latest checks with the click of a button, package updates have been a bit slower and more clunky. But this is all in the past, as package updates use the same update engine like our popular validation scripts, and can now seamlessly keep select packages up to date.

As such, users with active maintenance agreements can expect a lot more package updates to be made available to help you detect and defend against emerging threats. This new functionality also allows you to apply filters rules in JSON format directly into the management console, e.g. from KB articles or blog posts.

Permission Inventory

Keeping NTFS file permissions clean and updated can be a tedious task, even when you follow best practices (e.g. create groups for each file share and assign permissions to those groups), it can be quite difficult to stay on top of all file shares and folders, especially in medium and large-sized networks.

To which files and folders does a user have access? Who has access to critical folders?

EventSentry’s new permission inventory enumerates configured folders and their permissions (on a configurable schedule – e.g. daily), and presents them in the familiar Summary/Detailed view, making it straightforward to answer exactly those questions.

Watch the official training video for more information.

Improved Features

Every database-enabled feature in EventSentry can now store data in multiple databases, something that was previously only available in select features. This will make it easier to store all collected data in an active as well as an archive database simultaneously.

ADMonitor can now send data through the collector, the only component in EventSentry that did previously not have support for the collector.

EventSentray

EventSentray now also shows the network utilization and sports a useful “Internet Test” dialog which can test various aspects of your Internet connection – something that’s especially handy for mobile endpoints (aka “laptops”). The official training video has more information on EventSentray and how to configure & deploy it.

Compliance

Compliance requirements like CMMC, PCI, HIPAA, NIST and others continue to evolve and affect more and more companies. As important as compliance frameworks are with providing structure and guidance, companies often find it difficult to apply what are often vague rules and requirements.

Our new CMMC v2 and TISAX reports/dashboards bring together many features in EventSentry and provide a real-time status and overview of your compliance status. And our new dashboards provide actionable data that can be used to increase the security of your network – not just check boxes.

Usability Improvements

While every EventSentry release tends to incorporate little tweaks and improvements, v5.1 features many usability and troubleshooting enhancements:

  • Insertion strings in filters (when available) now show the name of the string instead of just the number, making it a lot easier to create and manage complex filters.
  • The “Save Configuration” prompt in the management console would often be displayed even when no changes were made. This has been improved significantly and you should see this prompt much less often in v5.1, with more improvements planned in future versions.
  • A new “GoTo” button will jump to any item in the tree that contains the specified text.
  • Collector users can see collector health right in the management console without having to access the web reports, while the collector status page in the web reports also shows the collector latency and throughput of individual agents.

EventSentry Training

In addition to a EventSentry’s comprehensive documentation all users now have access to a 60+ free training videos – with a total runtime of over 10 hours – organized into playlists. All videos are accessible on our YouTube channel in the Playlists section, with a training section on eventsentry.com coming soon as well. We highly encourage you to browse the new training videos – they are useful for new users and experienced users of EventSentry alike. Please upvote videos you enjoyed, we’re also happy for any feedback!