Category: EventSentry

Cleaning up Disk Space and automatic fragmentation reports via email

ingmar.koecher 0 Comment EventSentry, Tips & Tricks, Tools & Utilities

Even though disk storage is cheaper and faster than ever, for some reason I still run into disk space problems on occasion. The most common disk space problems I run into is a full C drive (why would you need more than 4Gb for the OS?) or a database that grows too large.

We have found and utilized several tools over the past years and I am going to share some of my approaches to quickly identify space hogs, free up disk space and deal with fragmentation.

Once a machine is low on disk space one will usually want to find out which files use up the most space and move them to a new volume or send them to data heaven for good. There are a lot of tools out there that visualize disk space consumption on a volume, but my favorite by far is Windirstat. Windirstat uses a treemap which displays every file in a colored rectangle and was inspired by KDirStat from Linux (the original author really wants to make sure you know what the original is). The size of the rectangle is proportional to the file size, so you will either want to look for clusters of many small files (e.g. to spot lots of unneeded temp files) or for large rectangles to identify any files you might not need anymore. I find it incredibly easy to spot files that can be safely deleted with Windirstat. The screenshot below shows what Windirstat looks like on my Vista laptop with a 64Gb HD.

Windirstat ScreenshotOf course, just running Windirstat alone doesn’t mean that you will be able to find files that can be safely deleted. But once you have identified files that do occupy significant amounts of disk space, you can engage in research to determine whether these files can be compressed, moved or deleted. Usual candidates are the temporary files, pagefile, IIS log files, NTBackup temp files and temporary installer MSI files (more on that below).

Windows will sometimes cache and leave installer files on your C drive, even when the application is no longer installed or has been upgraded. Depending on how long ago the OS was installed, this can be between a few hundred megabytes or nothing at all. You can delete those so-called “orphaned cached Windows Installer data files” with the msizap.exe utility, using the G command-line switch. Using msizap has been the last resort for me a few times, freeing up significant space on the C drive of servers when nothing else could be moved or deleted. Msizap is part of the Windows Installer 4.5 SDK which can be downloaded from http://www.microsoft.com/downloads/details.aspx?FamilyId=6A35AC14-2626-4846-BB51-DDCE49D6FFB6&displaylang=en. The screenshot below shows msizap in action.

Msizap ScreenshotNow, after cleaning up all the space we’d want to defragment our drive as well, right? Disk defragmentation software has been around since MS-DOS, but people are still debating whether defragmentation software, especially commercial one, is worth the effort. It is sort of like taking multivitamins – it most likely doesn’t hurt but there is no clear indicator that it cures diseases or makes you feel better after taking them.

MS-Dos DefragSome operating systems, most notably Linux and Mac OS X attempt to prevent fragmentation as much as possible and don’t even include defragmentation software, but the Windows software market is awash with both free and commercial defrag software. If you are interested in learning more about fragmentation and its cause, then Wikipedia has an article about defragmentation.

So is it worth it to use or invest in commercial defragmentation software? I think it depends. I have used various defragmentation programs over the last 10 years or so, and have seen one case where a MSSQL database on an extremely badly fragmented partition had became so slow that it was essentially unusable (yes, the database was for EventSentry 🙂 ). Defragging this partition with PerfectDisk solved the problem and the database was performing well after defragmentation. So yes, fragmentation can be bad if it gets out of control, though this is the only time I remember a defragmentation having such a significant impact. If you have a partition with little disk space available and lot write activity, then it will probably make sense to continuously defrag the partition to ensure optimal performance. Otherwise I think it is a luxury and will not yield significant disk performance benefits.

You can also use the -a switch of the defrag.exe utility that ships with Windows to analyze a drive and get basic metrics as to whether the drive should be defragmented or not. However, if you have a lot of machines then running defrag.exe on all of them manually can be tedious, especially since you would need to do that on a regular basis (e.g. monthly). Fortunately, you can use EventSentry‘s application scheduler feature to automate this task in three simple steps (in this example we will focus only on the system drive). Since the application scheduler logs output from any command-line utility you run to the event log, we can actually get an email when Windows thinks that a drive is fragmented.

  1. Create an embedded script (e.g. DefragCheck.cmd) that runs “defrag.exe %Systemdrive% -a -v“
  2. Create a system health package and add a new application scheduler object to it – making sure that both check boxes in regards to error levels are checked. Pick the embedded script @DefragCheck.cmd and schedule it to run. Everytime defrag.exe is executed, EventSentry will log an event to the event log with the output of defrag.exe.
  3. Create a new event log package and add a filter that matches the events generated (Log=Application, Source=EventSentry, EventID=10200) and additionally looks for the string *You should defragment this volume*.

Voila – now you will get an email every time defrag.exe determines that a drive is fragmented – and only if it’s fragmented.

Defrag.exe is of course only one of the many utilities out there that can determine fragmentation, and you will likely get different results from different utilities. For example, it’s very likely that defrag.exe tells you that a drive is not fragmented, when a different software (e.g. PerfectDisk) will tell you otherwise.

One scenario where you definitely do NOT want to use defragmentation software is on SSD drives, as they usually don’t suffer from the same random access delays and defragging will reduce the lifespan of the drive.

Best,
Ingmar.

EventSentry v2.90: Compliance Tracking for SOX, PCI, GLBA, HIPAA, FISMA, COBIT, …

ingmar.koecher 0 Comment Announcements, Event Log, EventSentry

This is round two in the new features available in EventSentry v2.90, and this time I’ll be covering the new compliance features.

Even though EventSentry was not originally designed to help with compliance, its event log consolidation capabilities made it an effective and economical solution to help our customers with their various compliance efforts throughout the years.

But while being able to filter and search through security events is helpful, it is not enough to quickly create reports that group information based on key elements, such as user creations, group modifications, policy changes and more.

In version 2.90 we addressed this by creating the new Compliance Tracking features which are based on the previous Tracking features.

This means that in addition to the “standard” event log consolidation that simply collects events and records them as is, compliance tracking intercepts specific events (e.g. account creation, logon/logoff, process creation), parses them, extracts the required information and records the relevant information in the EventSentry database.

Compliance Tracking covers the following auditing areas in Windows:

  1. Process Activity
  2. Console & Network Logons
  3. File Access Activity
  4. Account Management (User, Group & Computer accounts)
  5. Policy Changes
  6. Print Jobs

For example, finding out which group memberships changed over the last week is matter of two clicks in the web reports – and restricting a report to only reflect a particular group and/or action is just as easy.

But let me briefly outline the benefits of the individual tracking features:

Process Tracking
This feature records all process activity and lets you know which processes where started when, by whom, for how long and from which computer. This feature is not only useful for security purposes, but also helpful when troubleshooting or requiring statistical information (e.g. how often is PowerPoint being run).

Logon Tracking
This component tracks everything logon-related on your network, including console, successful as well as failed network logons. Using the console logon tracking for example, you can generate reports that show what time users logon and logoff, including from which computer, whether they are local admin and more details. Using the new network logon tracking, you can track successful as well as failed network logons. The included reports can reveal information such as which users logged on with a failed password, logon protocol distribution, most common reason for failed logons and more.

File Access Tracking
This feature is new in v2.90 and tracks all successful file access activity that has been enabled on files or directories. EventSentry does this by intercepting audit events that are generated when files or folders which are being audited. Since Windows Server 2003 and earlier don’t actually audit when objects are changed, but instead only audit the requested file access (click here for a related post), EventSentry can perform additional checks and verifications to complement the native auditing capabilities of the OS – such as checksum creation. Of course EventSentry also gathers additional information – such as the source computer from where a change was made.

Account Management Tracking
Also new in v2.90 is account management tracking, which encompasses user, group and computer account management tracking. This feature really makes life easier when you deal with large quantities of user, group and / or computer account changes.

For example, tracking a users group membership changes – even across computers and domains – is only a few mouse clicks away. Do you need to know which computer accounts were created in the last week in your domain? This only takes three clicks in the web reports.

Policy Change Tracking
Another feature added in v2.90, policy change tracking records the following “policy” events:

  • Domain Policy Changes
  • Audit Policy Changes
  • Kerberos Policy Changes
  • User Right Changes
  • Logon Right Changes
  • Trust Relationship Changes

Again, getting information about any of the above scenarios is extremely easy – such as seeing which user/logon rights were assigned in the last week or on which server the password policy was changed in the last 2 weeks.

Since none of tracking features are limited to hard-coded reports but instead are easily adaptable, they not only make your auditors happy – they provide you with valuable information. This allows you to utilize EventSentry not only for compliance but many other tasks, whether is security-related, for troubleshooting or something else.

As always, please see the documentation for more information. You can take a look at version history as well for a complete list of changes and new features in the 2.90 release of EventSentry.

Enjoy,
Ingmar.

EventSentry v2.90: Event Log Monitoring Changes

ingmar.koecher 4 Comments Event Log, EventSentry

Since we have just released EventSentry v2.90, we’ll be blogging about the improvements and new features in the coming weeks. Since event log monitoring is how it all started, my first post in this series will be about the improvements and new features in our event log monitoring engine.

Vista/Windows 2008
The biggest change in v2.90, in regards to event log monitoring, is of course the native support of the Windows Vista and Server 2008 event log API. As many of you know, Microsoft introduced a new API for event log monitoring while still keeping the legacy API in place for applications that don’t support the new API yet.

EventSentry v2.81 uses this legacy API with some work-arounds to monitor the new event logs, but I highly recommend upgrading to v2.90 if you’re monitoring Server 2008 and/or Vista event logs. Upgrading will result in less overhead and better formatting and presentation of events since the agents now access the event log with the native API. Naturally, the event log backup feature will backup event logs in the new evtx format on Vista/Server 2008 computers.

The new version also supports the new Operational event logs which are displayed under Application and Services Logs/Microsoft, for example the Microsoft-Windows-Backup/Operational log.

eventlogblog_290_eventviewer_1.pngThese operational logs need to be configured as custom event logs in EventSentry, by specifying the full path (e.g. Microsoft-Windows-Backup/Operational) as the name of the custom event log.

Please see one of my previous posts about the event log changes in Vista (which also applies to Server 2008) for more information.

Note that support for the new event log API is transparent, and there is still only one executable of the EventSentry agent for all versions of Windows.

64-Bit
EventSentry v2.81 did not format some events on 64-bit editions of Windows correctly, and we have resolved this problem in 2.90 which renders all events on 64-bit machines correctly. The EventSentry agent still runs as a 32-bit application in 2.90, but we have long-term plans to supply a 64-bit agent for x64 operating systems.

Filter Timers
Filter Timer filters allow you to ignore events that would otherwise trigger an alert, if they are followed by another event within a preset time period. For example, if an event indicating that a critical service is stopped is being immediately followed by another event indicating that the service is running again, then a filter would allow you to suppress both events.

Previously however, filter timers had to be setup exactly for each event pair. This meant that if you wanted to use a filter timer for 5 services, then you would have to create 10 events. Starting with 2.90 you only have to create 2 events now, as long as the first event and the clearing event share the same order of insertion strings – which is usually the case.

Please see the documentation for more information.

Action Trigger History
Selected actions (e.g. email, pager) now include the ability to log their trigger history – that is every time they are triggered by an event – to the database. This helps you confirm that a notification was in fact performed, and also gives you the ability to gather statistics about which actions are being triggered and how often.

eventlogblog_290_actiontrigger_1.pngThe action trigger history includes the following information:

•    Date/Time
•    Computer
•    Action Name, Action Recipients
•    Event Log Package, Filter Name
•    Event Log, Event Source, Event ID, Event Number

Please see the documentation for more information.

Web Reports: Error Explanation
Many events from the security event log, for example audit failure event 675, contain error numbers and failure codes inside the event that require you to research them in order to find out what they mean. Here is an example:

eventlogblog_290_event_1.pngYou can see that the failure code of 0x25 in itself doesn’t reveal too much, but if you view the same exact event through our web reporting, then the failure code is automatically explained for you:

eventlogblog_290_event_2.pngAs you can see in the screenshot above, the Kerberos failure code of 0x25 is automatically explained as “Clock skew too great”.

Copying / Pasting event details from Emails
If you have been using EventSentry for a while, then you’ve probably setup event exclusions more than once, most likely after receiving an email from one of the agents. Starting with 2.90, you can now copy the event in your email client and paste it into a new filter. The management console will parse the event properties and automatically fill in the following fields for you:

•    Event Log
•    Event Severity
•    Event Source
•    Event Category
•    Event ID

Please see the documentation for more information.

You can take a look at version history as well for a complete list of changes and new features in the 2.90 release of EventSentry.

Enjoy,
Ingmar.

NTFS Alternate Data Streams: Hiding data in plain sight since 1993

ingmar.koecher 0 Comment Event Log, EventSentry, Tips & Tricks, Tools & Utilities

I recently read an article in the Hackin9 magazine (worth taking a look if you haven’t heard about it) about alternate data streams (ADS) in NTFS. I had heard about this hidden feature in NTFS a long time ago actually, but over the years forgot about its existence again.

Background
In a nutshell, the NTFS file system, which was introduced with Windows NT 3.1, supports ADS – sometimes also referred to as “hidden streams”. This means that you can attach or associate any number of files to an existing file, yet those files will not be visible to the vast majority of file management applications – including explorer and the “dir” command (Vista can show ADS with a parameter). One thing I find interesting about streams is that a lot of people in IT do not seem to
know about them, even people otherwise very familiar with the Windows
Operating System.

Now, streams are created and accessed by appending the “host” file name with a colon, followed by the name of the stream. Let’s say you want to create a text file called financials.txt and hide it with winhelp.exe, you would run notepad C:\Windows\winhelp.exe:financials.txt. This will bring up notepad which will prompt you to create the file since it doesn’t exist (since the alternate stream is basically a file). You can then save any text in the hidden file and save it. You will notice that the file you just created will not show up when you do a directory list (dir C:\Windows) and will also not show up in the Windows Explorer. Note that the timestamp of the host file will change however.

Now there are of course a variety of utilities that have been developed in the last 15 (!) years that will allow you to find hidden streams, but more on that later. Hidden streams still exist on Vista and later, though the feature seems to have become more restrictive.

There are apparently no limits as to how many streams one can associate with a file, or the type of file that can be associated. This means that you can associate an executable as much as you can an ASCII file. There are however some limitations as to how user mode applications (e.g. notepad) can access hidden streams. Let’s go back to the previous example where we created the file financials.txt in winhelp.exe. If you open a command prompt and execute type C:\Windows\winhelp.exe:financials.txt, then you will not be able to see the contents of the hidden file. If you use notepad instead however, you will be able to see the file (notepad C:\Windows\winhelp.exe:financials.txt). This is probably because cmd.exe and its built-in commands up until Windows XP are not aware of alternate streams. ON a Windows XP machine I also could not open that same file if I tried to open it from inside notepad with the File -> Open command.

Creating Streams
Things get more interesting when you attach executables to files – and execute them! Let’s say I wanted to hide popular windows game solitaire inside the file C:\Windows\wganotify.log and call the stream “calc.exe”. Here is what you do:

type system32\sol.exe > C:\Windows\WgaNotify.log:calc.exe
start C:\windows\WgaNotify.log:calc.exe

Auditing Alternate Data Streams
Those of you interested in auditing will probably wonder how Windows tracks access to hidden streams in the event log. Well, there is good and bad news. The bad news is that object tracking (the famous event 560) does not show hidden streams, and instead only shows the “host” file name being accessed. Process Tracking on the other hand shows hidden streams in the expected manner. For the above example, a 592 event will show that file C:\windows\WgaNotify.log:calc.exe was executed.

Exploiting Streams
Scary, huh? This opens up a can of worms when you think about malware hiding inside otherwise innocent files – such as a log file. At appears as if most AntiVirus products do not detect hidden streams, at the same time there doesn’t seems to be a significant number of mainstream malware applications out there are that rely on hidden streams. I’m not sure why that is, since this feature seems almost too good to be true for the writer of any malicious applications. One reason might be that malware writers mostly target home machines, and many of those computers are still formatted with the FAT(32) file system, which of course doesn’t support ADS. This might change over time though, as more (home) computers use NTFS as their file system.

So after reading up on ADS, playing around with it last week, scanning my computer for hidden streams, I arrived at the inevitable question: What is the higher purpose of Alternate Data Streams? I mean, many applications don’t support it, most people don’t know about it, and a scan didn’t reveal any hidden streams besides a couple inside some Microsoft installers that apparently use them as some sort of meta data.

As it turns out, ADS was created for compatibility with the Macintosh HFS file system, which uses a data fork and resource fork to store data in a file (OS X now uses the HFS+ file system). But over the years (it’s been 15 after all) some developers at Microsoft decided to utilize this feature. For example, when you specify summary information about a file (right-click -> properties -> summary), then this information will be stored in ADS.

As mentioned earlier, there have been some improvements in regards to ADS with Vista and later. Vista can now show alternate streams with the /R switch of the “dir” command. My preliminary research also shows that hidden streams can no longer be executed in Vista or later – so what we did in the above example will not work. I think that’s a good thing, since there really is no practical reason (unless you develop malware) to do this. The screen shot below shows the output of a regular dir command and the dir /R command on a Windows 2008 server (note the file setupact.log).

ADS_Win2k8.jpgIn my humble opinion, Microsoft should get rid of alternate streams in future versions of Windows, and instead come up with some sort of structured way of embedding meta data in files. Anything contained in meta data should be non-executable and limited in size, e.g. 256kb.

Discovering Streams
So what does all this mean for you, the person responsible for security in your network? How can you find hidden streams and detect if streams are being added to files?

There are many free third-party utilities out there that show and manipulate hidden streams, but the discovery of this feature led us to extend the functionality of the File Monitoring feature of EventSentry to include the automatic detection of hidden streams in real-time. This means that any stream added, modified or removed from a file in a monitored location will be detected by EventSentry.

We have also developed a new command-line tool, adslist.exe, that will list all alternate data streams on a directory and optionally its sub directories. The tool is part of the NTToolkit v1.96 and I recommend that you schedule to run this tool with the Application Scheduler feature of EventSentry on a regular basis, or schedule it with the Windows Task Scheduler and email the results (adslist.exe C:\ /s). The advantage of using EventSentry is that the results of adslist.exe can automatically be emailed to you only if alternate streams were found. You can do this because the %ERRORLEVEL% is set to 1 by adslist.exe when one or more streams are found. The screenshot below shows what this would look like in the email sent by EventSentry:

EventSentry_ApplicationScheduler_ADSList.pngManipulating Streams
While Microsoft doesn’t offer a tool to search for and discover alternate data streams, they do offer a good explorer-extension that allows you to view and delete alternate data streams. You can download it from https://docs.microsoft.com/en-us/sysinternals/downloads/streams, the zip file contains the source code as well as another utility to create hard links on NTFS volumes. After extracting the archive, navigate to the \StrmExt\ReleaseMinDependency folder and run regsvr32 StrmExt.dll. You will then have an additional tab when viewing file properties in explorer called “Streams”:

StrmExt.jpgAnother way to get rid of hidden streams is to copy a file to a FAT[32] volume and then back to the NTFS volume, or – if you don’t have a FAT[32] volume available – simply compress and uncompress the file again.

Well, I hope this gives you a better understanding of alternate data streams, even if you were already familiar with them. Like I mentioned earlier, it doesn’t appear as if ADS is used for evil in a large scale quite yet (so no reason to panic!), but I believe it is better to be safe than sorry.

Event 4964: Special Groups Feature for Vista + Windows 2008 Entrepreneurs

ingmar.koecher 0 Comment Event Log, EventSentry, Pure Knowledge, Tips & Tricks

There is certainly a lot of talk about the benefits of using Vista, but a lot of administrators and users seem to be avoiding it and instead hold on to Windows XP – which now appears to have a better reputation than ever! Well, here is a small reason to upgrade to Vista or Windows Server 2008.

Microsoft introduced a new event, 4964, called the Special Groups Feature. The purpose of this feature is to log event 4964 to the security event log when a member of a group you specify logs on to a computer.

So let’s say you want to know when a member of a local Administrator group logs on to a computer (and with EventSentry you could get an email when that happens for example), then you can accomplish that with the special groups feature.

In order to use this feature you need to do three things:

  • Determine the SID of the group(s) you want to monitor
  • Specify the SID(s) of the groups you want to monitor in a registry key
  • Ensure that you are auditing the Special Logon Feature (enabled by default)

One way to obtain the SID of a group is to use the getsid.exe tool which is part of the Windows XP SP2 Support Tools and other Microsoft Resource Kits. Note that the primary purpose of this tool is to compare the SID of two user accounts (so it requires you to specify two user/group accounts), but you can just enter the same group name twice to get around this. Here is an example output of the tool:

getsid \\mydc “Domain Admins” \\mydc “Domain Admins”

The SID for account BUILTIN\Domain Admins matches account BUILTIN\Domain Admins
The SID for account BUILTIN\Domain Admins is S-1-5-21-9817441204-4587651373-9817264971-512
The SID for account BUILTIN\Domain Admins is S-1-5-21-9817441204-4587651373-9817264971-512

As you can see you need to point to tool to computer where the group exists, in our case I used a domain controller since I want to monitor if somebody from the Domain Admins group logs on to the computer. If you monitor a built-in group (e.g. Administrators) then you will see that the SID is much shorter and the same across all your computers.

Now that we know the SID, we can specify it in the registry. Navigate to key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Audit and create a new String with the name SpecialGroups.

The value for this new string will be the SID of the group you want to monitor, and you can separate multiple SIDs with a semicolon. For example:

S-1-5-32-544;S-1-5-32-123-54-65

You do not have to reboot after making this change, it is effective immediately with the first subsequent login. The event that is being logged will look similar to this (screen shot from the EventSentry Web Reports):

Special Groups Logon 4964 ScreenshotThe relevant information is shown in the lower part of the event in the New Logon section. Security ID shows the user that logged on, and Special Groups Assigned shows the group the account is a member of (of course this group has to be specified in the registry).

Voila. This feature probably makes most sense on critical servers, though I would recommend enabling it on all workstations as well since you probably want to know if a member of the local Administrators group logs on. But of course this also means that you need to be running Vista on your network :-).

Since this feature needs to be activated using the registry, you can use AutoAdministrator to push this registry change to multiple computers. AutoAdministrator has actually been rewritten from scratch and we will be releasing a new version 2.0 very soon.