How to REALLY monitor SMTP, POP3 and IMAP on Exchange 2003

Even though Microsoft Exchange Server 2010 has already been released, many organizations still use Exchange 2003. In this article I’ll explain how to thoroughly monitor the various Internet protocols that Exchange 2003 offers, including SMTP, POP3, IMAP (and NNTP for that matter). The reason why I’ll only be looking at Exchange 2003 is because there is a significant difference in architecture between Exchange 2003 and later versions.

It is a common misconception that you can effectively monitor the W3SVC service (commonly referred to as IIS, though IIS encompasses a lot more than just a web server) and other services provided through IIS, such as SMTP and POP3, by simply monitoring their associated service. It’s a misconception, because a given IIS-based service may contain multiple instances – most commonly the case with the World Wide Web Service which often hosts multiple independent web sites. The status of these instances can be controlled independently of the hosting service, though that service needs to be running of course.

Don’t despair though, most server-based windows applications, fortunately, can be monitored by ensuring that their respective service is – well – running. For example, to ensure that the Apache service is up, you “simply” make sure that the Apache service is running. The same goes for countless other services such as MySQL – even SQL Server (of course you can still detach individual databases in SQL Server).

Exchange 2003, due its partnership with the Internet Information Services 6.0, is different though. Yes, IIS and Exchange 2003 are tightly coupled, and if you intend to have your Exchange Server 2003 communicate with any other server using a standard Internet protocol such as SMTP, then you will need IIS.

eventlogblog_2010_01_exchange_iis_components.png

The screenshot above shows that the inetinfo.exe process hosts all the major services (bold name), and that each service can host one or more instance. For more details please see http://technet.microsoft.com/en-us/library/bb124674(EXCHG.65).aspx.

The three most common Internet services your Exchange 2003 server is running are probably SMTP, POP3 and IMAP4. While a lot of attention is being paid to the core Exchange services such as

• Microsoft Exchange Information Store (MSExchangeIS)
• Microsoft Exchange System Attendant (MSExchangeSA)

The services providing SMTP, POP3 and IMAP4 connectivity are usually similarly important, especially the SMTP service. Looking at the EventSentry service status page immediately reveals that the SMTP, POP3 and IMAP4 services are managed by IIS:

Thumbnail image for eventlogblog_2010_01_es_services.png

As you can see, IMAP4Svc, POP3Svc and SMTPSvc all use inetinfo.exe (Executable column) for their host process. So why is this important again?

Since all of these services support multiple instances INSIDE the service (inetinfo.exe), the host process will continue to run even when one or more instances inside the service are stopped. Since most installations only have one instance, stopping that one instance inside the service will still leave the service up and running. The effect of course is the same; the service is not available to the end users while the Windows service will happily continue to run.

A screen shot from the System Manager application shows instances listed inside:

eventlogblog_2010_01_system_manager.png

As you can see with the IMAP4 protocol, we have two virtual servers setup that are both hosted inside the “Microsoft Exchange IMAP4” service. To stubbornly illustrate my point further I took a screenshot that shows both IMAP4 instances stopped while the service itself is running:

eventlogblog_2010_01_system_manager_services.png

So I think we’re all in agreement now that monitoring the POP3, SMTP etc. services in Exchange 2003 is not enough if you want to ensure that these services are actually available. So how do we monitor all of these instances?
The easiest way is actually with a VBScript, which is included below. VBScript works well since the cscript.exe interpreter is readily installed on Windows 2003, so no additional installation of tools is required. The script enumerates all instances of a given protocol, and checks whether they are running or not. If at least one instance is not running, the tool will return 1, thus setting the ERRORLEVEL to 1.

This VBScript can then be embedded into EventSentry, which will then run the script at set intervals using the application scheduler, notifying you via email (with the proper filter setup) when an instance is stopped. There’s a screencast for that, you can view it at https://www.eventsentry.com/screencasts/eventsentry-application-scheduler/eventsentry-application-scheduler.htm. It shows you how to create an embedded script and setup EventSentry to notify you when the scripts returns an error. Note that the screencast uses an older version of the script which only monitored web sites (not SMTP, IMAP4, …), but the process of setting up the script with EventSentry is exactly the same.

You should be able to use the script as-is, just configure which protocols are monitored by adjusting the values in the “Define which protocols to monitor here” section. The script always prints all installed instances and their status, and any stopped instance is prefixed with an asterisk. Below is what an email from EventSentry looks like:

eventlogblog_2010_01_appscheduler_email.png

The line with the stopped instance won’t be yellow in the actual email, I just added this for readability. The script can also easily be modified to automatically start any stopped instances – simply add the line

Instance.Start

after line 102. This will still trigger an email (or error) to notify you that it was stopped, but a subsequent run of the script at the next monitoring interval should not trigger an error again if the start was successful.

A note of caution here though – I have seen the script hang indefinitely with this line added when an instance that is currently stopped can’t be started because it’s not configured correctly. Hence, it’s not included by default.

 

‘ Lists the state of all IIS protocols configured on the local machine
‘ and returns an %ERRORLEVEL% of 1, if at least one instance is not in
‘ the “Started” state.

‘ When scheduling this script with EventSentry’s application scheduler,
‘ make sure that the interpreter is set to “cscript.exe”

Option Explicit

Dim allInstancesAreRunning

Dim monitorSMTP, monitorPOP3, monitorIMAP4, monitorNNTP, monitorFTP, monitorWWW

‘ Define which protocols to monitor here

monitorSMTP  = 1
monitorPOP3  = 1
monitorIMAP4 = 1
monitorNNTP  = 1
monitorFTP   = 1
monitorWWW   = 1

‘ Define which protocols to monitor here

‘ ==================== EXECUTION STARTS HERE ====================
allInstancesAreRunning = EnumerateAllInstances

If allInstancesAreRunning = 0 Then
WScript.Echo vbCRLF & “WARNING: One or more IIS components are not running” & vbCRLF
End If

If allInstancesAreRunning = 0 Then
WScript.Quit 1
End If

‘ ==================== FUNCTIONS ====================
Function EnumerateAllInstances

EnumerateAllInstances = 1

If monitorSMTP = 1 Then
EnumerateAllInstances = EnumerateAllInstances And EnumerateInstances(“localhost”, “SMTPSVC”)
End If

If monitorPOP3 = 1 Then
EnumerateAllInstances = EnumerateAllInstances And EnumerateInstances(“localhost”, “POP3SVC”)
End If

If monitorIMAP4 = 1 Then
EnumerateAllInstances = EnumerateAllInstances And EnumerateInstances(“localhost”, “IMAP4SVC”)
End If

If monitorNNTP = 1 Then
EnumerateAllInstances = EnumerateAllInstances And EnumerateInstances(“localhost”, “NNTPSVC”)
End If

If monitorFTP = 1 Then
EnumerateAllInstances = EnumerateAllInstances And EnumerateInstances(“localhost”, “FTPSVC”)
End If

If monitorWWW = 1 Then
EnumerateAllInstances = EnumerateAllInstances And EnumerateInstances(“localhost”, “W3SVC”)
End If

End Function

Function MapServiceToInstance( Service )

If Service = “SMTPSVC” Then
MapServiceToInstance = “IIsSmtpServer”
ElseIf Service = “POP3SVC” Then
MapServiceToInstance = “IIsPop3Server”
ElseIf Service = “IMAP4SVC” Then
MapServiceToInstance = “IIsImapServer”
ElseIf Service = “W3SVC” Then
MapServiceToInstance = “IIsWebServer”
ElseIf Service = “NNTPSVC” Then
MapServiceToInstance = “IIsNntpServer”
ElseIf Service = “FTPSVC” Then
MapServiceToInstance = “IIsFtpServer”
End If

End Function

Function EnumerateInstances( Server, Service )
On Error Resume Next

Dim VirtualServerService
Dim Instance, InstanceID

EnumerateInstances = 1

Set VirtualServerService = GetObject(“IIS://” & Server & “/” & Service)

If Err.Number = 0 Then
InstanceID = MapServiceToInstance(Service)

For Each Instance in VirtualServerService

If Instance.KeyType = InstanceID Then

If SiteIsNotRunning(Instance.ServerState) Then
WScript.StdOut.Write “*”
EnumerateInstances = 0
End If

WScript.StdOut.Write Instance.ServerComment & ” (” & Service & “): ” & State2Desc(Instance.ServerState) & vbCRLF
End If
Next
End If

End Function

Function SiteIsNotRunning( nState )

If nState <> 2 Then
SiteIsNotRunning = 1
Else
SiteIsNotRunning = 0
End If

End Function

Function State2Desc( nState )

Select Case nState
Case 1
‘MD_SERVER_STATE_STARTING
State2Desc = “Starting”
Case 2
‘MD_SERVER_STATE_STARTED
State2Desc = “Running”
Case 3
‘MD_SERVER_STATE_STOPPING
State2Desc = “Stopping”
Case 4
‘MD_SERVER_STATE_STOPPED
State2Desc = “Stopped”
Case 5
‘MD_SERVER_STATE_PAUSING
State2Desc = “Pausing”
Case 6
‘MD_SERVER_STATE_PAUSED
State2Desc = “Paused”
Case 7
‘MD_SERVER_STATE_CONTINUING
State2Desc = “Continuing”
Case Else
State2Desc = “Unknown state”
End Select

End Function

 

You can also download the script from here.

Until next time,
Ingmar.

Announcing EventSentry v2.91

Now that EventSentry v2.91 has been released, I’m happy to have the opportunity to blog about our monitoring solution again.

The most significant new feature in EventSentry is the Health Matrix, a new way to see your network status in a space-efficient way. In fact, you can see the overall health status of your entire network on a single screen, even if it consists of hundreds of hosts.

We also made numerous other changes throughout the web reports, and added some exciting new filtering capabilities with our event log filters, as well as improved speed with the event log engine and file checksum generations.

EventSentry v2.91 also includes many minor improvements throughout the application, including service monitoring, process tracking and more. We have also updated EventSentry Light, and a new version will be released in the coming days after we have completed testing.

But now to the new features in version 2.91:

Health Matrix
In the health matrix, each host is displayed as a colored square, circle or rectangle, with the color indicating the overall health of the monitored computer. When all of the monitored components of a host are in an OK status, the color of the square is green. The color will change to orange or red when a problem is detected, depending on the number or severity of the issue.

clip0580.pngThe health matrix is highly customizable, for example both the size and shape of the icons can be adjusted depending on the size of the network (and your monitor).

clip0583.pngEvent Log Monitoring
In 2.91, the event log filtering engine was improved, resulting in reduced CPU usage of the event log monitoring component. Since the CPU usage of the EventSentry agent is already quite low, you will most likely only notice this improvement on hosts that generate an extremely large number of events, such as domain controllers.

Also new is the ability to filter events based on insertion strings in addition to just filtering based on the event message text. This means that one can now match individual strings inside event messages against strings, numbers, file checksums and group memberships. If you are not familiar with the term “insertion string”, then I highly recommend my previous post about event message files before you read on.

Consider the following hypothetical example: The environment-monitoring component of EventSentry logs event id 10908:

The temperature (78.21 degrees F) has fallen outside the configured range (60F to 76F).

which is defined as:

The temperature (%3 degrees %4) has fallen outside the configured range (%1%4 to %2%4).

This event obviously informs us, that the current temperature has exceeded a set limit. Now let’s say that we wanted to get an email when the temperature exceeds the limit, but also send a page when the temperature exceeds 90 degrees.

The new filtering feature allows you to do just that, by using the numerical comparison functionality with insertion strings (of course you would also need to set the hour/day properties). Assuming that you already have a filter in place for regular email notifications, you would simply setup an additional include filter that would evaluate insertion string 3 (%3) and only match if the number is above 90. See the screen shot below for the example. The result is a filter that only matches when then the temperature exceeds 90 degrees.

blog_es291_filter_1a.png2.91 also includes two more comparison options, file checksums and group membership. So, if an insertion string represents a filename (e.g. from a security event), then EventSentry can create a SHA checksum from the specified file and compare it with the value that you specified. Another example would be a security event that includes a username in an insertion string, in which case you could setup a filter that would only match if that user is a member of particular group you specify. Both examples are mostly applicable for security events, since those are most likely to contain either filenames or usernames.
Using file checksums, you can be notified whenever a user plays solitaire, even when the user renames the executable.

blog_es291_filter_2.pngSimply create a checksum of the file first using shachecksum.exe (included in the free NTToolkit, make sure you account for different OS versions and platforms) and intercept the corresponding 4688 event.

Service Monitoring

Service Monitoring now collects the username as well as the executable of a service. These additional properties are available in the web reports and in events generated, for example when the username of a service changes.

blog_es291_service_monitoring.pngSoftware Monitoring
Software monitoring has been overhauled in 2.91, and some limitations and bugs have been removed. On Vista, Win2k8 and later, Windows patches are now monitored and included in the software inventory. 64-bit software is now classified as such and searchable, and searching for installed Windows Updated patches has also been simplified.

SNMP Traps
EventSentry can now send version 2c and version 3 traps, previously only version 1 traps were sent by the agent. The SNMP trap daemon was originally set to be released as part of 2.91, but this feature has been pushed back to v2.92.

Web Reporting
We have made a number of improvements in the web reporting to make using our web-based interface easier:

•    Reports are now easily accessible from every page, in addition to the reports p
age.
•    The database usage page now shows the actual page name in addition to the table name.
•    The dashboard page has been overhauled
•    The network status page can be customized (performance counters & disks)

blog_es291_dashboard.pngMiscellaneous Improvements

There have of course been other improvements across the board, such as:

•    Notes can now be applied to computers
•    AD-linked groups can be sorted, and authentication properties can be set globally
•    Hardware monitoring now includes the IP address of an interface
•    Process tracking can capture the command line of a process
•    Logon tracking includes group information
•    File checksum generation has been optimized and will now use fewer CPU resources (affects file monitoring and file access tracking)
•    The minimum database interval for environment monitoring has been reduced to 5 minutes from 15 minutes
•    Software uninstallation events now include the same information as software installation events

If you have an active maintenance agreement, then this 2.91 release will of course be free of charge. If you are not already using EventSentry, then you can download a free 30-day evaluation version from https://www.eventsentry.com/downloads_downloadtrial.php.

Happy Holidays,
Ingmar.

Group Policy Software Deployment: Targeting the right computers with WMI filters

Group policy was introduced with Windows 2000, and is an easy way of centralizing many Windows settings. In addition to centralizing event log and firewall settings, I personally like the ability to deploy MSI-based software applications with Group Policy, since it makes it extremely easy to deploy new software packages.

Even though Software Installation only works only with MSI-based packages, it does make deploying MSI-based software packages extremely easy. Here is a short list of software (mostly tools for sysadmins) that you can deploy using Active Directory:

There are of course many more, and you can distribute most Microsoft client applications, such as Microsoft Office, through Group Policy as well.

We generally deploy software through Group Policy when three or more computers use it, since it’s very easy to create a new package (if you already have a network share etc. setup, then you can literally do it in 2 minutes).

grouppolicy_software_installation.pngBefore I list some of the useful WMI queries we use to target certain operating systems or computer types, there are a couple of things to note for those who are new to software deployment via group policy:

  • Software packages are always installed right after a reboot, so they’re mostly suitable for workstations.
  • The network share which hosts the MSI files needs to give the computer accounts (e.g. DESKTOP1$) at least read access. Generally, giving EveryOne Read access works well unless you have a reason to restrict access to the software packages that you distribute.

Since the mechanism to distribute software is based on group policies, any sort of software package you create inside a group policy, will need to be assigned to an organizational unit (OU).

Since OUs can contain a large amount of computers that might not all need that particular software package, you can use two techniques to narrow down which computers receive the software:

  1. Security Filtering
  2. WMI Filtering

Security Filtering
With this method, you create a security group in ActiveDirectory, place the computers that should get a particular software package into the group, and then specify this group in the Security Filtering list.

The screen shot below shows a group policy that will only be applied to members of the “Source Control Computers” group:


grouppolicy_security_filtering.png

WMI Filtering
With this method, you can filter the computers which are affected by your policy, based on common properties of the Operating System. For example, some packages might distinguish between 32-bit and 64-bit, some packages might only work on Vista or later, whereas other packages apply only to servers. With WMI, you can target the right computers without having to mess with group memberships (though you will probably still need to do that). For example:

  • 32-bit vs. 64-bit computers
  • only workstations
  • only computers running a certain OS
  • only computers with a certain amount of RAM
  • only computers of a certain brand

With WMI filtering, you just create the software group policy, for example:

  • 7-Zip 32-bit
  • 7-Zip 64-bit

and then apply the respective WMI filter to them. But lets cut to the chase, here are a few WMI queries that you can cut & paste:

Operating System 32-bit


Select * from Win32_Processor where AddressWidth = ’32’

Operating System 64-bit

Select * from Win32_Processor where AddressWidth = ’64’

grouppolicy_wmi_filter.pngWorkstation

Select * from WIN32_OperatingSystem where ProductType=1

Domain Controller

Select * from WIN32_OperatingSystem where ProductType=2

Server

Select * from WIN32_OperatingSystem where ProductType=3

Some filters require multiple WMI queries, which are just chained together.

Workstation 32-bit

Select * from WIN32_OperatingSystem where ProductType=1


Select * from Win32_Processor where AddressWidth = ’32’

Workstation 64-bit

Select * from WIN32_OperatingSystem where ProductType=1


Select * from Win32_Processor where AddressWidth = ’64’

grouppolicy_wmi_filter_multiple.pngWindows XP

Select * from WIN32_OperatingSystem where Version like ‘5.1.%’ and ProductType=1

Windows Vista

Select * from WIN32_OperatingSystem where Version like ‘6.0.%’ and ProductType=1

Windows 7

Select * from WIN32_OperatingSystem where Version like ‘6.1.%’ and ProductType=1

Windows 2003

Select * from WIN32_OperatingSystem where Version like ‘5.2.%’ and ProductType>1

Windows 2008

Select * from WIN32_OperatingSystem where Version like ‘6.0.%’ and ProductType>1

Windows 2008 R2

Select * from WIN32_OperatingSystem where Version like ‘6.1.%’ and ProductType>1

WIN32_OperatingSystem of course includes more information that can be useful for WMI queries, such as a descriptive name of the installed OS (“Name”) as well as the service pack installed (“ServicePackMajorVersion”).

Manufacturer (e.g. DELL)

Select * from WIN32_ComputerSystem where Manufacturer = ‘DELL’

Installed Memory (e.g. more than 1Gb)

Select * from WIN32_ComputerSystem where TotalPhysicalMemory >= 1073741824

Like I mentioned earlier, this is merely a small sample of the possible WMI queries one can use to filter group policies, but they should cover most relevant scenarios. Feel free to suggest other useful WMI queries and I will include them here.

For more information, check out these resources on WMI:

WMI
Secrets of Windows Management Instrumentation
Scriptomatic (Vista/Win2k8/Win7: run as administrator!)

Happy querying,
Ingmar.

Wish Sandwich – 5 (free) tools we wish Windows had

“Have you ever heard of a wish sandwich? A wish sandwich is the kind of a sandwich where you have two slices of bread and you, hee hee hee, wish you had some meat…”

These are part of the lyrics from the “Rubber Biscuit” song by “The Chips“, covered by the Blues Brothers in 1978. At the time, UNIX was almost 10 years old, the first version of BSD had been released, and Microsoft had their office in Albuquerque with Bill Gates being 23 years old.

It would take almost another 20 years before Windows NT 4 would be released. But back to the future now.

Well, after working with Windows for about 15 years now, I also wish that the base set of utilities that ship as part of Windows would have been updated and improved. It might seem odd, but one of the first things I do when I install a new release of Windows, is to open notepad, the calculator and paint – to see if they have improved.

Somewhat surprisingly, Windows 7 brings a lot of improvements to the core utilities that ship with Windows. Microsoft not only spiced up Paint and Wordpad by giving them the “Ribbon”, but also improved the calculator in ways never imagined before. Yeah!

blog_wish_paint_ribbon.pngBut these improvements do not satisfy a long-time Windows user! Having worked with Linux, OS X and Windows since the 3.0 days, I have my own list of apps that I use to substitute or extend some of the archaic tools that ship with Windows.

And here they are:

1. Notepad
What we know today as “Notepad”, was first seen in Windows NT 4.0. When Windows 2000 came out, notepad hadn’t changed. Well, fair enough – it had only been 4 years after all. A short while later Windows XP was released with a bang, but notepad was still the same. Windows 2003 showed that an upgrade to notepad obviously had low priority, and the release of Vista confirmed to me that notepad was clearly no longer under development. The recent release of Windows 7 crushed my hopes of Microsoft ever releasing an updated version of Notepad. Sigh.

So, why was Notepad left behind? Well, I have a few theories:

a)    The developer who originally developed Notepad has left the company, and nobody at Microsoft understands the existing code enough to make modifications.
b)    Companies developing third-party editors formed a powerful, mafia-like lobby, threatening Microsoft (presumably kidnappings) to never ever release an update to notepad, to ensure that third-party editors will continue to sell well.
c)    Microsoft deems Notepad complete, and cannot imagine how this robust application could be improved.
d)    Windows applications do not use text files, since all configuration is stored in the registry or databases. Even though unneeded, Notepad is provided as a courtesy and might be excluded from future version (like, telnet.exe!).

blog_wish_notepad2.pngWhatever the reason (I may never find out), fact is that Notepad hasn’t been updated in 13 years, and since Windows 8 won’t be out until 2012, probably won’t change in 16 years. That’s a lot of years for a software program.

Line Numbers?
Basic syntax highlighting?
Anyone?

So what could replace Notepad? Why, Notepad2 of course! To be fair, there are more powerful editors out there than Notepad2, but it’s free, light-weight and fast. Florian’s Notepad2 supports line numbers, syntax highlighting, line highlighting, encodings, Unix/Windows line endings, transparency and much more. An extended version from Kai Liu is also available here, most notably including code folding abilities. Tabs are not supported in either version, unfortunately.

My other favorite editor is Ultraedit, as it includes pretty much any feature you could ever want from an editor. A nice feature, for sysadmins in particular, is the ability to switch environments. The “System Administrator” view for example, allows you to show SSH/telnet/FTP windows along with the editor windows.

2. Command Prompt
Yeah, this hasn’t changed much since the early days either, though the introduction of the PowerShell deserves some credit. Using Linux regularly though, I miss some of the features like tabs, transparency and so forth.

blog_wish_console2.pngThe good news is, there is an excellent substitution out there called Console. The latest beta of version 2 features transparency, multiple tabs, appearance options and is free. It’s so free, that they even give you the source code if you want it! It works on all the machines I use (mostly Vista, soon to be Win7) and I’m very happy with it overall – though it is a beta still and you might run into a glitch every now and then. I sit around in the command line a lot, and having multiple tabs open is nice.

One option I really like is the ability to show the currently executing command as the tab title, which is useful because you can see when a long-running process finishes (see screenshot above where fping is running in the 2nd tab).

When downloading, get the latest beta and simply extract all files from the

Console2\bin\release

folder to a directory of your choice.

3. Desktops / Spaces
Linux, and Unix, had multiple desktops since the industrial revolution. Well, at least it seems that way. I’m not sure why this hasn’t been added to Windows yet, given that:

•    Every major Operating System OTHER than Windows includes it
•    Microsoft provides a tool (part of Sysinternals) that offers this functionality

Yes, in the age of affordable large monitors, multiple desktops aren’t really that necessary anymore. But, many of us still work on laptops and having multiple virtual desktops can help group different work into different workspaces.

I recommend Sysinternals’ Desktops, but there are more tools out there that do the same thing – though they are not all free.

blog_wish_desktops.png4. Launchy
It indexes all of your applications in the start menu, and you can simply launch them by typing their name – or part of their name. No
longer do you have to wade through dozens and dozens of menu items just to find a shortcut. Simply launch Launchy with ALT+SPACE and type a couple of letters. Voila!

blog_wish_launchy_eventsentry.png5. PuTTY
If you work with Windows and UNIX/Linux machines, then it’s pretty much impossible that you haven’t heard of PuTTY. It’s a free SSH client that no only provides SSH/Telnet functionality, but also comes with other SSH-related utilities like PSCP, PSFTP and PLINK (see previous post on this).

I’d love it if Windows would ship with a command-line SSH client, just like all UNIX and Linux distributions do.

Of course there is more, but these are the tools “desktop” that we really use on a daily basis.

If your computers are in an Active Directory domain and you want to roll out some of these tools with your computers by default, then I recommend reading our previous post: Your favorite tools and utilities always available everywhere.

What do you want for nothing?

Rubber Biscuit?