Tag: process monitoring

How to identify long-running processes

ingmar.koecher 0 Comment EventSentry, Monitoring, Tips & Tricks ,

I always enjoy visiting customer sites for training or consulting since I learn about their unique challenges and requirements, and how EventSentry can meet them.

During a recent visit an interesting question came up: How can I identify (certain) processes which run longer than a certain time period? It may sound like an odd requirement, but some software suites spawn worker processes which perform certain tasks which take a predictable amount of time, such as processing a document for example. If something goes wrong and one of the worker processes hangs, you’d want to know about it.

EventSentry does include a process monitoring feature which can ensure that a certain number of instances of processes are running, even taking their command line arguments into consideration; however it doesn’t evaluate the duration of process.

Even though you cannot do this out of the box (and given that most users don’t require this sort of thing we’re probably not going to add it), there is a pretty easy solution with a (VB)script and the application scheduler. As a reminder, the application scheduler is the standard way of extending EventSentry’s functionality.

Even though VB(Script) is not the most popular scripting language these days, we like to utilize it for a number of reasons:

* The interpreter (cscript.exe) is pre-installed on all versions of Windows
* It was developed on and for Windows, and can handle easy to moderate scripting pretty well
* It’s easy to read and customize, even by people who don’t write code on a regular basis

Of course you can utilize any scripting language with the application scheduler as long as the interpreter is installed. Now let’s see what this VBScript would look like (if you have ever used the Scriptomatic then the structure of this script may look familiar to you):

On Error Resume Next

Const wbemFlagReturnImmediately = &h10
Const wbemFlagForwardOnly = &h20

' Customize start
Const processName   = "parser.exe"
Const maxAgeSeconds = 120
' Customize end

Dim returnCode
returnCode = 0

Set objWMIService = GetObject("winmgmts:\\localhost\root\CIMV2")
Set colItems = objWMIService.ExecQuery("SELECT * FROM Win32_Process WHERE Caption='" & processName & "'", "WQL", _
                                      wbemFlagReturnImmediately + wbemFlagForwardOnly)

For Each objItem In colItems
    Dim secAge
    secAge = DateDiff("s", WMIDateStringToDate(objItem.CreationDate), Now())
   
    If secAge > maxAgeSeconds Then
        WScript.Echo "Process " & objItem.Caption & " (" & objItem.ProcessId & ") has been running for " & secAge & " seconds, since " & WMIDateStringToDate(objItem.CreationDate)
       
        returnCode = 1
    End If
Next

Function WMIDateStringToDate(dtmDate)
     WMIDateStringToDate = CDate(Mid(dtmDate, 5, 2) & "/" & _
     Mid(dtmDate, 7, 2) & "/" & Left(dtmDate, 4) _
     & " " & Mid (dtmDate, 9, 2) & ":" & Mid(dtmDate, 11, 2) & ":" & Mid(dtmDate,13, 2))
End Function

In a nutshell, the script uses WMI to retrieve all running processes and then subtracts the current timestamp from the process start time to determine the runtime (duration) of the process. If it exceeds the pre-configured threshold, the script will return 1 and subsequently log an error to event log.

To get started, first configure the process name and maximum duration in lines 7 & 8. Then, added the script as an embedded script (Tools -> Embedded Scripts) with a descriptive name. Remember to give the file the correct (.vbs) extension here.

Once the file is setup as an embedded script, you can reference it from the application scheduler or an action (although it wouldn’t make much sense to use this script as an action). Create a new system health package, or add the “Application Scheduler” object to an existing system health package. Make sure the package is assigned to the correct computer or group!

To finish, add a schedule to the newly created application scheduler object; in most cases you will want to use a “Recurring Schedule” which will run in regular intervals. On the main application scheduler dialog you will want to make sure that the “Log application return code > 0 to the event log as “Error” is checked. These types of events can then be forwarded to a recipient via email for example.

This script is a pure monitoring script, it won’t take any corrective action by itself. But the script could easily be modified to automatically terminate the process if it has been running for too long. For example, you could either terminate the process with the Terminate() method via WMI, or execute pskill (Sysinternals suite) from within the VBScript. The latter may be more reliable but will require that pskill is installed on all the machines running this script. A modified version of the script is shown below:

' using "Terminate()"
    If secAge > maxAgeSeconds Then
        WScript.Echo "Process " & objItem.Caption & " (" & objItem.ProcessId & ") has been running for " & secAge & " seconds, since " & WMIDateStringToDate(objItem.CreationDate) & ", and will be terminated"

        objItem.Terminate()  
        
        returnCode = 1
    End If

' using pskill
    If secAge > maxAgeSeconds Then
        WScript.Echo "Process " & objItem.Caption & " (" & objItem.ProcessId & ") has been running for " & secAge & " seconds, since " & WMIDateStringToDate(objItem.CreationDate) & ", and will be terminated"

          WshShell.Exec "PSKill " & objProcess.ProcessId  
        
        returnCode = 1
    End If

So there you have it, how to keep long-running processes in check. Since embedded scripts are integrated into the EventSentry configuration, there is no need to manage the script on the remote host.

A nice feature of EventSentry is that any email alert you will get will automatically include the output of the script – delivered straight into your inbox.

Automatically Restarting a Failed Windows Process

ingmar.koecher 0 Comment EventSentry, Monitoring, Tips & Tricks, Tools & Utilities, Uncategorized ,

Whether it’s a critical process running on a server or an application on your desktop – sometimes processes terminate and need to be restarted – immediately.

With EventSentry & EventSentry Light you can do just that: Automatically restart processes immediately after they terminate.

In the past, one drawback of EventSentry launching a process was the side effect that any process started by the EventSentry agent would run under the same account as the EventSentry agent itself (usually a privileged domain account or LocalSystem).

In this post I’ll discuss how you can work around that limitation in a secure manner using a scheduled task. When the critical process fails, instead of launching the process directly through a process action, EventSentry will trigger a scheduled task instead. Why? Because scheduled tasks allow you to configure under which user a task will run – and the user’s password is securely stored in Windows.

The recipe for accomplishing this feat is as follows:

  • Process Monitoring monitors the process
  • An event log filter looks for the “failed process” event and triggers a process action
  • The process action starts a scheduled task

Let’s look at this in detail. First, on the host where the critical but unstable task is running, you create a schedule task in the Windows “Task Scheduler”. Under General, give the task a descriptive name (“Start Super Important App”) and change the user under which the program should be running under. In most cases you will also want to make sure that you configure the task to run whether a user is logged on or not. Then, under “Actions”, add a new action “Start a program” which points to the executable that should be launched. After you click “OK” you will be prompted for the password for the user.

Scheduled Task
Creating a scheduled task

The next step is to setup process monitoring in EventSentry. Right-click “System Health” and create a new package and assign it to the computer(s) in question. Right-click the newly added package and select “Add – Processes”. Click the newly added object and add the name of the process which should be monitored. You can configure how many instances of the processes are required, and with which severity the event will be logged when the process is inactive.

process monitoring
Configuring process monitoring

Now we create a new “Process” action. Right-click the “Actions” container, select “Add” and enter a descriptive name (e.g. “Trigger Super Important App”). In the Filename field specify:

%SYSTEMROOT%\system32\schtasks.exe

And for the Command Line Arguments enter:

/Run /TN “Start Super Important App”

This uses the built-in Microsoft utility schtasks.exe to run the task we created in our first step. At this point EventSentry will monitor the specified process and log an event if the process is inactive. And while we do have an action to trigger the scheduled task, we still need to tell EventSentry when to launch that action.

EventSentry Process Action
Configuring a process action to start a scheduled task

For the next step, right-click the “Event Logs” container, select “Add Package” and give that package a descriptive name. Then assign the package to the same host. Right-click the newly added package and add a filter by clicking “Add Filter”. In the filter dialog, add the “ Trigger Super Important App” action to the action list and configure the following fields:

Event Log Include Filter Rule
Setting up a rule to trigger the process action

Event Severity: Information
Event Log: Application
Event Source: EventSentry
Category: Process Monitoring
Event ID: 10401
Content Filter (wildcard): *critical_app.exe*

Important Notes: The event severity will need to match whichever severity you selected when adding the process monitoring object in the system health package. The content filter can also be configured to match insertion string #1, in which case the wildcards are not necessary.

And that’s all there’s to it, simply save the configuration when you are done. If the process is running on a remote host then don’t forget to push the configuration to that host.