EventSentry Blog

Trapping CryptoLocker/CryptoWall with Honey (Part 2/3)

November 11, 2015February 23, 2020 ingmar.koecher 1 Comment Event Log, EventSentry, Monitoring, RansomWare, Tips & Tricks cryptolocker, cryptowall, file integrity monitoring, fim, ransomware

! Updates !
There has been a follow-up post to this article with even better approaches to defeating ransomware. I highly recommend that you jump directly to the most recent article which offers the best & easiest approach for protecting against Ransomware:

Defeating Ransomware with EventSentry & Auditing (Part 3)

When I wrote my first, original post about CryptoLocker (“CryptoLocker Defense for Sysadmins”), I didn’t intend there to be a part 2 or even a part 3. But alas, due to the “popularity” of CryptoLocker and the recent release of CryptoWall 4.0 I decided to write a much-needed sequel to my first blog post. Part #2 differs from the first part with a different (and more simple) detection “algorithm” combined with a more reliable way to stop the “Server” service when CryptoLocker is indeed detected.

Surprisingly (or not surprisingly), almost 2 years after I wrote my first article, CryptoLocker and its descendants like CryptoWall are still around, thriving, and keeping Sysadmins around the world busy. A recent report stated that CryptoWall 3.0 cost victims a combined $325 million, although it fails to mention whether this is an annual or lifetime figure. This is the same as the GDP of the small African country of São Tomé and Príncipe (population of about 200,000) in 2014.

Now there is something to think about – the criminals behind the various ransomware software collected as much money as a country with 200,000 people. Alright, this is all very interesting but doesn’t help us protect ourselves from ransomware so let’s focus.

In part #1 we used EventSentry’s file monitoring feature to index and inventory all files on a susceptible file share, a very accurate and resilient way to detect any sort of software which would modify large numbers of files in a short time period. While this approach works well, it does require more time to setup and may not work in real-time when monitoring extremely large directories. Consequently we’ll be using a different approach here, and we will look at yet another approach in part #3.

EventSentry’s file checksum monitoring feature was originally intended to monitor only key Operating System folders such as the System32 directory, but increased customer demand prompted us to tweak the feature over time to allow real-time monitoring of even very large folders (as is the case for file servers) as well. But enough of the past, let’s tackle Crypto*.

What’s New?
New versions of software (especially free) are usually exciting, but I’m guessing that the latest “improvements” rolled into the various types of ransomware, including CryptoWall, are only exciting for security researches and the people behind the ransomware. There are three major new features included (for free) in the latest version of CryptoWall:

Files are not only encrypted, but file names are now also mangled, making it almost impossible to link the encrypted file(s) with their originals.
Shadow copies are being deleted if possible, so that past versions of files are no longer accessible
The encryption process seems to be less linear and less complete, resulting in some folders being left alone and thus making detection more difficult.

The Theory
If you’ve been working in the IT (security) field for a while then you’ll have probably heard of honeypots before. Honeypots are usually systems emulating a production server with the purpose of detecting an attacker and potentially triggering counter-measures or alerts.

We’ll apply the concept of honeypots to detect CryptoLocker, but instead of emulating entire systems we’ll plant on or more fake files throughout on one or more file shares with the assumption that any CryptoLocker infection will attempt to change and encrypt those files. Once detected, we can trigger a counter measure such as stopping the server service. The three biggest risk factors with this approach are:

Accidental modification by a user
CryptoLocker detects (and skips) the honeypot files
The bait files get modified too late

But not to worry – we can mitigate all of the risks.

Accidental Modification
Since any unsuspecting user with write access may accidentally modify or delete our bait file, it’s possible that some users curiosity may result in some sort of a accidental DoS attack. Making the file read-only defeats the purpose of detecting CryptoLocker of course, since CryptoLocker itself won’t be able to modify it. I was able to come up with two possible solutions for this problem:

1. Give the file a boring name which discourages users from opening it (e.g. meeting_notes_cl1.docx)
2. Put clear instructions into the file in large font, instructing users not to modify or delete the file. I’d recommend against mentioning any words like CryptoLocker, Virus, etc since CryptoLocker may be parsing the contents of the file.

Honeypot is not sweet enough
Since we don’t have access to CryptoLocker and its constantly evolving code, we don’t know whether it has any honeypot detection capabilities, and if it does, how it attempts to detect them. Since I’m rather safe than sorry, I’m assuming that it has some basic capabilities. It could be as simple as skipping files which are smaller than a pre-defined threshold or looking for specific file names. E.g., based on this article CryptoLocker could now skip any file named meeting_notes_cl1.docx. To maximize our chances for success:

1. Make sure the file is not too small and exhibits properties of other office documents (e.g. 1Mb in size, multiple pages)
2. Give the file a unique, meaningful name, see previous paragraph.

Once you have created the file, place it strategically on your file server among other office documents. I recommend deploying multiple honeypot files if you have multiple file shares. It may be advisable to give the files unique names (e.g. meeting_notes_cl1.docx, meeting_notes_cl2.docx, …) as well.

Too little, too late
The bait file getting modified too late is the biggest risk unfortunately. If you have a directory with 50,000 files but only one bait file, then it won’t help us to detect CryptoLocker. Since we don’t know how CryptoLocker enumerates files (alphabetical, sorted by size, …), it’s probably best to sprinkle them throughout the various vulnerable file shares, using file names which start various letters of the alphabet, e.g.:

a_meeting_notes_cl.docx
m_meeting_notes_cl.docx
s_meeting_notes_cl.docx
z_meeting_notes_cl.docx

A name pattern is not required but helpful when configuring EventSentry later, since it allows you to just specify a wildcard (e.g. *_meeting_notes_cl.docx) instead of specifying dozens of files manually.

Creating multiple bait files is particularly important for newer versions of CryptoLocker which doesn’t always parse/encrypt all directories. So it’s best to create multiple bait files and distribute them across multiple directories, e.g.:

marketing\m_meeting_notes_cl.docx
sales\a_meeting_notes_cl.docx
accounts_payable\z_meeting_notes_cl.docx

This way we’ll have a higher chance of detecting malicious behavior. CryptoWall is fast (of course depending on the speed of the infected host) and can often encrypt tens of thousands of files in an hour.

Implementation
We will use EventSentry’s File Checksum Monitoring feature to monitor the bait files and trigger events when one or more of these files are changed or deleted (=renamed). When they are, we will trigger a script which will stop the server service on the file server in order to avoid more damage being done. Click here to learn more about EventSentry’s architecture.

Monitoring files only for (checksum) changes is no longer sufficient since newer variants of CryptoWall not only modify but also rename (and subsequently delete) documents.

In EventSentry, create a new System Health package with the name “CryptoLocker Detection” and assign it to any server on your network that is monitoring with EventSentry and is serving files through a file share. Now, add the “File Checksum Monitoring” object to the package and ensure the following:

“Monitor folder(s) in real time” is checked
Disable (uncheck) both “Only verify checksum when ..” optimizations

Then, click the “plus” icon to add the (first) folder where a bait file exists to the list of monitored folders. There are a few things to consider when setting this up

The folder/directory name should be specified as it exists on the file server, UNC paths are not recommended.
Check the “Include Sub Directories” check box you are monitoring files in sub folders
Check “Detect File Deletions” and “Detect File Checksum Changes”. File size increases and decreases may also be checked but is not required.
Configure the “Files” section to “Only monitor files that are included below” and specify the file name either with a full (relative) path or with a wild card.
Select a severity of “Error” under “Log to Event Log as”

Example
Your file server has a directory called C:\FileShares\Marketing with two sub directories, Ads and Images. If we were to add a bait file to both subdirectories (say specs.docx and meeting1.docx) then we would specify C:\FileShares\Marketing as the folder, and then add

Ads\specs.docx and
Images\meeting1.docx

as the files to be monitored. This is because we always specify the path relative to the main folder being monitored when specifying the file names.

Splendid, EventSentry will now log an event to the event log when any of these files change. Try it out – open the file in word, make a change & save – you should get an alert in the event log almost instantly.

Dive! Stopping the Server Service
Stopping the server service may seem like a drastic step, but it’s unfortunately the most efficient way to prevent an impending CryptoLocker infection from spreading. Sure, blowing up the bridge might seem crazy at first, but if it prevents an army of Zombies (who obviously can’t swim) from entering your town, then we can probably live with the collateral damage.

You can stop a service in 2 ways with EventSentry; with the “Service / Process Control” action as well as with a custom script. Creating a “Service / Process Control” action is easier, but only works for stopping services which have no dependencies. You can probably guess where I’m going with this – the server service depends on other services (e.g. when the “File Sharing Role” is enabled) and thus cannot be stopped with the EventSentry action. Consequently we will go a different route and create a process action instead, which essentially allows you to trigger any process, script etc. Better safe than sorry.

Right-click the Actions container and click “Add” to create a new action called “Stop Server Service”, and select “Process” as the action type. Specify “net.exe” as the Filename, and “stop lanmanserver /yes” as the command line arguments. The “/yes” switch ensures that any service which depends on the “Server” service also gets stopped.

Connecting the dots
Since we now assume that a modification of one or more of our bait files only happens when a CryptoLocker outbreak is under way, the only thing missing now is to have the file change event trigger the process action and shut down the service.

EventSentry uses the concept of “Event Log Filters” to link events to actions, such as sending an email and/or triggering a process. Filters need to be part of an “Event Log Package”, and we can now either create a new package or add our filter to an existing package. For documentation purposes and to keep things orderly we will create a new event log package called “CryptoLocker Prevention”.

We do this by selecting the “Packages – Event Logs” container and clicking “Add” from the ribbon, you can also right-click that container. Give it a descriptive name and select the package, which we now need to assign to one or more hosts and/or groups. Click “Assign” in the ribbon to assign the package, you can also make the package global by clicking the respective button.

With the package all ready to go, we now need to add the filter. With the package still selected, on the ribbon click the “Add” button under “Event Log” and select “Include”. This event log filter, as is, would not apply to any event, since no event log and no severity is selected.

Anything detected by the EventSentry agent (e.g. a file checksum change, service status change, low disk space) is logged to the Application event log with the source “EventSentry”, a matching category (e.g. “File Monitoring”) and usually with a configurable or dynamic severity. In our case the file checksum change events will be logged as Errors, as configured earlier.

So let’s first configure the event properties as shown in the screenshot:

Log: Application
Event Severity: Error
Source: EventSentry
Category: File Monitoring

We also add the “Stop Server Service to the list of actions to be triggered. Since we may have other system health packages which log File Monitoring events, we want to make sure that this filter only applies to those, which we do by restricting the filter further with an event id as well as with a Content Filter.

For CryptoLocker we want to get notified about every change that happens to our bait file. Whether it’s deleted, a checksum change or a file size change. As such, we leave the event id field empty and specify the “File Monitoring” category instead.

Important Note: If you are running a German version of Windows, the category will need to be specified in German (“Dateiüberwachung”) since EventSentry is localized for German.

Our filter could still apply to unrelated file checksum changes (e.g. OS files were changed by a Windows Update), but since any file checksum change event includes the package name which triggered the event, we can filter based on that name (we called the package “CryptoLocker Detection”) to ensure that we only match file changes from CryptoLocker. In the “Content Filter” section click the “+” button to add a new content filter.

The quickest way to specify the content filter is to leave the “Wildcard match” setting in place and simply specify *CryptoLocker Detection* as the content filter. A more elegant way is to use an Insertion String match and selecting insertion string 5, which represents the package name (click “Preview” to see the insertion string numbers).

The setup is now complete, and you can now push the configuration to the remote host(s) which has the bait files and should be protected. If you have multiple file servers with a different directory structure, then you can easily create multiple system health packages which contain a file monitoring object, and assign them accordingly. For example, you could create packages named:

CryptoLocker Detection Server1
CryptoLocker Detection Server5

The process action doesn’t have to be duplicated, since the stopping the service is the same process for all hosts. The event log filter may need to be adjusted depending on how it was setup. A wild card like *CryptoLocker Detection* would match “CryptoLocker Detection Server5” as well, but an insertion string filter would need to be modified to something like CryptoLocker Detection* in order to match multiple more than one package.

An alternative to email alerts. Part 2: Integrating EventSentry with Slack

October 12, 2015January 9, 2017 ingmar.koecher 0 Comment Event Log, EventSentry, Monitoring eventsentry, monitoring alert management, slack

Slack is a flexible, web-based messaging app for teams which supports multiple (mobile) platforms with the goal of streamlining communication and collaboration. In some ways, Slack feels like a combination of Hipchat, IRC and Dropbox (you can also send alerts to Hipchat from EventSentry).

By creating channels (e.g. chat rooms) and also allowing for direct 1:1 communication, interacting with your team is straightforward and easy. What’s more, you can send 3rd party feeds such as Twitter notifications directly into a channel, giving you the ability to see data from multiple sources in one central location.

Slack naturally lends itself to receiving EventSentry notifications with its web-based API. And, since you can create multiple channels, it’s easy to divert EventSentry alerts into different channels, e.g. #alerts-windows and #alerts-development.

Since Slack offers native clients (e.g. Windows, MacOSX) in addition to their web-based site, you can choose to opt in to receive a visual notification every time an alert is received by Slack.

If you’re not already using Slack, then you can sign up for free at http://slack.com and see for yourself whether this is a communication platform you will want to use. If you already use Slack, then you will just need to make sure that the “slackbot remote integration” is setup. You can find information on how to enable this integration here: https://api.slack.com/slackbot.

Once the Slackbot Remote Control integration is configured in Slack, follow these steps within the EventSentry Management Console using the slackbot URL provided by Slack to set up an HTTP action. The HTTP action will be triggered by one or more filters in an event log package to submit events/alerts to Slack.

1. Add a new action
In the left pane right click “Actions” and select “Add Action”

Adding an EventSentry action

2. Selecting the correct action type
On the “Action Selection” screen type your name for the new action (Ex: Slack), select “HTTP” and then press OK

3. Configuring the HTTP action
In the right pane you should now see the settings for your new Slack action, update the URL field with the slackbot URL provided by Slack. If you did not specify a channel in your url after the token add “&channel=%23” followed by the name of your channel.

Example: https://team.slack.com/services/hooks/slackbot?token=zk2jR22I34AK24IpEd7tdyroGt&channel=%23es-alerts

4. Configuring the data to submit

In the right pane configure the Data you’d like slackbot to post to the channel, the following is suggested for basic information:

$EVENTCOMPUTER: $EVENTID:$EVENTSOURCE:$EVENTCATEGORY by $PACKAGE - $FILTER
 $EVENTMESSAGE

Additional variables can be found in our documentation.

Once the action is configured click “Test” and you should see the test message in Slack.

5. Finishing up

You can now apply the action to any existing or new packages and filters. To learn more about filters review the documentation or walk through this tutorial.

Save the EventSentry configuration and push the configuration to the remote hosts.

Done!

Reliable SMS Alerting with the SMSEagle

February 18, 2015May 4, 2017 ingmar.koecher 0 Comment EventSentry, Monitoring, Tools & Utilities, Uncategorized sms alerting, text message alerting

Note: This article was updated in May 2017 to reflect some of the improvements SMSEagle has made to its devices, most notably support for additional wave lengths and power adapters which make the device work flawlessly in most countries in the world.

Emails are still the alert type of choice for most network administrators, but they come with a major pitfall: They either rely on your local email server, the Internet connection, or worse: both.

So if you need to get notified when your email server or Internet connection are down – and most likely you will want to get notified – text messages (aka as SMS) are a convenient way out of this monitoring conundrum.

Email to SMS, or web-based SMS services can help in some situations, but you need a different kind of beast when the Internet connection is down.

This is where hardware solutions like the SMSEagle come in. The SMSEagle is essentially a cell phone with an Ethernet jack, a web server + API, allowing you to send text messages either through its web interface or API (preferred).

The SMSEagle and similar devices usually only require three things:

A valid SIM card
A physical location with coverage from your provider
An Ethernet connection to your network

With the SMSEagle in place, you no longer have to fear loss of Internet connectivity or mail server downtime – a text alert is just a few seconds away when combined with a monitoring solution like EventSentry. In fact, the SMSEagle is somewhat unique in that it actually offers its own basic network monitoring capabilities which can be used in addition to EventSentry – or to monitor the host running the EventSentry Heartbeat Monitor.

Since the SMSEagle features a web-based API, EventSentry can submit certain alerts, e.g. pertaining to Internet connectivity or the availability of a specific host and/or service, directly to the SMSEagle through its HTTP action, which will then send a SMS alert directly to your mobile device of choice.

The SMSEagle accepts mini SIM cards and works with the UMTS 800/850/900/AWS1700/1900/2100 MHz and GSM/GPRS 850/900/1800/1900 MHz wave bands and thus should work with all major cell providers. The latest models of the SMSEagle also ship with power plugs which will work in most countries, including Europe, the U.S., Australia and many others.

The setup of the device is straightforward: You first insert the SIM card, connect it to the Ethernet and power on the device. The device automatically assigns itself IP address 192.168.0.101 but can also retrieve its IP address via DHCP. More instructions & details are available in the manual.

Once you’ve setup the SMSEagle and verified that SMS messages can be sent and received, you can create an EventSentry HTTP action and point it to your SMSEagle device. The HTTP action includes a number of templates to quickly load all required fields for a specific HTTP API; simply select the SMSEagle from the drop-down and specify all the required fields such as host name and so forth. Use the test button to ensure the action is setup correctly.

Once the action is setup it can be referenced by one or more filters so that it is triggered under the right circumstances. Assuming that you are already monitoring a host outside your network (e.g. your ISPs DNS server, a public web site) to determine whether your Internet connection is available or not, you would want to look for EventSentry event 11000, which indicates when a host changes its availability status, such as:

Host ispdns (Internet) changed its PING status from OK to ERROR. The reason for the status change was: “100% packets lost”.

A filter setup for this event would look like something like this:

EventSentry Heartbeat Filter — Filter for EventSentry heartbeat alert

Here, we are looking for an application error generated by the Heartbeat Monitoring category of the EventSentry source. We’re also further restricting the filter to only alert us on status changes of the “ispdns” host when it goes off-line. Since the SMSEagle is listed as the action, this particular event (alert) will be sent to the SMSEagle action.

The insertion strings can be determined by either clicking on the “Lookup” button on the filter dialog, or by clicking on the “Preview” button when adding a content filter.

If you’re located in the Europe or the United States then the SMSEagle is a non-brainer. Otherwise, there are a few other devices out there who work similarly , and as long as they offer a HTTP-based API, integrating EventSentry with them should be easy.

How to identify long-running processes

January 16, 2015January 9, 2017 ingmar.koecher 0 Comment EventSentry, Monitoring, Tips & Tricks process duration, process monitoring

I always enjoy visiting customer sites for training or consulting since I learn about their unique challenges and requirements, and how EventSentry can meet them.

During a recent visit an interesting question came up: How can I identify (certain) processes which run longer than a certain time period? It may sound like an odd requirement, but some software suites spawn worker processes which perform certain tasks which take a predictable amount of time, such as processing a document for example. If something goes wrong and one of the worker processes hangs, you’d want to know about it.

EventSentry does include a process monitoring feature which can ensure that a certain number of instances of processes are running, even taking their command line arguments into consideration; however it doesn’t evaluate the duration of process.

Even though you cannot do this out of the box (and given that most users don’t require this sort of thing we’re probably not going to add it), there is a pretty easy solution with a (VB)script and the application scheduler. As a reminder, the application scheduler is the standard way of extending EventSentry’s functionality.

Even though VB(Script) is not the most popular scripting language these days, we like to utilize it for a number of reasons:

* The interpreter (cscript.exe) is pre-installed on all versions of Windows
* It was developed on and for Windows, and can handle easy to moderate scripting pretty well
* It’s easy to read and customize, even by people who don’t write code on a regular basis

Of course you can utilize any scripting language with the application scheduler as long as the interpreter is installed. Now let’s see what this VBScript would look like (if you have ever used the Scriptomatic then the structure of this script may look familiar to you):

On Error Resume Next

Const wbemFlagReturnImmediately = &h10
Const wbemFlagForwardOnly = &h20

' Customize start
Const processName   = "parser.exe"
Const maxAgeSeconds = 120
' Customize end

Dim returnCode
returnCode = 0

Set objWMIService = GetObject("winmgmts:\\localhost\root\CIMV2")
Set colItems = objWMIService.ExecQuery("SELECT * FROM Win32_Process WHERE Caption='" & processName & "'", "WQL", _
                                      wbemFlagReturnImmediately + wbemFlagForwardOnly)

For Each objItem In colItems
    Dim secAge
    secAge = DateDiff("s", WMIDateStringToDate(objItem.CreationDate), Now())
   
    If secAge > maxAgeSeconds Then
        WScript.Echo "Process " & objItem.Caption & " (" & objItem.ProcessId & ") has been running for " & secAge & " seconds, since " & WMIDateStringToDate(objItem.CreationDate)
       
        returnCode = 1
    End If
Next

Function WMIDateStringToDate(dtmDate)
     WMIDateStringToDate = CDate(Mid(dtmDate, 5, 2) & "/" & _
     Mid(dtmDate, 7, 2) & "/" & Left(dtmDate, 4) _
     & " " & Mid (dtmDate, 9, 2) & ":" & Mid(dtmDate, 11, 2) & ":" & Mid(dtmDate,13, 2))
End Function

In a nutshell, the script uses WMI to retrieve all running processes and then subtracts the current timestamp from the process start time to determine the runtime (duration) of the process. If it exceeds the pre-configured threshold, the script will return 1 and subsequently log an error to event log.

To get started, first configure the process name and maximum duration in lines 7 & 8. Then, added the script as an embedded script (Tools -> Embedded Scripts) with a descriptive name. Remember to give the file the correct (.vbs) extension here.

Once the file is setup as an embedded script, you can reference it from the application scheduler or an action (although it wouldn’t make much sense to use this script as an action). Create a new system health package, or add the “Application Scheduler” object to an existing system health package. Make sure the package is assigned to the correct computer or group!

To finish, add a schedule to the newly created application scheduler object; in most cases you will want to use a “Recurring Schedule” which will run in regular intervals. On the main application scheduler dialog you will want to make sure that the “Log application return code > 0 to the event log as “Error” is checked. These types of events can then be forwarded to a recipient via email for example.

This script is a pure monitoring script, it won’t take any corrective action by itself. But the script could easily be modified to automatically terminate the process if it has been running for too long. For example, you could either terminate the process with the Terminate() method via WMI, or execute pskill (Sysinternals suite) from within the VBScript. The latter may be more reliable but will require that pskill is installed on all the machines running this script. A modified version of the script is shown below:

' using "Terminate()"
    If secAge > maxAgeSeconds Then
        WScript.Echo "Process " & objItem.Caption & " (" & objItem.ProcessId & ") has been running for " & secAge & " seconds, since " & WMIDateStringToDate(objItem.CreationDate) & ", and will be terminated"

        objItem.Terminate()  
        
        returnCode = 1
    End If

' using pskill
    If secAge > maxAgeSeconds Then
        WScript.Echo "Process " & objItem.Caption & " (" & objItem.ProcessId & ") has been running for " & secAge & " seconds, since " & WMIDateStringToDate(objItem.CreationDate) & ", and will be terminated"

          WshShell.Exec "PSKill " & objProcess.ProcessId  
        
        returnCode = 1
    End If

So there you have it, how to keep long-running processes in check. Since embedded scripts are integrated into the EventSentry configuration, there is no need to manage the script on the remote host.

A nice feature of EventSentry is that any email alert you will get will automatically include the output of the script – delivered straight into your inbox.

An alternative to email alerts. Part 1: Using Trello to manage EventSentry’s alerts

October 2, 2014January 10, 2017 ingmar.koecher 0 Comment EventSentry, Monitoring, Tips & Tricks, Tools & Utilities monitoring, monitoring alert management, trello

Trello is a simple yet powerful and innovative task management / collaboration platform for teams. With Trello, the developers have basically taken the familiar concept of traditional white boards where you add and remove tasks (by writing on them), and moved it to an easy-to-use online tool.

While Trello doesn’t attempt to replace the more complex project management and collaboration tools available (including its own FogBugz platform), it makes keeping track of small ToDo lists and tasks surprisingly simple, while still supporting advanced features such as due dates, attachments, assignments and more. Of course, Trello also includes a very capable mobile app for iOS and Android (I only tested the iOS version).

And best of all, it’s completely free if you stick with the basic (and for most people completely sufficient) functionality. But what does Trello have to do with EventSentry and cutting down on emails?

We’re always looking for innovative ways to make managing alerts easier and more productive, especially in larger teams. While email alerts certainly serve a purpose and can be quite useful, alerts dispatched via email suffer from a few disadvantages:

Emails sent to multiple recipients make it difficult for the recipient to know whether the alert has been acted upon or not
Alerts which have already been resolved by a team member still remain in your inbox
Emails often get lost amidst other emails and potentially critical alerts may get overlooked

How Trello Works
Trello is organized into boards, each of which can have one or more lists, each of which have multiple cards. Since Trello offers an API, you can use EventSentry’s HTTP action to submit events (alerts) directly to one (or more) Trello lists.

And this is where the fun starts. Once in Trello, alerts (now cards, or “alert cards”) can be acted upon in a variety of creative and useful ways. You can:

Receive alerts in your browser when a card is created
Move a card to a different list (e.g. “Resolved”, “Under Investigation”, …)
Assign one or more people to a card
Add comments to a card
Assign a due date to a card
Mark a card as important (you can even define your own color codes)
Receive periodic summary emails if you don’t visit the board

All of these features make managing alerts in teams with multiple SysAdmins much easier. When an alert comes in, anybody can act on it (e.g. add themselves) or assign it another team member. Any changes are immediately visible to all other team members in real-time (and we at NETIKUS love anything real-time).

Integrating EventSentry with Trello is a 3-step process:

Sign up for Trello, create a board and customize the associated lists
Get an API & access key & determine ID of your list
Setup HTTP action in EventSentry and create/modify rules

Signing up for Trello
To get started, navigate to http://www.trello.com and sign up with an email address. After you log in for the first time, you will automatically get the “Welcome Board” which will show you all the things you can do with Trello. Since we don’t want to use the default board, we click the big PLUS icon on the top right instead and select “New Board”.

Give the board a descriptive name, e.g. “EventSentry Alerts”. Once created, the board will contain three default lists. You can either leave the list names as they are, or customize them as shown in the screen shot below. I chose “Active”, “Working on” and “Resolved”.

Getting an API and access key
Now that you’ve signed up, the next logical step is to get the API key so that EventSentry can start submitting events to Trello. So while you are logged in, navigate to https://trello.com/1/appKey/generate and note down (aka copy & paste) the first value “Key”, a 32 character-long hexadecimal value. This is the “main” key for your user account, and will be used whenever you (or EventSentry) make an API request.

The API key doesn’t actually let us access data from the boards, for which we’ll need an access key. There are different types of access keys with customizable expiration dates available, but in this case we’ll just get a read/write key without an expiration date. Navigate to the following URL to get a universal read/write access key and substitute APIKEY with the key you obtained just before:

https://trello.com/1/authorize?key=APIKEY&name=EventSentry&expiration=never&response_type=token&scope=read,write

You will end up with a dialog similar to the one shown above, where you need to click the green “Allow” button. This will issue another hexadecimal key, this time 64 characters in length. Note this key down as well. Of course you can be less generous and issue keys which expire automatically, e.g. after 30 days. See the Trello docs for more details on the different “expiration” options available.

Getting the list ID
Our end goal is to submit cards to the “Active” list on our “EventSentry Alerts” board. In order to add a new card to this list however, we’ll need the list’s ID. Equipped with our main key and access key, we’re almost there. First, navigate to your “EventSentry Alerts” board in Trello (or whichever board you want to submit cards to) and note down the URL. For example, if the URL is https://trello.com/b/gePT9Wax/eventsentry-alerts, then you’ll want to extract the text between the /b/ and the board name, gePT9Wax in this case. Now, navigate to the URL below, and replace APIKEY with the API key, and ACCESSKEY with the access key:

https://api.trello.com/1/boards/gePT9Wax?lists=open&key=APIKEY&token=ACCESSKEY

This will return detailed results in JSON format similar to this:

{"id":"561e92617481e9a123aef3aff”,
 "name":"EventSentry Alerts”,
 "desc":”",
 "descData":null,
 "closed":false,
 "idOrganization":null,
 "pinned":true,
 "url":"https://trello.com/b/gePT9Wax/eventsentry-alerts”,
 "shortUrl":"https://trello.com/b/gePT9Wax”,
 "prefs”:  { ……… }
},
,"lists”:
[
 {"id":"561e92617481e9a123aef3b00","name":"Active","closed":false,"idBoard":"561e92617481e9a123aef3aff","pos":16384,"subscribed":false},
 {"id":"561e92617481e9a123aef3b01","name":"Working on","closed":false,"idBoard":"561e92617481e9a123aef3aff","pos":32768,"subscribed":false}, {"id":"561e92617481e9a123aef3b02","name":"Resolved","closed":false,"idBoard":"561e92617481e9a123aef3aff","pos":49152,"subscribed":false}
]
}

What we are interested in is the list id of our “Active” list, 561e92617481e9a123aef3b00 in the example above. With the last missing piece of the puzzle in our hands, we’re now ready to setup a HTTP action in EventSentry.

Configuring EventSentry
Right-click the actions container or utilize the ribbon to create a new HTTP action. In the action dialog, specify the following URL, replacing LISTID with the list id we just obtained:

https://api.trello.com/1/lists/LISTID/cards

In addition to the URL, we’ll need to specify at least 4 form fields:

key: APIKEY
token: ACCESSKEY
name: $EVENTCOMPUTER $LOG $EVENTSOURCE $EVENTCATEGORY $EVENTID
desc: $EVENTMESSAGE

The key and token fields need to be replaced with your API key and access key, whereas the name and desc fields can be customized to suit your needs: what I have shown above is just an example which should work reasonably well in most cases. You can add or remove other event variables as you wish. The upcoming v3.1 will include Trello in the template list to make this a bit easier.

Screenshot EventSentry HTTP Action Trello — Configuring an EventSentry HTTP action for Trello

Once the action is configured, click the Test button to ensure that all IDs have been specified correctly. If the test succeeds, then you should see a new card in the “Active Alerts” list in the EventSentry Alerts board.

Of course an action alone will not forward any alerts to Trello, so you will need to make some changes to your filters and packages. You can either modify existing filters / event log packages and replace the email action with the new Trello HTTP action, or add the Trello action to existing event log packages / filters. Remember that actions can be defined on a package-level through the package properties as well which can help save time.

Managing Alert Cards
Once your first alert card arrives in the “Active” lists and is analyzed by a team member, a few actions can be taken:

You can add a team member to the card, essentially assigning the alert to them. You can add multiple team members as well
If the event is a false alert, it can be moved to a “False Alert” list, which would indicate that an exclusion filter should be setup in EventSentry
You can assign a due date, if the alert requires a resolution by a specific date
You can add a comment to the card
You can label the card (e.g. “Important”)
You can archive & delete the card

EventSentry alerts shown on a trello board

As you can see, despite its simplicity, Trello offers quite a few features to manage and collaborate. This ensures that alerts don’t disappear in an email inbox somewhere and instead are acted upon, while also allowing collaboration with comments, due dates and such.

Additional Tips & Tricks for Trello
In order to get alerted when a new alert card is created in the EventSentry Alert boards, you’ll need to subscribe to the board. This ensures that you will get a notification on your mobile phone, browser (when enabled http://blog.trello.com/how-to-use-trello-like-a-pro/) or email every time there is activity on a board. Activities include new cards being created, cards being moved to a different list, users being added to cards and so forth.

Note: You will not get a notification if the EventSentry Agent is submitting new cards while using your access key (only other users will see the alerts). This is because Trello assumes that you are creating the cards, and subsequently not notifying you about them.

One way to circumvent this restriction is to create a “service” account (e.g. eventsentry.yourcompany@gmail.com) and issue the access token under this user. Then, everybody will see the alerts.

But don’t stop there!
Of course you can use Trello for what it was originally designed to do as well – manage tasks. We’ve found it to be a great and easy way to handle ToDo lists for teams, resulting in more transparency and efficiency. Assigning a task is quick and easy, and team members can easily track progress with projects – without pesky emails floating around between team members.

Now you just have to get all your To-Do items actually done too. But at least I can now move my “Create Trello Blog Post” card into the “Done” list. And that feels good.