Monitoring Archives - Page 11 of 12

How to dynamically toggle your Wireless adapter with EventSentry

January 22, 2012January 9, 2017 ingmar.koecher 0 Comment EventSentry, Monitoring, Tips & Tricks, Tools & Utilities eventsentry toggle network adapter status with netsh

Most of the time I work on a Lenovo laptop running Windows 7, and I’m overall quite happy with the laptop (especially after the mainboard was replaced and it stopped randomly rebooting). However, a minor nuance had been bugging me for a while: If I plugged my computer into a LAN (I have a docking station at work and at home) while a wireless signal was also available (and configured on the laptop), Windows 7 would keep both connections active.

1. The Problem

So I’d have my laptop in the docking station, connected to a 1Gb Ethernet network, and yet the laptop would also be connected to a WiFi network. While not a big deal per se, it does have a few advantages to automatically disable the WiFi connection when already connected to Ethernet:

Avoid potential connectivity issues
Increase security by not transmitting data via Wifi when not necessary
Increase battery life when connected to a LAN
Because you can!

Now, Lenovo equips most (if not all) laptops with a software called “Access Connections”, a pretty nifty and free tool! One of the things it can do, is disable a Wireless adapter when the computer is connected to Ethernet. However, it never re-enables it when you disconnect from the wired network (at least I haven’t found a way), and besides not everybody has Lenovo laptop.

So how could I disable the WiFi connection automatically when I connected the laptop to an Ethernet, yet automatically re-enable it when there is no Ethernet connection?

2. The Research

After some intense brainstorming, I remembered two things:

Most Ethernet NIC drivers log event to the System event log when a network port is connected/disconnected.
A while back, I used the netsh command to configure DNS servers from the command line. Maybe one could toggle the state of network adapters with this tool as well?

3. Evidence Gathering

The first one was easy, a quick look at the system event log revealed the following event:

A similar event is logged when the “network link” has connected. The event shown here is specific to the driver of my laptop’s network card (an Intel(R) 82577LM adapter), but most newer drivers will log events when a cable is disconnected or the link is otherwise lost. If you are already running EventSentry with its hardware inventory feature enabled, then you can obtain the name of the network adapter from any monitored host on the network through the hardware inventory page, an example is shown below.

Coming up with a way to enable and disable a particular network connection with netsh.exe was a bit more challenging, but I eventually cracked the cryptic command line parameters of netsh.exe.

Enable a connection
netsh interface set interface “Wireless Network Connection” ENABLED

Disable a connection
netsh interface set interface “Wireless Network Connection” DISABLED

And yes, you do need to specify the word “interface” twice. If you do find yourself wanting to automate network adapter settings with scripts and/or the command line frequently, then you should check out this link.

4. The Solution

So now that we have all the ingredients, let’s take a look at the recipe. In order to accomplish the automatic interface toggling, we need to create:

an embedded script called wifi_enable.cmd, using the command line from above
an embedded script called wifi_disable.cmd, again using the command line from above
a process action “Wifi Enable”, referencing the above wifi_enable.cmd embedded script
a process action “Wifi Disable”, referencing the above wifi_disable.cmd embedded script
an event log filter for event source “e1kexpress” and event id 27, triggering the “Wifi Enable” action
an event log filter for event source “e1kexpress” and event id 32, triggering the “Wifi Disable” action

A couple of clarifications: First, you do not need to use embedded scripts, you can create the scripts in the file system too and then point the process action to those files. I prefer embedded scripts since I don’t have to worry about maintaining the script, as it gets distributed to remote hosts automatically when needed. Second, the event source and event id will depend on the network card you have installed on your network, the above example will only work with Lenovo T410 laptops.

So what happens is pretty straightforward: When I connect my laptop to a LAN, the Intel NIC driver writes event id 32 with the event source e1kexpress to the system event log. EventSentry intercepts the event and triggers the Wifi Disable action, which in turns runs the netsh.exe process, disabling the WiFi connection.

5. Setting it up in the management console

Embedded Scripts
You can manage embedded scripts via Tools -> Embedded Scripts. Click “New”, specify a descriptive name (e.g. wifi_enable.cmd) and paste the command line netsh interface set interface “Wireless Network Connection” ENABLED into the script content window. Then, do the same for the wifi_disable.cmd script, but this time use the netsh interface set interface “Wireless Network Connection” DISABLED command line. You can leave the interpreter empty as long as you give the filename the .cmd extension.

Actions

Create two process actions, one pointing to wifi_enable.cmd, and one pointing to wifi_disable.cmd. You can access these embedded scripts by clicking the pull-down – you should see the embedded script(s) you created in step one – each prefixed with the @ symbol. The resulting dialog should look like this:

I recommend enabling both “Event Log Options”, as this will help with troubleshooting. Now we just need the event log filters, and we are all set.

Like I mentioned earlier, you can also reference any external process or .cmd file with the process action, if you’d rather not use embedded scripts.

Event Log Filters
Since we’ll need two filter, we’ll create a new event log package called “Toggle Wifi” by right-clicking the “Event Log Packages” container and selecting “Add Package”. Inside the package we can then add the two filters: One to trigger the “Wifi Enable” action when the NIC drivers logs its event indicating that the network cable was unplugged, and one that will trigger the “Wifi Disable” action when the NIC drivers logs that the network cable was plugged in. The filter will look similar to this, but note that the event source as well as event id will depend on the network card and driver.

That’s pretty much it. If you enabled the event log options in the process action earlier, then you can see the output from the process action in the event log, as shown below:

Here are some links to the official EventSentry documentation regarding the features used:

Why complex passwords may be less secure than you think

August 20, 2011August 15, 2017 ingmar.koecher 0 Comment Event Log, Monitoring, Pure Knowledge, Tips & Tricks password security expiration monitor changes account lockout

Passwords are everywhere. You use them to log on to your network, login to business applications and Facebook, check your personal email, and more.

I’ll be rethinking passwords in this blog, and what you can do to make authenticating with passwords more secure.

As it turns out, the British comedian Nick Helm won an award in Edinburgh for the funniest joke, just one day before I posted this article. He won for the joke: “I needed a password eight characters long so I picked Snow White and the Seven Dwarves.”

Of course, passwords have been around for a while, even though more advanced ways to authenticate like fingerprint readers and biometric scans exist today. Still, passwords prevail as the primary method to authenticate for the majority of networks and computer systems. One-time pads like RSA’s SecurID are another secure alternative, but any system can be exploited as recent events have shown. Even fingerprint readers can be fooled: either with brainpower or through more “traditional” methods.

Password Cracking goes Mainstream
By 1999, Windows NT 4.0 started gaining a lot of traction in networks across the globe. Of course we all know that with popularity comes quantity, and with quantity comes
increased exposure. As more and more networks were using Windows NT (to authenticate among other things), a new piece of software called l0phtcrack (it had a GUI!) was gaining popularity. What l0phtcrack could do – and quite easily I might add – was download all password hashes from the Windows NT user database, and then run both brute-force and dictionary attacks on those hashes.

If a user chose a password that was in the English dictionary, l0phtcrack could often crack it within seconds. If the password was a bit more complicated, it would take a couple of days. Due to the way Windows NT stored password hashes (in the name of compatibility with LanManager), passwords with 7 or fewer characters were particularly easy to crack. And, as CPUs grew
stronger and faster, the time required to run those brute-force attacks kept getting shorter. Of course this general mechanism is and was not restricted to Windows NT and l0phtcrack; you could do the same thing with any password hash. For example, I used a Perl script (utilizing a dictionary text file) back in 2001 against hashes obtained from our NIS system to show the UNIX admins that the NIS installation was, politely speaking, insecure.

So choosing a password that is in any dictionary is clearly not a good idea (and really shouldn’t be allowed when setting the password) since a dictionary attack can be fast. An easy way to prevent against a simple dictionary attack is to require users to choose an additional non-letter character. Since words in dictionaries usually don’t contain characters other than letters,
this is certainly a step in the right direction.

Secure Password for Dummies
Technically, adding a single non-letter character to an English word would indeed prevent a dictionary attack. Yes, a bad one! A persistent and motivated attacker (and most attackers are persistent and motivated) could modify their dictionary attack, and automatically prepend and append numbers to dictionary words, so that a password like “house7” or even
“house1!” could still be found. This may sound like a lot of work, after all this would increase the time a dictionary attack takes around 20-fold “0house, 1house … house0, house1, ….house9”. True, but dictionary attacks are so fast that this technique would still be preferable to a brute-force attack. An attacker would still prefer a 4-hour dictionary attack over a 2-month brute-force attack (I made those numbers up, but the idea is that dictionary attacks are a lot faster). It also turns out that users tend to use the same numbers / special characters in their passwords, e.g. “1!”, “99”, “123” and so forth. Even worse, there appear to be a set of “favorite” passwords: http://www.schneier.com/blog/archives/2006/12/realworld_passw.html.

The obvious way to protect against a dictionary attack is to not use words from a dictionary in the first place. Indeed, many authentication systems require the use of letters with uppercase/lowercase, numbers as well as special characters. A requirement like this will surely protect us against even the most sophisticated dictionary attacks. Mission
accomplished. Easy!

Not so fast. Attackers still have a few more options at their disposal. The attacker can:

Look for software vulnerabilities so that they
can inject their own malware that would give them access to the
server/workstation/network. They would then simply create a new user, reset the
password of an existing user, or – if possible – just download whatever data
they need. Event Log Monitoring can help here, since you can get notified when a new user is created/deleted or
a password is changed (you could setup a filter to email you when a user
password is changed between 11pm and 6am for example).
Employ social engineering techniques to get
access to the password, either through physical access, a phone call or
something similar. A combination of (1) & (2) is most common, as an attacker will
send a malformed PDF (or similar) to the target, which will then implant some
Trojan horse.
Use a brute-force attack to guess the password,
either against an offline database (if the attacker was lucky enough to obtain
one), or run the attack directly against the login system (a web site, a
Windows domain, etc.). Owasp has a good article about brute-force attacks against
web sites, which can be very susceptible to these types of attacks.

Brute-force attacks generally only work in two cases:

One has access to an encrypted password database.
The system one is trying log on to does not
employ an account-lockout technique, so that a brute-force attack can be aimed
directly at a logon portal.

Most network systems do employ an account lockout mechanism, and I highly recommend you enable this on systems which support it. Many systems, in particular web sites, do not support this functionality, however, so brute-force attacks are still a real risk.

When enabling account lockout, it’s important to keep your end users in mind. Your users will ultimately need to log on to a network in order to do their work, and if the system locks them out every time they type in a wrong password twice, then your support team will spend a lot of time unlocking user accounts, and your users (depending on how calm they are)
will be more or less annoyed. A log monitoring solution like EventSentry can email you when an account lockout occurs on a system (e.g. on Windows through the event log, on other devices through Syslog).

A patient system administrator on the phone troubleshooting a logon problemSo, when we brute-force a password, we try every combination regardless of dictionary and such. We start at “a” and make our way to, say, “ZZZZZZZZZZZ”. Consequently, when the required password length is short (say 7 characters), a brute-force attack will be faster than when the required length is large (say 15 characters).

All this begs the ultimate question: Is it better to use a short complex password like C0mP1eX! (8 characters), or a long more simple password like ClimbingUpATree (15 characters)? Time to bring out the calculator. In order to come up with a conclusion, we’ll create three password policies: One that requires complex but shorter passwords, and two with longer but less
complex passwords.

Password Policy 1:
“Complex Is Best”

Minimum length: 8 characters
Required character groups: One lowercase letter, one uppercase letter, one number, one special character out of: !@#$%^&*()_+[]{}
Possible passwords: 1,370,114,370,683,140 (yes, that’s one quadrillion)

That’s mighty complex, but a password like C0mp1ex! would be valid.

Password Policy 2:
“A little long is enough”

Minimum length: 10 characters
Required character groups: One lowercase letter, one uppercase letter, one number. Special characters are allowed but not required.
Possible passwords: 839,299,365,868,340,000 (that’s 839 quadrillion and a little bit)

Of course there would be even more possible passwords if a user decides to include a special character in their password (after all the policy only specifies the minimum requirement, and we wouldn’t dare prohibit additional complex characters now, would we?). This policy is 612 times more complex than the previous policy, even though it
only requires two more characters. It’s flaw, however, is that a user could potentially use insecure passwords like Gardenhose1 which could be guessed with a sophisticated dictionary attack.

Password Policy 3:
“The longer, the better”

Minimum length: 15 characters
Required character groups: One lowercase letter, one uppercase letter
Possible passwords: 54,960,434,128,018,700,000,000,000 (that’s 54 septillions, 960 sextillions, 434 quintillions – you get the idea)

Phew, you need a lot of GPUs and a time machine to brute-force a password from that selection – and that’s without even requiring a user to include a number! This policy is 40 billion (40,117,105,202 to be exact) times more complex than the first policy. And long passwords are not hard to come up with – just use a simple sentence like “Idontlikepasswords55″ is a pretty long password (20 characters) and not that hard to remember at all.

A compromise?
As you can see, length trumps complexity in most cases, but as is often the case in computer security, things aren’t always as simple as they seem. The numbers are correct, but a longer password without complex requirements might, as mentioned before, encourage a user to choose a password that could be guessed with a sophisticated dictionary attack.

For example, “Gardenhose1” would match the 2^ndpolicy’s requirements but not be very secure. Users also tend to use family names, user names and the like in their passwords. A smart attacker could leverage this and adapt their dictionary attack accordingly. So if “Jean Reno” was to use “JeanReno1948” as his password, then this would still not be as secure as assumed – despite the 12-character length.

We can see that every additional character in the length of a password increases the possible combinations exponentially, more so than a few additional special characters. Still, not requiring special characters at all might allow the end user to pick passwords that are in a dictionary. Requiring extremely complicated passwords, on the other hand, will make it difficult for many users to remember them, and your end users might resort to writing their passwords down on a post-it, the bottom of their keyboard, or come up with “secure” passwords like ASDFasdf1!. Yes, annoyed users can be very creative. Put yourself in the shoes of somebody who is not familiar with security and needs to choose a password, would you voluntarily choose something like T3a#fE@8 ?

You do not want your users doing thisPassword expiration policies, while certainly important, can also result in a backlash from your users. Users can get very annoyed with overly ambitious expiration policies, and fight back with number schemes or passwords stored on post-it notes (see http://open.salon.com/blog/unemployedmarx/2011/02/17/changing_passwords if you don’t believe me). The result is the opposite: weaker passwords and less security. How about a “secure” 9-letter password that even exceeds our first policy?

Cats2010!
Cats2011!

Q1cats2011!
Q2cats2011!

This is heaven for an attacker: as long as the user sticks to the same pattern, the attacker will always know the password – even if the user changes it every 3 months. See “Changing Passwords” (http://www.schneier.com/blog/archives/2010/11/changing_passwo.html) for a more thorough discussion on this topic.

So what’s the solution? As often, probably a little bit of everything. Dictionary passwords need to be avoided like the plague, so we’ll never get around requiring some complexity. Complexity alone can be misleading though, so a minimum length of 12 characters seems like a good baseline. In addition, enable account lock-out techniques and set a (reasonable) maximum password age. A pretty good password policy would look like this:

18 characters minimum
Lowercase, uppercase & numbers
180-day password age
No part of first name, last name, username, etc. allowed in password

Also, don’t forget to educate your users, so that people know why and how. Tell them that corporate espionage is a real threat, and suggest the use of a sentence for a password. Of course there will always be naysayers, but the majority of your user base should understand this.

Abuse
I have seen web sites (e.g. banking) require me to use a complex password, yet require that it shall be no longer than 8 characters! Whatever the reason behind something like that, it’s far from secure and counterproductive. Even if I’d want to choose a strong 14-letter password I couldn’t, and I would have to settle for something less secure.

Password Reuse
Another often overlooked risk is the reuse of passwords. Nowadays, people are required to use passwords at a multitude of web sites and systems. Some of those web sites store confidential information (SSN, credit card), but many don’t.

The more often one uses the same password, the higher the risk that it is compromised. As such, your password – if used at more than one place – is only as strong as its weakest link. Don’t use the same password that you use for your banking web site on your photo-sharing site!

I personally don’t care too much if some cracker hacks the photo-sharing site I use, and downloads (and cracks) all the passwords. But I do care if my password to my banking web site is compromised. An attacker may not be able to easily guess a password at bankofamerica.com, but if I use the same password as my photo-sharing site, then I’m just asking for trouble. Recycling and reuse are a good thing – but not with passwords.

I hope this longer than expected article inspired you to review your corporate password policy, and maybe even your personal password habits. If you made it this far then I have included some relevant links regarding … well … passwords!

Nick Helm’s password joke:
http://www.bbc.co.uk/news/uk-scotland-14646532

Interesting Statistics:
http://www.passwordresearch.com/stats/statindex.html

A Strong Password Isn’t the Strongest Security:
http://www.nytimes.com/2010/09/05/business/05digi.html

Graphical Passwords: A Survey:
http://www.acsac.org/2005/papers/89.pdf

So long,
Ingmar.

EventSentry v2.92 + iPhone App

June 23, 2011March 11, 2025 ingmar.koecher 0 Comment Announcements, EventSentry, Monitoring eventsentry 2.92 release iphone mobile app performance charts

EventSentry v2.92
The main new features in version 2.92 are:

Network Services component
Embedded performance / environment charts
HTTP action
Hardware Inventory & Monitoring

Please see the release history or the documentation for a complete list of all bug fixes and new features.

Network Services
The network services component is the main new feature in EventSentry, and adds powerful functionality – the ability to receive SNMP traps – to our monitoring solution. Any network device that supports SNMP v1, v2c or v3 can be configured to send traps to EventSentry, which can then either forward the trap details via email, or log them to the database for archival and searching purposes.

Unlike other products, configuring the snmp trap daemon is easy and uncomplicated. Simply specify any MIBs you want to load, configure any SNMP communities and/or v3 users, and (optionally) setup text filters to include/exclude specific traps based on their content.

The network services component now also includes the syslog daemon, which was previously embedded in the EventSentry agent. This change includes higher availability, since the network services will cache incoming (snmp and/or syslog) packets if the main database is temporarily unavailable.

Another new “feature” is the ability to run the network services daemon on Linux and OS X operating systems. We developed all code in the network services to run on multiple platforms, and will continue to do so with all upcoming new components in EventSentry. Cross-platform functionality is still experimental, and as such we don’t currently have an installer for non-Windows operating systems. If you are interested in running the network services daemon on Linux or OS X, then please send a request to our support team.

Visualizing performance and environment data with embedded charts
Performance monitoring and alerts have always been a popular and useful feature in EventSentry, but a numerical alert can only convey so much information. If an alert shows that the CPU usage has been exceeded, say 80% over a time period of 10 minutes, then it is often helpful to see the performance data in a chart. This would show whether the CPU usage spiked during the beginning of the 10-minute interval and is back in an acceptable range, or whether the CPU usage has been mostly constant. In the past, you would need to review the performance history through the web-based reporting (which is still an option and desirable for long-term analysis), but starting with v2.92 email alerts can now include an embedded chart which shows exact performance information from the monitored time interval.

For example, if your performance alert is configured to generate an alert when the CPU usage exceeds 90% for more than 30 minutes, then you can receive an attachment in addition to the text alert, showing performance data from the last 30 minutes.

Embedded charts are also included with temperature and/or humidity environment alerts, an example chart is shown below:

HTTP Action
To better interact with web-based solutions like iService, EventSentry now includes a new notification type: HTTP. With the new HTTP action, you can submit event data to any HTTP or HTTPS-based form, making it extremely easy and reliable to interact with web-based systems (e.g. ticketing systems). Setting up a HTTP action is easy: Simply provide the URL, authentication and specify any event fields that you want to submit.

Hardware Inventory & Monitoring
Also new are network link speed monitoring, which can alert you when the NIC speed changes. This can be useful if a host is accidentally connected to a different switch, or simply to inventory all network connections.

EventSentry also monitors connected USB storage devices and can alert you when a drive is added or removed. The S.M.A.R.T. status of drives is also monitored, and junction points are now taken into consideration when evaluating the disk space of a logical drive. Also new is a “physical disks” section in the web reports.

iPhone App
Why yes, there is an app for that! How convenient would it be to review the health status of all your monitored hosts from your iPhone – including health statistics, hardware details and more?

Dream no more, with the EventSentry iPhone app you can review the status of all your servers from your phone (so long as it has access to your web reports of course).

Upon launch the app shows a searchable list of all monitored hosts and their respective health status. Tapping on a host will show detailed information, such as performance data, hardware information and more. Please take a look at the official iTunes page for the EventSentry app for more information and screenshots.

If you are an Android user then there is no reason to worry, an Android version of the EventSentry Mobile app is planned later this year.

Again, please see the version history for a complete list of features and fixes in v2.92.

Do not trust thee RAID alone

February 19, 2011May 28, 2019 ingmar.koecher 0 Comment Miscellaneous, Monitoring, Pure Knowledge, Tips & Tricks raid server availability redundancy backups raid array levels

I’m assuming that most readers are familiar with what RAID, the “Redundant Array of Inexpensive Disks”, is. Using RAID for disk redundancy has been around for a long time, apparently first mentioned in 1987 at the University of California, Berkeley (see also: The Story So Far: The History of RAID). I’m honestly not sure why they chose the term “inexpensive” back in 1987 (I suppose “RAD” isn’t as catchy of a name), but regardless of the wording, a RAID is a fairly easy way to protect yourself against hard drive failure. Presumably, any production server will have a RAID these days, especially with hard drives being as inexpensive as they are today (unless you purchase them list price from major hardware vendors, that is). Another reason why RAID is popular, is of course the fact that hard drives are probably the most common component to break in a computer. You can’t really blame them either, they do have to spin an awful lot.

Source: ServerBurn4 by Topato

Lesson #1: Don’t neglect your backups because you are using RAID arrays
That being said, we recently had an unpleasant and unexpected issue in our office with a self-built server. While it is a production server, it is not a very critical one, and as such a down-time of 1-2 days with a machine like that is acceptable (albeit not necessarily desired). Unlike the majority of our “brand-name” servers, which are under active support contracts, this machine was using standard PC components (it’s one of our older machines), including an onboard RAID that we utilized for both the OS drive as well as the data drive (it has four disks, both in a RAID 1 mirror). Naturally, the machine is monitored through EventSentry.

Well, one gray night it happened – one of the hard drives failed and a bunch of events (see myeventlog.com for an example) were logged to the event log, and immediately emailed to us. After disappointingly reviewing the emails, the anticipated procedure was straightforward:

1) Obtain replacement hard drive
2) Shut down server
3) Replace failed hard drive
4) Boot server
5) Watch RAID rebuilding while sipping caffeinated beverage

The first 2 steps went smoothly, but that’s unfortunately how far our IT team got. The first challenge was to identify the failed hard drive. Since they weren’t in a hot-swappable enclosure, and the events didn’t indicate which drive had failed, we chose to go the safe route and test each one of them with the vendors supplied hard drive test utility. I say safe, because it’s possible that a failed hard drive might work again for a short period of time after a reboot, so without testing the drives you could potentially hook the wrong drive up. So, it’s usually a good idea to spend a little bit of extra time in that case, to determine which one the culprit is.

Eventually, the failed hard drive was identified, replaced with the new (exact and identical) drive, connected, and booted again. Now normally, when connecting an empty hard drive, the raid controller initiates a rebuild, and all is well. In this case however, the built-in NVidia RAID controller would not recognize the RAID array anymore. Instead, it congratulates us on having installed two new disks. Ugh. Apparently, the RAID was no more – it was gone – pretty much any IT guys nightmare.

No matter what we tried, including different combinations, re-creating the original setup with the failed disks, trying the mirrored drive by itself, the RAID was simply a goner. I can’t retell all the things that were tried, but we ultimately had to re-create the RAID (resulting in an empty drive), and restore from backup.

We never did find out why the RAID 1 mirror that was originally setup was not recognized anymore, and we suspect that a bug in the controller firmware caused the RAID configuration to be lost. But regardless of what was ultimately the cause, it shows that even entire RAID arrays may fail. Don’t relax your backup policy just because you have a RAID configured on a server.

Lesson #2: Use highly reliable RAID levels, or configure a hot spare
Now I’ll admit, the majority of you are running your production servers on brand-name machines, probably with a RAID1 or RAID5, presumably under maintenance contracts that ship replacement drives within 24 hours or less. And while that does sound good and give you comfort, it might actually not be enough for critical machines.

Once a drive in a RAID5 or RAID1 fails, the RAID array is in a degraded state and you’re starting to walk on very thin ice. At this point, of course, any further disk failure will require a restore from backup. And that’s usually not something you want.

So how could a RAID 5 not be sufficiently safe? Please, please: Let me explain.

Remember that the RAID array won’t be fully fault tolerant until the RAID array is rebuilt – which might be many hours AFTER you plug in the repaired disk depending on the size, speed and so forth. And it is during the rebuild period that the functional disks will have to work harder than usual, since the parity or mirror will have to be re-created from scratch, based on the existing data.

Is a subsequent disk failure really likely though? It’s already pretty unlikely a disk fails in the first place – I mean disks don’t usually fail every other week. It is however much more likely than you’d think, somewhat depending on whether the disks are related to each other. What I mean with related, is whether they come from the same batch. If there was a problem in the production process – resulting in a faulty batch – then it’s actually quite likely that another bites the dust sooner rather than later. It happened to a lot of people – trust me.

But even if the disks are not related, they probably still have the same age and wear and, as such, are likely to fail in a similar time frame. And, like mentioned before, the RAID array rebuild process will put a lot of strain on the existing disks. If any disk is already on its last leg, then a failure will be that much more likely during the RAID array rebuild process.

RAID 6, if supported by your controller, is usually preferable to a RAID5, as it includes two parity blocks, allowing up to two drives to fail. RAID 10 is also a better option with potentially better performances, as it too continues to operate even when two disks fail (as long as it’s not the disks that are mirrored). You can also add a hot spare disk, which is a stand-by disk that will replace the failed disk immediately.

If you’re not 100% familiar with the difference between RAID 0, 1, 5, 6, 10 etc. then you should check out this Wikipedia article: It outlines all RAID levels pretty well.

Of course, a RAID level that provides higher availability is usually less efficient in regards to storage. As such, a common counterargument against using a more reliable RAID level is the additional cost associated with it. But when designing your next RAID, ask yourself whether the savings of an additional hard drive is worth the additional risk, and the potential of having to restore from a backup. I’m pretty sure that in most cases, it’s not.

Lesson #3: Ensure you receive notifications when a RAID array is degraded
Being in the monitoring business, I need to bring up another extremely
important point: Do you know when a drive has failed? It doesn’t help much to have a RAID when you don’t know when one or more drives have failed.

Most server
management software can notify you via email, SNMP and such – assuming
it’s configured. Since critical events like this almost always trigger
event log alerts as well though, a monitoring solution like EventSentry can simplify the notification process.
Since EventSentry monitors event logs, syslog as well as SNMP traps, you can take a uniform approach to notifications. EventSentry can notify you of RAID failures regardless of the hardware vendor you
use – you just need to make sure the controller logs the error to the
event log.

Lesson #4+5: Test Backups, and store backups off-site
Of course one can’t discuss reliability and backups without preaching the usual. Test your backups, and store (at least the most critical ones) off-site.

Yes, testing backups is a pain, and quite often it’s difficult as well and requires a substantial time commitment. Is testing backups overkill, something only pessimistic paranoids do? I’m not sure. But we learned our lessen the hard way when all of our 2008 backups were essentially incomplete, due to a missing command-line switch that recorded (or in our case did not) the system state. We discovered this after, well, we could NOT restore a server from a backup. Trust me: Having to restore a failed server and having only an incomplete, out-of-date or broken backup, is not a situation you want to find yourself in.

My last recommendation is off-site storage. Yes, you have a sprinkler system, building security and feel comfortably safe. But look at the picture on top. Are you prepared for that? If not, then you should probably look into off-site backups.

So, let me recap:

1. Don’t neglect your backups because you are using RAID arrays.
2. Use highly reliable RAID levels, or configure a hot spare.
3. Ensure you receive notifications when a RAID array is degraded
4. Test your backups regularly, but at the very least test them once to ensure they work.
5. Store your backups, or at least the most critical, off-site.

Stay redundant,
Ingmar.

Creating your very own event message DLL

November 4, 2010June 13, 2022 ingmar.koecher 1 Comment Event Log, Monitoring, Pure Knowledge, Tips & Tricks, Tools & Utilities log to event log message file DLL perl python

If you’ve ever wrote code to log to the Windows event log before (e.g. through Perl, Python, …), then you might have run into a similar problem that I described in an earlier post: Either the events don’t look correctly in the event log, you are restricted to a small range of event ids (as is the case with eventcreate.exe) or you cannot utilize insertion strings.

In this blog post I’ll be showing you how to build a custom event message DLL, and we’ll go about from the beginning to the end. We’ll start with creating the DLL using Visual Studio (Express) and finish up with some example scripts, including Perl of course, to utilize the DLL and log elegantly to the event log.

Let’s say you are running custom scripts on a regular basis in your network – maybe with Perl, Python, Ruby etc. Your tasks, binary as they are, usually do one of two things: They run successfully, or they fail. To make troubleshooting easier, you want to log any results to the event log – in a clean manner. Maybe you even have sysadmins in other countries and want to give them the ability to translate standard error messages. Logging to the event log has a number of benefits: It gives you a centralized record of your tasks, allows for translation, and gives you the ability to respond to errors immediately (well, I’m of course assuming you are using an event log monitoring solution such as EventSentry). Sounds interesting? Read on!

Yes, you can do all this, and impress your peers, by creating your own event message file. And what’s even better, is that you can do so using all free tools. Once you have your very own event message file, you can utilize it from any application that logs to the event log, be it a PowerShell/perl/python/… script or a C/C++/C#/… application.

To create an event message file, you need two applications:

The reason you need the platform SDK, is because Visual Studio Express does not ship with the Message Compiler, mc.exe, for some reason. The message file compiler is essential, as without it there will be no event message file unfortunately. When installing the platform SDK, you can deselect all options except for “Developer Tools -> Windows Development Tools -> Win32 Development Tools” if you want to conserve space. This is the only essential component of the SDK that’s needed.

An event message file is essentially a specific type of resource that can be embedded in either a DLL file or executable. In EventSentry, we originally embedded the message file resources in a separate DLL, but eventually moved it into the executable, mostly for cleaner and easier deployment. We’ll probably go back to a separate message DLL again in the future, mostly because processes (e.g. the Windows Event Viewer) can lock the event message file (the executable in our case), making it difficult to update the file.

Since embedding an event message file in a DLL is more flexible and significantly easier to accomplish, I’ll be covering this scenario here. The DLL won’t actually contain any executable code, it will simply serve as a container for the event definitions that will be stored inside the .dll file. While it may sound a little bit involved to build a DLL just for the purpose of having an event message file (especially to non-developers), you will see that it is actually surprisingly easy. There is absolutely no C/C++ coding required, and I also made a sample project available for download, which has everything setup and ready to go.

In a nutshell, the basic steps of creating an event message file are as follows:

1. Create a message file (e.g. messagefile.mc)
2. Convert the message file into a DLL, using mc.exe, rc.exe and link.exe

Once we have the message file, we will also need to register the event message file in the registry, and associate it with an event source. Keep in mind that the event source is not hard-coded into the message file itself, and in theory a single event message file could be associated with multiple event sources (as is the case with many event sources from Windows).

So let’s start by creating a working folder for the project, and I will call it “myapp_msgfile”. Inside that directory we’ll create the message file, let’s call it myapp_msgfile.mc. This file is a simple text file, and you can edit it with your favorite text editor (such as Ultraedit, Notepad2 or Notepad++).

The file with the .mc extension is the main message file that we’ll be editing – here we define our event ids, categories and so forth. Below is an example, based on the scenario from before. Explanations are shown inline.

MessageIdTypedef=WORD

LanguageNames=(
English=0x409:MSG00409
German=0x407:MSG00407
)

Here we define which languages we support, and by which files these languages will be backed. You will have to look up the language id for other languages if you plan on supporting more, and you can remove German if you only plan on supporting English.

MessageId=1
SymbolicName=MYTOOL_CATEGORY_GENERAL
Language=English
Tasks
.
Language=German
Jobs
.

Our first event id, #1, will be used for categories. Categories work in the exact same way as event ids. When we log an event to the event log and want to include a category, then we only log the number – 1 in this case.

MessageId=100
SymbolicName=TASK_OK
Language=English
Task %1 (%2) completed successfully.
.
Language=German
Job %1 (%2) war erfolgreich.
.

This is the first event description. The “MessageId” field specifies the event id, and the symbolic name is a descriptive and unique name for the event. The language specifies one of the supported languages, followed by the event message text. You end the event description with a single period – that period has to be the only character per line.

MessageId=101
SymbolicName=TASK_ERROR
Language=English
Task %1 (%2) failed to complete due to error “%3”.
.
Language=German
Job %1 (%2) konnte wegen Fehler “%3” nicht abgeschlossen werden.
.

MessageId=102
SymbolicName=TASK_INFO
Language=English
Task Information: %1
.
Language=German
Job Information: %1
.

Since we’re trying to create events for “custom task engine”, we need both success and failure events here. And voila, our event message file now has events 100 – 102, plus an id for a category.

So now that we have our events defined, we need to convert that into a DLL. The first step now is to use the message compiler, mc.exe, to create a .rc file as well as the .bin files. The message compiler will create a .bin file for every language that is defined in the mc file. Open the “Visual Studio Command Prompt (2010)” in order for the following commands to work:

mc.exe myapp_msgfile.mc

will create (for the .mc file depicted above):

myapp_msgfile.rc
msg00407.bin
msg00409.bin

With those files created, we can now create a .res (resource) file with the resource compiler rc.exe:

rc.exe /r myapp_msgfile.rc

which will create the

myapp_msgfile.res

file. The “/r” option instructs the resource compile to emit a .res file. Now we’re almost done, we’re going to let the linker do the rest of the work for us:

link -dll -noentry -out:myapp_msgfile.dll myapp_msgfile.res

The myapp_msgfile.res is the only input file to the linker, normally one would supply object (.obj) files to the linker to create a binary file. The “-noentry” option tells the linker that the DLL does not have an entry point, meaning that we do not need to supply a DllMain() function – thus the linker is satisfied even without any object files. This is of course desired, since we’re not looking to create a DLL that has any code or logic in it.

After running link.exe, we’ll end up with the long awaited myapp_msgfile.dll file.

The end. Well, almost. Our message file is at this point just a lone accumulation of zeros and ones, so we need to tell Windows that this is actually a message file for a particular event log and source. That’s done through the registry, as follows:

Open the registry editor regedit.exe. Be extremely careful here, the registry editor is a powerful tool, and needs to be used responsibly :-).

All event message files are registered under the following key:

HKLM\System\CurrentControlSet\Services\eventlog

Under this key, you will find a key for every event log as well as subkeys for every registered event source. So in essence, the path to an event source looks like this:

HKLM\System\CurrentControlSet\Services\eventlog\EVENTLOG\EVENTSOURCE

I’m going to assume here that we are going to be logging to the application event log, so we’d need to create the following key:

HKLM\System\CurrentControlSet\Services\eventlog\Application\MyApp

In this key, we need to following values:

TypesSupported (REG_DWORD)
EventMessageFile (REG_EXPAND_SZ)

TypesSupported is usually 7, indicating that the application will log either Information, Warning or Error events (you get 7 if you OR 1[error], 2[warning] and 4[information] together).

EventMessageFile is the path to your message DLL. Since the type is REG_EXPAND_SZ, the path may contain environment variables.

If you plan on utilizing categories as well, which I highly recommend (and for which our message file is already setup), then you need two additional values:

CategoryCount (REG_DWORD)
CategoryMessageFile (REG_EXPAND_SZ)

CategoryCount simply contains the total number of categories in your message file (1, in our case), and the CategoryMessageFile points to our message DLL. Make sure that your message file does not contain any sequence gaps, so if your CategoryCount is set to 10, then you need to have an entry for every id from 1 to 10 in the message file.

We could create separate message files for messages and categories, but that would be overkill for a small project like this.

Now that we have that fancy message DLL ready to go, we need to start logging. Below are some examples of how you can log to the event log with a scripting language. I’ll be covering Perl, Kix, and Python. Me being an old Perl fan and veteran, I’ll cover that first.

PERL
The nice thing about Perl is that you can take full advantage of insertion strings, so it can support event definitions containing more than one insertion string.

use strict; use Win32::EventLog;

# Call this function to log an event

sub logMessage
{
my ($eventID, $eventType, @eventDetails) = @_;

my $evtHandle = Win32::EventLog->new(“Your Software Application”);

my %eventProperties;

# Category is optional, specify only if message file contains entries for categories
$eventProperties{Category} = 0; $eventProperties{EventID} = $eventID; $eventProperties{EventType} = $eventType; $eventProperties{Strings} = join("\0", @eventDetails);

$evtHandle->Report(\%eventProperties);

$evtHandle->Close;
}

# This is what you would use in your scripts to log to the event log. The insertion strings
# are passed as an array, so even if you only have one string, you would need to pass it
# within brackets (“This is my message”) as the last parameter

logMessage(100, EVENTLOG_INFORMATION_TYPE, (“Database Backup”, “Monitoring Database”, “Complete”));
logMessage(102, EVENTLOG_INFORMATION_TYPE, (“Step 1/3 Complete”));

PYTHON
Python supports event logging very well too, including multiple insertion strings. See the sample code below:

import win32evtlogutil import win32evtlog

# Here we define our event source and category, which we consider static throughout
# the application. You can change this if the category is different

eventDetails = {‘Source’: ‘MyApp’, # this is id from the message file
‘Category’: 1} # which was set aside for the category

# Call this function to log an event

def logMessage(eventID, eventType, message, eventDetails):
if type(message) == type(str()):
message = (message,)
win32evtlogutil.ReportEvent(eventDetails[‘Source’], eventID, eventDetails[‘Category’], eventType, tuple(message))

logMessage(100, win32evtlog.EVENTLOG_INFORMATION_TYPE, (“Database Backup”, “Monitoring Database”), eventDetails)
logMessage(102, win32evtlog.EVENTLOG_INFORMATION_TYPE, (“Step 1/3 complete”), eventDetails)

KIXTART
The pro: Logging to the event log using KiXtart is so easy it’s almost scary. The con: It only supports message files that use one insertion string.

LOGEVENT(4, 102, "Database Backup", "", "MyApp")

EventSentry Blog

Category: Monitoring

How to dynamically toggle your Wireless adapter with EventSentry

Why complex passwords may be less secure than you think

EventSentry v2.92 + iPhone App

Do not trust thee RAID alone

Creating your very own event message DLL