I’ll start with, the code link and quick doc are at the end of this post. If you’re not into reading the backstory, feel free to scroll on down and grab it. OTHERWISE…
Someone I demo’d to a while back posed a neat idea. They had used other monitoring software in the past and one thing he didn’t want to happen was alert fatigue. This is a real problem actually for any monitoring software and the team it feeds alerts to. We chatted for a bit about this, and suffice to say, Uptime has a lot of built-in intelligence with regards to how it should get in touch with your team members and even how that might change for different times of day and severities of alerts. Yes I’m talking about escalation paths, alternate contact methods, and not bugging the ever-living heck out of managers who might otherwise be copied on IT distribution lists. I won’t bother you with the entire conversation as it’s not as relevant here. I do however want to talk about having your monitor actually speak its mind. Literally. I see monitors on NOC walls all the time. Maybe you even have one behind you.
Behind you is the operative word. I’ve seen (and worked with) a room of folks, everyone with their backs to a bank of 50 inch plasmas (but hey they kept it nice and warm in there!). This generally doesn’t do much more than provide an “ahhh” factor for people touring the NOC. This, thankfully, doesn’t happen often. But I digress. So this guy I mentioned earlier, while commiserating for a bit, says to me, “Hey, it would be real cool if you could make our monitor make some kind of noise or speak the alert when one popped up.” I thought about this a while and agreed. It could either be really cool or just as annoying as getting lots of emails. So let’s explore this idea a bit.
First of all, alert fatigue HAPPENS because of volume first, and irrelevance second. Mix the two and you have a fatal, annoying, cocktail. This I believe (and marketing emails) is why the email sort / filter rule was invented! The first thing you have tasked to you now is to have relevant alerts. Personally, I don’t see a reason to knee jerk email on things like CPU consumption. In my continual effort to be pragmatic about all things save my project car and bacon, this is a subject worth breaching (the alerting not the bacon.) Without getting too deep into it, as it would make another great blog post, Uptime Infrastructure Monitor (UIM from here on out) has the ability to alert on aggregate conditions. We call them “applications” in Uptime. Imagine you configure UIM to monitor each piece of the puzzle that makes a system function and function well. For something like SharePoint that would be the web servers, a few web page load times, application servers, the SQL server and some performance counters, and general availability and performance across those technologies. In UIM, we would create applications to represent the multitude of components that SharePoint is made of. Maybe an application that represents active directory, the SQL cluster, user experience for a few team sites, etc. Then one more application that includes all of those… We don’t have to get crazy deep, we just need to know does each piece work and do they work OK. Pragmatic huh? Now we get the option to alert on any bit of it along the way, but wouldn’t you rather get an email like, “SharePoint user experience is critical and has been so for 5 minutes.” and another telling you perhaps the database performance is in the toilet? Pretty quickly you get the idea where the problem likely is… Now before I get off into the weeds more, let me stop there. You’re going to alert however you like, to whomever, whenever, you like. That’s part of the beauty of this product.. Right alerts to the right folks at the right time for the right reasons. For the example here we’re going to create a spoken alert on the application level itself and not down in the details. While I may suffer from excessive verbosity, your monitoring station doesn’t have to.
Now you’ve gotten all that setup, let’s make it TALK! YEAH! BACON TIME!
The first bit. Powershell remoting… This is a likely necessity unless your monitoring station is a physical box that said monitor you wish to make talk is plugged directly into. It is likely not, and is some old workstation people, including your asset management folks (or your niece), have forgotten about. We need to enable Powershell remoting on the box that we want to talk in order to command it to do so. It is disabled by default because well, with the right credentials you can do anything with PowerShell remoting. Don’t get any ideas buster! Or do, but don’t blame them on me 🙂
Step 1: Enable-PSRemoting on the remote workstation that will be speaking.
- Open Powershell as administrator. This must be run on Server 2012 and newer or any machine where PSRemoting is disabled.
- Run Enable-PSRemoting
If you are unable to run Enable-PSRemoting and receive an error like below
Set-WSManQuickConfig : <f:WSManFault xmlns_f=”schemas.microsoft.com/…/wsmanfault” Code=”2150859113″
Machine=”localhost”><f:Message><f:ProviderFault provider=”Config provider”
path=”%systemroot%system32WsmSvc.dll”><f:WSManFault xmlns_f=”schemas.microsoft.com/…/wsmanfault”
Code=”2150859113″ Machine=”LAPTOPIA.demobox.loc”><f:Message>WinRM firewall exception will not work since one of the
network connection types on this machine is set to Public. Change the network connection type to either Domain or
Private and try again. </f:Message></f:WSManFault></f:ProviderFault></f:Message></f:WSManFault>
At line:116 char:17
+ Set-WSManQuickConfig -force
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [Set-WSManQuickConfig], InvalidOperationException
+ FullyQualifiedErrorId : WsManError,Microsoft.WSMan.Management.SetWSManQuickConfigCommand
THEN
Run Get-NetConnectionProfile to see what adapter is in the Public state.
PS C:WINDOWSsystem32> Get-NetConnectionProfile
Name : demobox.loc
InterfaceAlias : vEthernet (internal)
InterfaceIndex : 24
NetworkCategory : DomainAuthenticated
IPv4Connectivity : NoTraffic
IPv6Connectivity : NoTraffic
Name : meanie5 2
InterfaceAlias : Wi-Fi
InterfaceIndex : 16
NetworkCategory : Public
IPv4Connectivity : Internet
IPv6Connectivity : NoTraffic
Run
Set-NetworkConnectionProfile -Name “meanie5 2” -NetworkCategory Private
Re-run Enable-PSRemoting
And you should be good to go.
Next :
Enable unsigned script execution (or go and sign it!)
On your monitoring station, open PowerShell in administrative mode
Run
Set-ExecutionPolicy bypass
Drop the files from Github (the bat and PS1 file) in the uptime/scripts folder. Edit the batch file
C:WindowsSysWOW64WindowsPowerShellv1.0powershell.exe -ExecutionPolicy Bypass -file c:uptimescriptsspeakalert.ps1 HOST_TO_MAKE_TALK female 1 “The service, %UPTIME_SERVICENAME%, has entered the %UPTIME_SERVICESTATE% state, and has been so for %UPTIME_DURATION%. its host, %UPTIME_HOSTNAME%, is in the %UPTIME_HOSTSTATE% status.”
Leave the % enclosed stuff alone. You’ll need to change HOST_TO_MAKE_TALK with the resolvable hostname of the machine from the first step. IF you want, go ahead and edit the alert message text. I wasn’t a literature major afterall (can you tell… U of H Honor’s College, BSc Psychology baby! Go Coogs!). You also will likely need to modify the script path. I always use C:uptime when I install because it is just SO MUCH EASIER! But if you didn’t that’s ok, just put it in there. If you used program files you will likely want to use Progra~1 (another reason to use c:uptime!!!)
The last bit is super easy. Just create an action profile to launch it. From the services tab up top, then on the left, create action profile… It should look something like so:
This is because the action profiles ALWAYS run on the monitoring station. Now they may call rexec to execute a command on an agent bound server, or do some SNMP private command to shut a port down on your router.. Heck, use your imagination, but regardless, it all starts from the monitoring station.
So we call our batch file, it executes PowerShell, that executes our script, that opens the remote command pathway to the machine plugged into your screen and that makes it talk. Now, this DOES ASSUME you have either an HDMI cable connecting a big TV to said computer and that has audio and the volume is turned up or you have external speakers, etc…
Now I did sneak some parameters in here. If you want it to talk in a smoky scottish accent, well you’re outta luck. Microsoft hasn’t given us that option yet. Feature request guys! Also can we chat about those boring progress bars?? Anyway. You’ll notice I put speed and gender in there. Feel free to tweak it of course if you want. I expect the more we get into voice stuff the more voices they’ll include.
As always, comments, feedback, etc are appreciated. If you made it this far, THANKS for reading! Here’s the link to the Github repository along with the quickie DOC to get you going.