Aggregating Logs And Monitoring Them With up.time and logstash

by Sep 1, 2015

Log monitoring is an important part of understanding your systems’ and applications’ health and are a valuable tool for troubleshooting when there are issues.  Traditional endpoint monitoring can tell you if something is up or down, and performance metrics can hint at root cause, but it’s in the logs that you can find directly actionable troubleshooting information. You may have used up.time’s Log File Monitor (available on the Grid) before to monitor logs on your systems. This works well, but in many cases customers want to aggregate and manage their logs in a central location – either out of convenience or due to compliance requirements.

We are considering adding more full-featured log management, aggregation, and monitoring to up.time and would love to hear your opinions on that. In the meantime, if you want to aggregate your logs and monitor them in up.time, here’s a way we’ve tested and recommend to do that.

Welcome To Logstash

To aggregate logs, you will want to run Logstash. This consists of a server, which will hold the aggregate logs, and forwarders on each system with logs you want to aggregate. Logstash is Apache2 licensed open source.

These instructions assume a Windows logstash server because that is the most common (and complex) case; a Linux server would work slightly more smoothly if it is an option.

Log Aggregation With Logstash

Install Logstash Server

  1. Download the Logstash 1.5.1 (or later) server software from https://www.elastic.co/downloads/logstash to the server on which you want to aggregate your logs. This does not have to be the up.time server, just any server where you will have the up.time agent with the Log File Monitor plugin installed.
  2. Ensure you have Java JDK v7 or higher installed on the server by running “java -version” (the Logstash server is a Java application).  Add a system variable named “JAVA_HOME” (System/Properties/Advanced/Environment Variables) with the path to the JDK (for example,  C:\Program Files\Java\jdk1.8.0_05). Logstash-forwarder is a go app and so should run natively without additional install requirements.
  3. Unzip logstash into a directory on the logstash server, for example c:\logstash.
  4. Create a file called “logstash.conf” and put it into the logstash server directory. Configure the server.  The logstash forwarder uses the “lumberjack” protocol so in logstash.conf add a stanza:
input {
  lumberjack {
    port => 12345
    ssl_certificate => "C:/logstash-1.5.1/logstash.crt"
    ssl_key => "C:/logstash-1.5.1/logstash.key"
  }
}
filter {
  if [type] =~ "windows" {
    mutate {
      split => { "file" => ":" }
	  }
	}
		
	if [type] =~ "windows" {
    mutate {
	  replace => { "file" => "%{file[1]}" }
	  }
	}
}
output {
  file {
    path => "C:/logstash-1.5.1/logs/%{host}/%{file}.%{+yyyy-MM-dd}"
	message_format => "%{message}"
  }
}
  1. This should pull the files across and create a directory tree mirroring the directories your logs are coming from (the mutate filters are a hack to remove the drive letter, e.g. C:, from the beginning of Windows paths).
  2. Naturally the logstash forwarder will need to be able to reach port 12345 on your logstash server. Make any network or Windows firewall changes required for this.
  3. Generating the SSL certificate should be done however you usually do it in your environment – here is a cert generation HOWTO that might help for a pure self-signed cert: http://www.akadia.com/services/ssh_test_certificate.html
  4. You can run the server to test it from the command line just as “logstash agent -f logstash.conf”. You should see:
    c:\logstash-1.5.1\bin>logstash agent -f logstash.conf
    io/console not supported; tty will not be manipulated
    Logstash startup completed
    Just control-C to stop it.
  5. Now we will set up the server to run as a Windows service. Download http://nssm.cc/download and unzip it, we’ll assume to C:\nssm. (If you have some other preferred way of turning a general executable into a Windows service you can use it as well).
  6. Now open a Command Prompt using “run as administrator” and type:
    cd C:\logstash\bin
  7. And then type the following:
    C:\nssm\win64\nssm.exe install logstash
  8. In the popup GUI, fill in the following:
  9. Path: C:\logstash\bin\logstash.bat
  10. Startup directory:C:\logstash\bin
  11. Arguments: agent -f C:/logstash/bin/logstash.conf
  12. Start the logstash service on the server. It will show up as “nssm” in the services list.

Install Logstash Forwarders

  1. Download the logstash forwarders for the platforms you need to collect logs from also from https://www.elastic.co/downloads/logstash. On windows this is logstash-forwarder_windows_386.exe, depending on your Windows version you may need to do something to allow it to execute (like go to its Properties and hit the “allow execution of downloaded file” button.
  2. Put the forwarder into a directory like C:\logstash-forwarder on the target box. (See https://github.com/elastic/logstash-forwarder for more details)
  3. Create a file called “logstash-forwarder.conf” in the logstash-forwarder directory.
  4. Configure the client. In logstash-forwarder.conf add something like the following, altered for your environment. The type of windows is used to parse the filename correctly (dropping the drive letter) in the logstash.conf.
{
{
    "network": {
        "servers": [
            "localhost:12345"
        ],
        "ssl certificate": "C:/logstash-1.5.1/logstash.crt",
        "ssl key": "C:/logstash-1.5.1/logstash.key",
        "ssl ca": "C:/logstash-1.5.1/logstash.crt",
        "timeout": 15
    },
    "files": [
        {
            "paths": [
                "C:/Program Files/uptime software/uptime/logs/*"
            ],
			"fields": { "type": "windows" }
        }
    ]
}
  1. You then need to be careful with your initial load.  By default the forwarder, when run, will send all files touched today but it will send them from their very beginning.  This will end up resending chunks of log files.  So you will want to run it once normally first to get the entire files, and then stop it and run it again as “logstash-forwarder -config logstash-forwarder.conf -tail=true”. This tells it to follow the end of the file instead of pulling the entire file when it restarts.
  2. Turn it into a service as well using nssm using steps similar to those described for the server above or use your favorite alternate method. If you have very high volume logs you may want to raise the spool size with the command line parameter “-spool-size 100” or similar.

Alternate Option: NXLog

If you need more Windows functionality around event logs than logstash-forwarder has, or if you have trouble getting it running, you can instead use NXLog (http://nxlog.org/products/nxlog-community-edition/download) as the log shipper. http://stackoverflow.com/a/26874871 has examples of NXLog and logstash configuration to use to hook them up.

Another alternative option is to use syslog on Linux and syslog-ng on Windows and then use a syslog input on the server.

Helpful Tips On Logstash Installation and Configuration

  1. Warning note: Later versions of Windows get upset when you try to run .exe’s you have downloaded from the Internet. You will often have to go into the file properties to allow this; on Windows 8 you actually have to do some registry editing: http://www.seankreps.com/windows-8-tips/windows-8-cant-run-downloaded-exe-files/
  2. If you get errors when you try to start logstash or your logstash-forwarder you’ll want to make sure your config file is valid JSON.  Try pasting it into JSONLint (http://jsonlint.com/). Also you can run logstash –configtest -f logstash.conf for it to do configuration validation for you.
  3. On the logstash-forwarder on windows, you will see “registry rotate” errors, this is normal and due to an unfixed logstash-forwarder bug (https://github.com/elastic/logstash-forwarder/pull/272).

Log Monitoring

Install up.time Log Monitoring

  1. Install the up.time agent on the logstash server using the instructions for your OS found here: http://docs.uptimesoftware.com/display/UT/Installation+and+Quick-Start+Guide (this assumes you have an up.time server already running, if not, you’ll want to do that first).
  2. Install the up.time Log File Monitor plugin from the Grid (Services -> Add Service Monitor -> Want More? Search for monitors. -> Log File Monitor) to your up.time server, if it hasn’t been already.
  3. Install the Log File Monitor agent files (http://the-grid.uptimesoftware.com/plugin/log-file-monitor.html) onto the agent on your logstash server.
  4. Set up all the log file monitors you desire on your up.time server hitting your logstash server. Add a service monitor and you will find the “Log File Monitor” option appearing under the Operating System Monitors section. You can pick directories and log file names via regex and then look for regexes appearing in them.
Add a Log Monitor
Add a Log Monitor

Log Rotation

Your logs are coming in daily chunks with the date appended to them. You can easily script a removal yourself – on Linux if you wanted to compress log files more than a day old and delete ones more than a week old, you would use

/usr/bin/find /var/log/rsync/ -mtime +7 -delete
/usr/bin/find /var/log/rsync/ -mtime +1 -exec gzip -q {} \;

On Windows, use a tool like https://gallery.technet.microsoft.com/scriptcenter/31db73b4-746c-4d33-a0aa-7a79006317e6 that has similar functionality.

Helpful Links:

If you want to install Elasticsearch and Kibana to provide search and dashboarding on these collected logs, you can! And make sure and let us know what you’d like out of up.time from a log monitoring perspective.