Analecta Cyber Company Blog: Log Aggregators: Deciding Between Off the Shelf or Rolling Your Own


Log Aggregators: Deciding Between Off the Shelf or Rolling Your Own

Logging best practices Analecta LLC Log Aggregators - Deciding between off the shelf or rolling your own - Analecta Banner The NIST Cybersecurity Framework calls on organizations to monitor their information systems to identify unauthorized use, unauthorized local, remote and network connections, and indicators of potential attacks. To do this, NIST suggests collecting and correlating event data from multiple sources and sensors.

Logging gives an excellent picture of your security posture by allowing you to:
  • Capture trends across the network
  • Understand how your systems and networks are performing
  • Quickly identify malicious events
  • Be notified if a device or website becomes unavailable
  • Troubleshoot and find root causes of some issues

Logging system behavior and analysis

There are several ways to gather all the logs into one central repository. You can build your own system to aggregate log files from all of your devices, but the real question is do you really want to? Even with a small number of devices, you may quickly become overwhelmed parsing data from each source, instead of analyzing data to find potential threats. We previously discussed using logs for managing remote access. Here we discuss the pros and cons of collecting data with a home-built log aggregator versus purchasing a log aggregator package.

Logging best practices Analecta LLC
Logging gives information about your internal network so you can quickly identify malicious events. However, anomalies cannot be detected unless logs are being audited. It is far more effective to find trends across more complete datasets, rather than examining individual logs.

Log Aggregation 101

In order to aggregate logs, you need to configure your systems to record logs. Here are a few recommendations to make the process of collecting and parsing log information more manageable:
  • Do include a timestamp
  • Do format in JavaScript Object Notation (JSON)
  • Do turn on logging
  • Do write messages in a human-readable form
  • Do log all application errors
  • Maybe log warnings (with the understanding that this could potentially fill all your logs)
  • Don’t log insignificant events
  • Don’t log informational data in production
  • Don’t log anything a human can’t read or react to

Now that the systems are logging properly, how do we get the logs from one system to the centralized location? The first option is to copy the log files from the source location to the desired system. This is not true aggregation and cannot be performed in real-time, so the benefit to this strategy is minimal. Even the smallest networks can generate too much information to be analyzed manually. 

Another strategy would be to use a daemon like syslog/rsyslog/syslog-ng to automate the logging processes. Most likely, one of these programs is probably already on your system - you just need to configure it. A challenge is that you may spend quite a bit of time configuring the daemon instead of other tasks.

Log aggregation tools

It is usually more effective to have a program do the log aggregation and assist with the analysis. There are a number of different-priced options available. A few off-the-shelf tools in this space include the following:

Elastic stack logo
Elastic Stack: This is the newest evolution of the ELK Stack (short for Elasticsearch, Logstash and Kibana), the most popular open-source log aggregation tool on the market.
apache flume logo
Apache Flume: This tool can efficiently collect, aggregate and move huge amounts of information. It can also be used to move data from your different applications and stores them on the Hadoop Distributed File System (HDFS.
Splunk Logo
Splunk: Splunk was founded in 2003. It gives you easy log aggregation as well as a number of other features for viewing and analyzing logs and creating reports and searches.
graylog logo
Graylog2: Graylog2 stores your events on Elasticsearch or MongoDB. It presents a user interface to analyze or search your logs.

Cloud or “as a service” tools
Instead of relying on locally hosted software to manage your log aggregation, you can also turn to the cloud or find an “as a service” provider. A few of the cloud-based tools include:

Amazon web services logo
AWS: Amazon Web Services offers a centralized logging solution for collecting, analyzing and displaying logs on AWS across multiple accounts and AWS Regions - a custom Elastic Stack through AWS.
Solarwinds papertrail
PaperTrail: Solarwinds’ PaperTrail helps you manage all your logs no matter which server they come from. It counts GitHub and InstaCart as customers.
Rapid7 logintries logo
LogEntries: Rapid7’s LogEntries focuses more on log aggregation and analysis. You can search, monitor or visualize your messages using this service.
Solarwinds loggly logo
Loggly: Solarwinds’ Loggly is a cloud-based service that allows you to easily access and analyze all your logs in real time.

Keep moving forward

Still unsure on where to start? We’ll be happy to help you analyze your networks and situation and go over popular logging frameworks, full-text searching, structured logging, monitoring and alerting. We’ll help you find the solution that best fits your organization and security priorities.  Send us an email at or visit our Cybersecurity website.

Analecta Cyber is a Maryland-based cybersecurity firm providing cyber risk assessments for small and medium sized businesses. Analecta is a trusted partner to help companies achieve their cybersecurity objectives.

No comments :

Post a Comment