LOCATION

36/7, West Rajiv Nagar
Gurgaon, India
  +91-8431380442
  enquiry@hybridskill.com


91 Springboard
Co-working place 3rd Floor, 175 & 176,
Bannerghatta Main Rd, Dollars Colony
Bengaluru, Karnataka 560076
  +91-8431380442
  enquiry@hybridskill.com

All you need to know about logging in 2020



My friend a system administrator recently switched to an organization that grew fast.  Last week his manager called a team meeting at first hour of the morning and informed him that from last one week multiple servers are going through an extreme load in midnight hour. He was assigned the job to find out what is happening. While we were discussing as to what may be the cause. He shared that they do not have any centralized logging server. A centralized logging server is a major component of operational and cyber security infrastructure for IT organizations. It can store hardware and software logs of entire infrastructure at one place. Many small organization considers logging servers as less important than monitoring servers and firewalls. In the situation of the outages, the logging server is very useful to find out useful piece of information for security breach incidents. A dedicated centralized logging is crucial part for PCI compliance. There are a large number of logging servers available. Choosing the right one may overwhelm you if you are going to set up your first logging server. This article will help in understanding a wide array of Linux system monitor web interface based logging solutions and to choose which suits your infrastructure needs. I will discuss about strategy behind choosing the system event log viewer and will provide a list of most popular web based log viewer.

System logs Vs Application logs

You have to be clear about why server log analysis is important for you.  An incident logging system can be fine-tuned for application customized server monitoring dashboard. It can be helpful to find application bugs and security breaches with detailed logs. Most operating systems come within an operating system logging software like Syslog. These logs can be helpful in root cause analysis and identifying hardware issues like faulty storage disks or high latency network connections. System log data may be useful for PCI compliance levels. Advanced application logging servers can take actions on the basis of system log while in cyber-attack. Application log data can be helpful in testing application performance and to improve the design and implementation of application.

Dedicated logging server VS Monitored logging server

An incident logging system and performance monitoring dashboard works very closely with each other. Often both overlaps, there are many solutions available that fill the gap between monitoring and logging. Popular solutions like Goaccess, Nagios, ELK stack and Kibana which integrates both logging and monitoring of IT and cloud infrastructure as well as IOT monitoring. Analytics dashboard tools needs huge storage and are compute intensive. Small logging tools like Syslog, Event viewer, kiwi syslog viewer are comparatively lighter on system resources.

Logging server vs SIEM

Security Information and Event Management (SIEM) provides greater insight into security, network and system data combined. It can analyze real-time data and helps in laying out long term security and operations policy. SIEM is complex software as it is an added layer of time and cost for teams who do not use SIEM in their stack in the past and are new to it. It requires a whole dedicated team for monitoring and alert management for SIEM. Logging solutions are less expensive and needs manual log analysis while generating an incident report. SIEM Implements the audit policy, security policy and operational policy. It is a hybrid solution which can help you in Linux troubleshooting, root cause analysis in case of security breaches and in designing a proactive policy to avoid future events.

Storage

Storage is a big concern with logging activity. Networking, computational, IoT and security devices produce tons of log data every hour. If there is no proper policy about the retention of this data then this data can fill the storage very fast. A proper storage policy helps to manage storage resources for a longer period of time. Often it is accompanied by audit policy for compliance. Logging servers can be configured to store data on SAN or NAS devices. For less sensitive logs cloud is a cheaper and advanced storage point.  Cloud companies have also come up with monitoring and logging solutions like Cloud watch and Azure monitor which can be integrated with on-prem resources as well.Moreover, the Selection of the right logging tool depends upon the existing technology and technical expertise of the team. In most cases, it is advisable to choose a new technology that requires the least learning curve and increase efficiency.

Here is a list of PROS and CONS of 35 most popular logging servers

1.) Logentries

Pros:

Easy for analysis of logs

Rich GUI and monitor graphics

Supports REST API

Can be accessed over a browser

Can log data from Routers, Switches, firewalls, and distributed machines

Supports Application logging

Easily can be integrated with cloud, like Heroku, AWS and many more

Cons:

Expensive

Less retention period

Lack of privacy measures for team collaboration

2.) GoAccess

Pros:

FAST and responsive

Lightweight

Backed by a strong open source community

Can export data in CSV

Collects data in JSON format

Cons:

Only terminal-based no GUI

Lack of enterprise support

Prone to security vulnerabilitiy

3.) Logz.io

Pros:

Can be integrated with cloud platforms AWS, Azure, etc

Vibrant graphics and detailed system view

Can be integrated with ELK stack and Graphana dashboard

Provide security alerts

Cons:

Expensive

Compute intensive, requires huge systems

Hard to maintain historic data

Need a big storage

4.) Graylog

Pros:

Rich dashboard

Can process a large amount of data

Collects operating system and application data

Can collect data from a wide array of devices

Strong support

Cons:

Expensive

Less responsive dashboard

Agent-based

5.) Splunk

Pros:

Powerful dashboard

Supports application logging

Presents analytical data

Supports application data

Can collect data from a wide range of sources

Cons:

Expensive than similar open-source alternatives

Inconsistent dashboard performance

Very frequent updates

6.) Logmatic

Pros:

Perform better than its counterparts

Simple and intuitive GUI dashboard

Supports cloud platform AWS, Azure, Heroku cloud

Scalable design can be scaled to 100’s of host

Cons:

Not very rich dashboard

Supports less number of host

Costly

Lack of support

Less popular than other logging servers

7.) Logstash

Pros:

Rich set of plugins available to choose from

Most popular logging solutions

Rich dashboard

Part of ELK stack

Real-time search

Supports REST API

Multi-language support

Strong support

Cons:

Resource consuming

Less responsive dashboard

Too much less useful options

8.) Sumo Logic

Pros:

Supports application log data

Supports a wide range of devices

Can send alerts over slack channel, mail and phone

Support cloud-based infrastructure

Cons:

Less responsive support

Does not support on-premise data

No community support

9.) Solarwinds Papertrail

Pros:

Intuitive UI

Easy to setup

Supports application data

Can send alerts on Phone, email

Supports major cloud providers

Cons:

Fewer features

No analytics, require manual analysis of logs

10.) Fluentd

Pros:

Open-source

Supports monitoring

Can monitor systems and containers as well

Lightweight requires fewer resources

Supports Plugin

In streaming data processing

Cons:

Security bugs

Hard to install

There is no message delivery acknowledgment

11.) Syslog-ng

Pros:

Lightweight logger can be integrated with Splunk and other monitoring tools

Free And open source

Very rich filtering capability

Highly portable, can be used with almost all *ix flavors

Cons:

Does not provide log analytics

Does not supports windows

No support

12.) Rsyslog

Pros:

Very fast in comparison with other centralized logging servers.

Supports multithreading

Provide application logs of major applications like SQL Server, Postgres, etc.

Open source and free. Comes default with many UNIX systems

Supports wide range network protocols

Cons:

No centralized dashboards

Does not provide automatic analysis of logs

Does not support windows

No support available

Terminal-based only

13.) LOGalyze

Pros:

Free and open source

Have SIEM capability

Rich GUI for real-time analysis of logs and security events

Supports application logging

Can be integrated with Syslog and Rsyslog for log collections

Support for REST API clients

Cons:

Security vulnerabilities

Supports only *ix based systems

No support plans

14.) jKool

Pros:

Vibrant centralized dashboard for analysis of logs

Collects application data

Lightweight and fast in comparison of other monitoring and logging servers

Support in the streaming data processing

You can filter live real-time data to get an insight into metrics.

Cons:

Not actively maintained

Lack of support

15.) Flume

Pros:

Free and open source

Backed by active open source community

Supports high volume big data

Can collect data over the network

Cons:

NO centralized dashboard

Does not support windows

Limited log analysis available

16.) Cloudlytics

Pros:

Support data collection from the cloud

Network log analysis

Centralized dashboard for managing logs

Provide financial insight into cloud

Cons:

Supports only AWS, no support of other public clouds

NO free plans available for testing

Does not support on-prem infrastructure

17.) Scalyr

Pros:

Customizable dashboard for system logging, application logging, and security logs

Real-time analysis of logs

Fast performance

Free tier available

Can export log reports over REST APIs

Can monitor Kubernetes cluster

Cons:

Limited control over data

Less secure

Expensive for high volume data requirements

18.) Octopussy

Pros:

Open source and free

Can be integrated with major monitoring tools like Zabbix, Nagios

Features an inbuilt tool for creating incident reports

Provide application log capability of applications like SQL, PostgreSQL

Available for both Windows and Linux platforms

Can be used to collect network logs from network devices like routers and switches

Cons:

Not actively developed

No Support available

Security bugs reported

19.) LOGStorm

Pros:

Supports a wide range of data sources

Provides insight into security breaches and vulnerabilities

Helps in the formulation of rich incident reports including monitoring, security, and machine log data

Provides application log metrics support

Cons:

Expensive for a large amount of data

No free tail available

Cannot send alerts on mobile

20.) NXLog

Pros:

Open-source

Centralized dashboard for log analysis

Can collect logs from on premise, cloud data source

Provides application log metrics for performance analysis of running applications

Free tier available with limited features

Cons:

Security vulnerabilities

Does not provide monitoring data

Supports Limited data source

Does not support real-time data processing

21.) Sentinel Log Manager

Pros:

Can be used to collect log data from distributed systems and IoT devices

Features one-click report creation

Can predict storage requirements and cost analysis of hardware requirements

Can send alerts over phone/email or slack channel

Rich, easy and intuitive GUI for log analysis

Free trial available

Cons:

No support for application log

Limited data source available

No support for cloud data collection

22.) XpoLog

Pros:

Provide insight into system logs as well as security

AI-enabled issue detection

Proactively scans systems logs for potential issues

Centralized dashboard to audit the log analysis

Available for SaaS, PaaS and public cloud

Single window for security, system log, and network logs

Free trial available

Cons:

Expensive license only monthly based payment

Less alert endpoint device supported

Not able to integrate with popular services like AWS redshift

23.) Netsurion EventTracker

Pros:

Provide Log monitoring, Threat monitoring, and global SOC center

Simple and powerful Centralized Dashboard

Intelligent system that can predict future threats by analysis history

One point solution for system forensics and threat mitigation

One-click report creation

Network monitoring and vulnerability assessment

Supports on-premise and public cloud platform

Cons:

Very expensive, one-time payment option only

Does not offer application logging and application security

Cheaper alternatives available

Very complex to learn, no training available

24.) LogRhythm

Pros:

Provide security and log monitoring

A proactive alert mechanism for threat reporting and threat detection

Centralized GUI based dashboard, very easy to use

Highly scalable for a large volume of data sources

Perform Log analysis, log parsing and report creation easily

Cons:

No support for log scripting in log analysis

No active support available

The dashboard cannot be customized

Less efficient log analysis feature

The agent is too memory intensive

25.) McAfee Enterprise log manager logging

Pros:

Intelligent log collection for log operational troubleshooting and security assessment

Can store log on NAS/SAN devices

Support log collection from an on-prem data source or cloud platforms

Can be used with or without MacAfee integrated security manager service

Rich GUI based centralized dashboard for effective monitoring

Cons:

Support integration only with MacAfee devices

The flash-based user interface, prone to security vulnerabilities

Complex GUI with lots of similar features

Cannot be integrated with cloud-based monitoring tools

26.) Cryptology

Pros:

Customizable Dashboard for different data sources

Provides System logs and network logs

Can be integrated with Syslog

Simple Centralized dashboard for real-time analysis

Support archiving of logs and creation of the incident report

Can channelize critical log data for easy and fast log analysis

Cons:

Not actively maintained

Less documentation, support available

No free tier

27.) Humio

Pros:

GUI based dashboard for system log and network log analysis

Provide support for application logging

Automatic and manual log analysis feature

Can monitor on-premise, and cloud data sources

Can be integrated with other metrics based on dashboards

New dashboard features can be added by using REST API

Cons:

Expensive license

Less efficient dashboard, very hard to write new scripts for analysis

No free tier license available, only quota-based pricing model

 

28.) Timber

Pros:

Provide real-time insight into raw data

Contains minute details of each log

Collect lots of different log levels

Supports automatic and manual log analysis

Provide an audit trail for a single user

Large retention period

Searching and filtering log capability

Cons:

Inconsistent documentation

No free trial available

Log assembly and organization is complex and less easy to work with

Less efficient support

29.) Coralogix

Pros:

Provide monitoring of system log data, metrics data, and network data

Can be integrated with popular dashboard platforms like ELK stack, AWS cloud watch

Jason based log filtering and parsing

Free trial available with limited data volume

Cons:

Cluttering GUI

No support for auditing

No support for automatic report creating

 

30.) SentinelTrails

Pros:

Provide audit for individual users

Audit reports can be secured by using block chain technology

New modules can be integrated by using REST API

Collect logs from network devices like router, switches, etc.

Provide support for regulatory compliance of HIPAA, PCI-DSS, etc.

Cons:

Small community around

Less technical documentation available

Parsing and filtering of live data is not supported

31.) Kiwi Syslog Server

Pros:

Collect data from network devices, SNMP events, and windows events

Centralized GUI based dashboard for log analysis

Capable of sending alerts on email and other applications

Implement auditing compliance for PCI DSS, HIPAA, etc.

A large and active user community

Cons:

Supports only windows

No support for parsing and filtering of live data

Less feature in the dashboard in comparison of other logging servers

32.) Filebeat

Pros:

Popular as an extension for log collection in ELK stack

Lightweight agent based can export logs to elk and Kibana dashboard

Can be integrated with many open-source matrix-based dashboards

Capable of collecting application logs of Nginx, Apache, SQL, etc.

Can be used to collect log data of network devices

Cons:

ELK stack is very memory consuming, can’t run on low memory CPU

Can be hard to configure especially when you want to change the destination of the application after deployment

Does not support parsing and filtering of live data

Very hard to troubleshoot security incidents

 

33.) Snare agent

Pros:

A lightweight log collection agent

Can be integrated with other REST-based log analyzer dashboard

Supports the collection of application logs like SQL etc.

Capable of providing compliance-ready data

Collect data from Linux, Windows, Mac, Solaris and network devices also.

Collect data from the firewall to troubleshoot security incidents

Can be integrated with compliance dashboards like ELK stack and Kibana

Free trial available

Cons:

Does not have own server for log analysis

No GUI support

Hard to troubleshoot installation and configuration issues

No active user community

34.) Logsniffer

Pros:

Free and open-source log collector, parsing and analyzer

Rich UI for real-time log analysis

No memory restrictions of data collection

Highly scalable can be integrated

Capable to send alerts on multiple channels including email and phone alerts

Support on-premise data center 

Cons:

Project is abandoned no longer actively maintained

Slow and memory-intensive

No support of cloud infrastructure

35.) Scribe

Pros:

Highly scalable, Designed by Facebook to collect real-time log data from thousands of servers

Open source and free Apache Linux

No memory threshold can collect petabytes of data

Useful for very large data centers

Works on on-premised data center

Cons:

Project abandoned, no longer maintained by Facebook

No support for collection of network device

No GUI support



Article by Umesh Upadhyay


Leave a Reply