Posts

Showing posts from September, 2023

What are Beats, how do they work

Image
  What are Beats? Beats is a platform for lightweight data shippers that are designed to send various types of data to Elasticsearch or Logstash for processing, indexing, and visualization. Beats is part of the Elastic Stack and is used to collect data from various sources such as logs, metrics, and network packets. Beats are easy to install, lightweight, and have a small footprint, making them ideal for various distributed systems. Beats consists of four different types of data shippers, each designed for a specific purpose: Filebeat : Filebeat is used to collect log data from various sources, including log files and the standard output of applications. Filebeat is lightweight and efficient and can be used to send data to Elasticsearch or Logstash for processing and analysis. Metricbeat : Metricbeat collects metric data from various sources, including servers, applications, and operating systems. Metricbeat is designed to collect real-time metrics and can be used to monitor s...

What is ELK and Installing ELK stack (elasticsearch, logstash, kibana) in Ubuntu

Image
   What is ELK Stack The ELK stack is a collection of three open-source tools — Elasticsearch, Logstash, and Kibana — that work together to collect, store, and analyze data. Here is how each tool works: Elasticsearch : Elasticsearch is a search and analytics engine that provides a distributed, real-time search and analytics platform. It stores data in a distributed index, which allows for fast, real-time search and analysis of data. Elasticsearch is highly scalable, and it can handle a large volume of data with ease. Logstash : Logstash is a data pipeline that collects, filters, and transforms data from different sources before sending it to Elasticsearch. It can collect data from different sources such as log files, databases, and message queues. Logstash can also perform data transformation and filtering to ensure that only the relevant data is sent to Elasticsearch. Kibana : Kibana is a data visualization tool that allows users to create dashboards, visualizations, and repo...

Sending Zeek logs to ELK using Filebeats

Image
Before sending logs we must modify local.zeek file and add the below line at the end of the file. @load policy/tuning/json-logs.zeek Use the find command to find local.zeek file. find / -name local.zeek local.zeek path Now copy the path and edit the file using nano and add the line. local.zeek Save and exit from editor, now check the configuration of zeekctl sudo zeekctl check zeekctl check Once the you get zeek scripts are ok, deploy zeekctl sudo zeekctl deploy zeekctl deploy Go to the zeek logs path directory which is /usr/local/zeek/logs/current. Check if logs are generated properly. logs Go to Kibana dashboard head over to integration section and search for zeek, in the integration page on bottom right go to “Also available in Beats”. You will be redirected to Filebeats Zeek integration page. Copy the commands from the Step1 and open new terminal window and run the commands. Which will download and extract the filebeats. Now edit the filebeat.yml file, change the paths to the zeek ...