Skip to content

Log monitoring - EFK Stack

EFK - Elasticsearch, Fluentd, and Kibana

  • Elasticsearch is a distributed and scalable search engine, used commonly to sift through large volumes of log data. Its' primary purpose is to store and retrieve logs from fluentd.

  • Fluentd is a log shipper. It is an open-source log collection agent which supports multiple data sources and output formats. It can also forward logs to elasticsearch. Fluentd is a unifying layer between systems that generate and store log data.

  • Kibana is a UI tool for querying, data visualization and dashboards. It is a query engine which allows exploring log data through a web interface, building visualizations for events logs, and is query-specific to filter information for detecting issues. Virtually any type of dashboard can be built using Kibana. Kibana Query Language (KQL) is used for querying elasticsearch data. Here Kibana is used to query index data in elasticsearch.

NOTE: The data retention period of Elasticsearch is 7 days. (This means logs can be accessed only for the past 7 days, including the current day. A curator-cronjob will be run daily to check and delete logs which are older than 7 days).

How to Access Kibana Dashboard

1. Browse the Kibana Dashboard of IFS Remote Solution [Log in to Kibana]

2. Authentication

  • The user must enter the username and password to log in to dashboards. kibana Login

  • The user can extract the username and password from the IFS-Remote file location in their windows machine: ifsroot > config > secrets > kibana_creds file.

  • The initial screen displayed upon user login. View after the login

How to Discover Kibana Logs

  • Navigate to Discover from the main menu. kibana discover select from navbar

  • View Records. kibana discover view recods

How to Query Container Logs using KQL

After navigating to Kibana Discover, the user will be able to use KQL to efficiently discover logs.

kibana KQL

The Kibana Query can be added into the search filter text box. (e.g. kubernetes.container_name: "ifsapp-odata") The desired time period can also be selected.

How to View Dashboards

  • The dashboards will get automatically imported, and there will be no requirement to import them manually.

  • Once logged in, the user will be automatically redirected to the dashboards.

  • Use any of the following links to navigate. Kibana dashboards default intro

​

  • Cluster-level Summary dashboard view Kibana dashboards Cluster level summary

​

  • Container-level Summary dashboard view Kibana dashboards Container level summary