Analyzing JSON logs with the Elastic Stack

You can collect logs from your Open Liberty server and display them in a dashboard for analysis and troubleshooting purposes. Use JSON logging to emit logs from Open Liberty and send them to the Elastic Stack, where they can be managed and visualized.

The Elastic Stack is a log analysis platform that consists of Elasticsearch, Logstash, Kibana, and Beats. You can use these products to collect, ship, search, and visualize your Open Liberty log data.

If your Open Liberty server is deployed in a managed environment, such as a Kubernetes cluster, your logs are typically aggregated by agents in that environment. In cases where your server logs are not managed by an external agent, you might be able to configure Filebeat to forward your logs to Logstash. If you can install Filebeat on the same host as your server, you can configure Open Liberty to write logs in JSON format and Filebeat can forward the logs.

The following procedure assumes that your Open Liberty server is in an environment where Filebeat has access to the server logs. If you can’t install Filebeat on the same host as your server, you can use the Logstash collector feature to forward your logs to Logstash for analysis by the Elastic Stack. This feature internally collects log data at run time and sends it directly to a Logstash server. For more information, see Forwarding logs to Logstash with Logstash Collector.

Before you begin

Configuring Elasticsearch, Logstash, and Filebeat for your Open Liberty server

Complete the following steps to configure Elasticsearch, Logstash, and Filebeat to work with your Open Liberty logs.

  1. Download the liberty_logstash.conf sample Logstash configuration file from the directory that corresponds to your version of the Elastic stack in the Sample dashboards for Liberty repository.

  2. In the Logstash configuration file, customize the port:port_number Beats port and the Elasticsearch_host_name:port_number Elasticsearch host with your port number and Elasticsearch host value.

  3. Download the sample Filebeat configuration file for your Filebeat version from the Sample dashboards for Liberty repository.

    • For Filebeat 5.x, use the liberty_filebeat5.yml file.

    • For Filebeat 6.x, use the liberty_filebeat6.yml file.

    • For Filebeat 7.x, use the liberty_filebeat7.yml file.

  4. In the Filebeat configuration file, change the path of the log to the location of the messages.log file.

You can now use Elasticsearch, Logstash, and Filebeat to collect, ship, and search your logs.

Setting up a Kibana dashboard to visualize your logs

  1. Start the Open Liberty server.

  2. Start Elasticsearch, Logstash, Kibana, and Filebeat.

    See the Elastic website for instructions.

  3. Open Kibana in a browser and create an index.

    Click Management > Index Patterns.

    • For Kibana 7, 6, and 5.6, complete the following steps:

      1. Enter logstash-* as the Index Pattern.

      2. Click Advanced Options, and enter logstash-* as the Index Pattern ID.

      3. Select ibm_datetime as the Time filter field name. Click Create.

    • For Kibana 5.0 - 5.5, select ibm_datetime as the Time filter field name. Click Create.

  4. In the Sample dashboards for Liberty repository, download one or more of the sample dashboards from the directory that corresponds to your version of Kibana.

    • For Kibana 5.x, download JSON files that start with kibana5.

    • For Kibana 6.x, download JSON files that start with kibana6.

    • For Kibana 7.x, download JSON files that start with kibana7.

  5. Import the dashboard into Kibana.

    Click Management > Saved Object > Import, and select a dashboard that you downloaded in step 4.

  6. View the dashboard.

    Click Dashboard > Open, and select the dashboard that you imported in step 5.

Results

You can now send your Open Liberty logs to the Elastic Stack and analyze them by using the sample Kibana dashboards.

See also