Forwarding logs to Logstash with Logstash Collector

You can collect logs from your Open Liberty server and display them in a dashboard for analysis and troubleshooting purposes. The Logstash Collector feature collects logs from Open Liberty and sends them to a remote Logstash server, where they can be managed and visualized by products such as Elasticsearch and Kibana.

Logstash is an open source log management tool that prepares log data for analysis. Logstash can process and transform data, and then send it to various outputs, such as Elasticsearch, Kafka, PagerDuty, or an email server.

When to use the Logstash Collector feature

If your Open Liberty server is deployed in a managed environment, such as a Kubernetes cluster, your logs are typically aggregated by agents in that environment. In this case, you do not need to use the Logstash Collector feature.

In cases where your server logs are not managed by an external agent, you might be able to configure Filebeat to forward your logs to Logstash. If you can install Filebeat on the same host as your Open Liberty server, you can configure Open Liberty to write logs in JSON format and Filebeat can forward the logs. For more information, see Analyzing JSON logs with the Elastic stack.

If you can’t install Filebeat on the same host as your Open Liberty server, you can use the Open Liberty Logstash collector feature to forward your logs to Logstash. This feature collects log data at run time and sends it to a Logstash server.

Although this feature can send log data for analysis by any of the available output plug-ins from Logstash, many users choose to use Logstash with Elasticsearch and Kibana. These three tools are widely used together to collect, manage, and visualize log data. For more information, see the Elastic website.

The Logstash Collector feature was tested with the following products:

  • Logstash V5.3.x, Elasticsearch V5.3.x, and Kibana V5.3.x

  • Logstash V6.4.x, Elasticsearch V6.4.x, and Kibana V6.4.x

  • Logstash V7.x, Elasticsearch V7.x, and Kibana V7.x

Before you begin

Set up Logstash, Elasticsearch, and Kibana by following the instructions from Elastic.

Configuring your server to forward logs with the Logstash Collector feature

The following procedure demonstrates how to collect your Open Liberty log data with the Logstash Collector feature. You can then send it to a Logstash server, store it in Elasticsearch, and view it in a Kibana dashboard.

  1. Create or acquire certificate and key pair files to enable SSL or TLS connections for Logstash.
    The following example command for OpenSSL generates a certificate and key pair. Customize the number of days that the keys are valid as needed:

    openssl req -x509 -newkey rsa:2048 -keyout logstash.key -out logstash.crt -days 365 -nodes

  2. Download a sample Logstash configuration file and an index template file from the Sample Logstash Collector dashboards for Liberty repository.

    1. Download the liberty_logstash.conf Logstash configuration file and the liberty_logstash_template_ibm.json index template file from the directory that corresponds to your Elastic version.

    2. Complete the following steps in the liberty_logstash.conf file:

      • Customize the Beats ssl_certificate and ssl_key paths with the certificate and key pair files that you generated in step 1.

      • Customize the Elasticsearch_host_name:port_number Elasticsearch hosts value with the hostname and port number from your Elasticsearch configuration.

      • Customize the Elasticsearch template path with the path to your liberty_logstash_template_ibm.json index template file.

  3. Complete the following steps for each Open Liberty server that you want to collect events from:

    1. Acquire or create a keystore for the Open Liberty server. To create a self-signed certificate, run the securityUtility command from the wlp_install_dir/bin folder, as shown in the following example. Customize the server name, password, and subject as needed:

      d:\wlp\bin\securityUtility createSSLCertificate --server=server1 --password="Liberty" --subject=CN=myHostname,OU=server1,O=ibm,C=us

      The key.p12 default keystore file is created under the wlp_install_dir/usr/servers/server1/resources/security directory. The output from this command is similar to the following example:

      Creating keystore wlp_install_dir/usr/servers/server1/resources/security/key.p12

Created SSL certificate for server server1. The certificate is created with CN=myHostname,OU=server1,O=ibm,C=us as the SubjectDN.

Add the following lines to the server.xml to enable SSL:

    <featureManager>
        <feature>ssl-1.0</feature>
    </featureManager>
    <keyStore id="defaultKeyStore" password="{xor}EzY9Oi0rJg==" />

      Add the server configuration snippet from the command output to your server.xml file.

    2. Import the logstash.crt file that you created in step 1 into the key.p12 file in your server by running the Java developer kit keytool command. In the following example, customize the wlp_install_dir directory and server name as needed. When prompted for a password, use the certificate password that you specified in step 3a:

      keytool -importcert -noprompt -alias logstash -file logstash.crt -keystore wlp_install_dir/usr/servers/server1/resources/security/key.p12 -storepass Liberty

    3. Use the following command to install the Logstash Collector feature if it is not already installed on your server:

      d:\wlp\bin\featureUtility installFeature logstashCollector-1.0

    4. Configure the Logstash Collector feature in your server.xml file.

    5. Enable HTTP access logging.

  4. Start Elasticsearch, Logstash, and Kibana. See the Elastic website for instructions.

  5. Start the Open Liberty server and generate some events.

You configured your server to use the Logstash Collector feature to send you logs to a remote Logstash server. You can now use the Liberty sample Kibana dashboards to create visualizations of your logs.

Setting up a Kibana dashboard to visualize your logs

After you configure your server to collect and forward logs with the Logstash Collector feature, complete the following steps to view your log events in a Kibana dashboard.

  1. Open Kibana in a browser and create an index.

    • For Kibana 7, 6, or 5.6, click Management > Index Patterns.

      1. Enter logstash-* as the Index Pattern.

      2. Click Advanced Options, and enter logstash-* as the Index Pattern ID.

      3. Select datetime as the Time filter field name. Click Create.

    • For Kibana 5.0 - 5.5:

      Click Management > Index Patterns, and select datetime as the Time filter field name. Click Create.

  2. In the Sample Logstash Collector dashboards for Liberty repository, download a sample dashboard from the directory that corresponds to your version of Elastic.

  3. Import the dashboard into Kibana.

    Click Management > Saved Object > Import, and select a dashboard that you downloaded in step 2.

  4. View the dashboard.

    Click Dashboard > Open, and select the dashboard that you imported in step 3.

You configured your Open Liberty servers to send events to your Logstash server with the Logstash Collector feature. You can now view visualizations of your log events in the Liberty dashboards by using Kibana.

See also