Ingesting Data into an OpenSearch Cluster with Logstash

Prerequisites

Complete the following tasks before proceeding with the steps described in this topic:

• Download and install Java Development Kit (JDK) version 8 or version 11 on your local machine.

• Create an OpenSearch cluster and make note of the connection information. See Creating an OpenSearch Cluster and Getting an OpenSearch Cluster's Details.

• Create and connect to a compute instance. The instance must be in the same VCN as the OpenSearch cluster. See Creating a VM Instance.

• Connect to the OpenSearch cluster from your local machine. See Task 3: Test the connection to OCI Search Service – OpenSearch endpoint

Install and Configure LogStash

1. Download the Logstash OSS with OpenSearch Output Plugin from OpenSearch Ingest Tools (2.8) and install it on your local machine. This is currently only available for Linux and MacOS based systems.

2. Extract the downloaded Logstash tarball and then navigate to Logstash directory.

3. Create a pipeline configuration file, logstash.conf, in the config subdirectory of the Logstash directory.

   Following is a pipeline configuration file example that reads the log file from your local machine and ingests the log data to an OpenSearch cluster. Replace the applicable text with the details for your cluster. Specify the cluster's API endpoint for host, see Getting an OpenSearch Cluster's Details. For user and password, specify a user that has sufficient permissions to ingest data for the OpenSearch cluster. For example, you can use the primary user account for role-based access control that you specified when you created the cluster, see Role-Based Access Control in Search with OpenSearch.

   input {
     file{
       path => "/path/to/log/file/application.log"
       start_position => "beginning"
     }
   }
    
   filter{
     #Optional and can be empty
   }
    
   output {
     opensearch {
     hosts => "<cluster API endpoint>"
     user => "<cluster user>"
     password => "<cluster password>"
     index => "<index name>"
     ssl_certificate_verification => true
     }
   }

4. Start Logstash by running the following command:

   bin/logstash -f config/logstash.conf

After Logstash starts, it continuously reads the file system for new logs using its file input plugin. Logstash then uses the OpenSearch output plugin to ingest new log data to an OpenSearch cluster.

You can verify that the log data was ingested into the OpenSearch cluster by connecting to the cluster's OpenSearch Dashboard to check for the index named what you specified for index in the pipleline configuration file. See Task 6: Connect to OpenSearch Dashboards and Quickstart guide for OpenSearch Dashboards.