Connecting to Remote {es} Clusters

Siren Federate provides the capability to query data from an {es} remote cluster through the Remote Clusters Module and the Federate Connector APIs.

Through Federate Connector APIs, datasources and virtual indices can be managed using the Federate REST API or the web user interface available from Siren Investigate.

NOTE: This connector, unlike the JDBC connector, supports wildcard index patterns.

Configuring the Remote Cluster

To send queries from a cluster (let’s call it the coordinator) to remote {es} clusters, the remote clusters must be configured as described in Configuring remote clusters.

The Siren Federate plugin has to be installed on the remote clusters.

This example shows how to set up the remote {es} clusters:

curl -X PUT http://localhost:9200/_cluster/settings -H 'Content-type: application/json' -d '
{
    "persistent": {
        "cluster": {
            "remote": {
                "remotefederate": {
                    "seeds": [
                        "127.0.0.1:9330"
                    ]
                }
            }
        }
    }
}
'

Configuring the Datasource

A datasource must first be defined as an alias to the remote cluster. Datasources are created in the coordinator cluster using the Federate REST API.

curl -X PUT http://localhost:9200/_siren/connector/datasource/remotefederateds -H 'Content-type: application/json' -d '
  {
    "elastic": {
      "alias": "remotefederate"
    }
  }
  '

Configuring the Virtual Index

Let’s assume our remote cluster remotefederate has indices called logs-2019.01, logs-2019.02, …, logs-2019.12.

Using a Wildcard Index Pattern

Let’s define a virtual index on the coordinator cluster that matches the wildcard index pattern logs-* using the Federate Virtual Index API:

curl -X PUT http://localhost:9200/_siren/connector/index/logsvi -H 'Content-type: application/json' -d '
{
  "datasource": "remotefederateds",
  "resource": "logs-*",
  "key": "_id"
}
'

Assuming the coordinator cluster has an index called machines which contains information on IP addresses on machines of interest, and that we would like to find out about the logs associated to these machines, you can execute the following Federate JOIN query to do so:

curl -X GET http://localhost:9200/siren/logsvi/_search -H 'Content-Type: application/json' -d '
{
    "query": {
        "join": {
            "indices": [
                "machines"
            ],
            "on": [
                "logs_ip_hash",
                "machines_ip_hash"
            ],
            "request": {
                "query": {
                    "match_all": {

                    }
                }
            }
        }
    }
}
'

logs_ip_hash is the IP field in the index logsvi and machines_ip_hash is the IP field in the index machines.

Known limitations

In order to take advantage of Federate with a remote cluster, at the moment a coordinator Federate cluster must run 6.8.6-10.3.4 up and the remote cluster must run Federate version from 6.5.4-10.2.0 up.

{sg} Compatibility

The connector is compatible with {sg}. One can define {sg} users with roles to secure the remote clusters and the coordinator cluster.

Each cluster must have the same user that has permissions to access the cluster datasources, indices and virtual indices in order to properly execute Federate search requests on remote clusters.

Using curl and a {sg} user called admin, the command would start like this:

curl -k -uadmin:password -X PUT https://localhost:9200/<some API request> ...

More information is available on the {sg} website.