Running Elasticsearch 8.0 with Testcontainers

Feb 17, 2022

6 minutes read

TLDR; With the release of Elasticsearch 8 and its new secure-by-default approach running tests with testcontainers is a little different and will be covered here.

Note: I will try to keep this post updated as I suppose the way of running tests will change over the next month and it will become easier, so please make sure to check out the most recent changes in testcontainers. I used 1.16.3 at the time of writing this article.

What is secure by default in Elastic Stack 8 and above?

Running with TLS and user authentication enabled by default poses some additional challenges. If no cert is specified a self signed cert will be generated. This requires application to be aware of that certificate in order to not throw errors when connecting to Elasticsearch.

The same is true when a password gets generated on startup. Both need to be shared with the application, before a connection can be established successfully.

This has not only to be solved for the applications, but also for every component within the stack.

On top of that, this also has to be solved, when running Elasticsearch within a docker container, be it for test or production purposes.

Some problems could be ‘solved’ with a false sense of security, by using fixed credentials or the same certificate, but this would not add any security, if other users can guess certificates and passwords. Instead, each new deployment creates new certs, passwords and encryption keys.

With this knowledge, we have to run our testcontainers tests slightly different.

Setting up Testcontainers

Just to state the obvious, testcontainers is a java library to support running docker containers as part of your tests making it ridiculously easy to run services like Postgres, Redis or Elasticsearch in end-to-end tests. I am a big fan and have written earlier about Using Testcontainers To Test Elasticsearch Plugins.

In the best case, you can start an Elasticsearch container without any further setup and run the tests and be done with it. On the other hand you should also try to resemble the production setup as close as possible. This likely includes encryption and authorization.

Container initialization

private static final String IMAGE_NAME = 
        "docker.elastic.co/elasticsearch/elasticsearch:8.0.0";

private static final ElasticsearchContainer container = 
        new ElasticsearchContainer(IMAGE_NAME)
                .withExposedPorts(9200)
                .withPassword("s3cret");

This configures an Elasticsearch container for the 8.0.0 version and sets a password. This prevents setting a randomized password on startup which needs to be extracted from the logs after startup. This means via user elastic and password s3cret you have the required credentials to access Elasticsearch as a super-user and either run the tests using that user or create a user with only the required credentials to run your tests.

There is one specialty with using withPassword() - this automatically configures xpack.security.enabled to make sure authentication is enabled. Setting this however disabled TLS and leaves us with plain text HTTP. While this might work for your tests, we want to have full blown TLS and thus need to remove this setting. This can be done via

container.getEnvMap().remove("xpack.security.enabled");

A custom wait strategy

A wait strategy in Testcontainers defines what should be executed in order to check if the container is up and running. The current strategy connects the HTTP port of Elasticsearch to figure out if the service is up and running.

Unfortunately this will fail, as the wait strategy is not aware of the self signed certificate of the Elasticsearch instance and thus making the TLS connection will throw an exception. We could change the wait strategy to accept all TLS certificates, but this would also apply to any other connection made using HttpsURLConnection and thus is not feasible. Trusting all SSL certs is definitely not the way to go.

There is another common strategy to decide when a service is up in Testcontainers: The LogMessageWaitStrategy waits for a certain log message to appear if the service has started. Elasticsearch emits a log line that only consists of started at some point, that can be used for this - so let’s try that instead and get rid of the certificate trust issues.

container.setWaitStrategy(
        new LogMessageWaitStrategy().withRegEx(".*\"message\":\"started\".*"));

Using the correct cert in the Elasticsearch Java Client

Now that we can figure out that the Elasticsearch node has started, we can go on and extract the self signed CA cert. This can be done with a docker cp command manually

docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .

In Testcontainers it is even easier as there is already a copyFileFromContainer method ready to be run.

byte[] certAsBytes = container.copyFileFromContainer(
                  "/usr/share/elasticsearch/config/certs/http_ca.crt",
                  InputStream::readAllBytes);

With the cert ready, we can create our Elasticsearch client

final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, 
        new UsernamePasswordCredentials("elastic", "s3cret"));
final RestClientBuilder builder = RestClient.builder(host);

builder.setHttpClientConfigCallback(clientBuilder -> {
    clientBuilder.setSSLContext(SslUtils.createContextFromCaCert(certAsBytes));
    clientBuilder.setDefaultCredentialsProvider(credentialsProvider);
    return clientBuilder;
});

RestClient restClient = builder.build();
ElasticsearchTransport transport = new RestClientTransport(restClient, 
        new JacksonJsonpMapper(new ObjectMapper()));
ElasticsearchClient client = new ElasticsearchClient(transport);
ElasticsearchAsyncClient asyncClient = new ElasticsearchAsyncClient(transport);

The snippet above configures basic auth for every request as well as setting an SSL context. The SslUtils.createContextFromCaCert() method looks like this:

public static SSLContext createContextFromCaCert(byte[] certAsBytes) {
  try {
    CertificateFactory factory = CertificateFactory.getInstance("X.509");
    Certificate trustedCa = factory.generateCertificate(
            new ByteArrayInputStream(certAsBytes)
    );
    KeyStore trustStore = KeyStore.getInstance("pkcs12");
    trustStore.load(null, null);
    trustStore.setCertificateEntry("ca", trustedCa);
    SSLContextBuilder sslContextBuilder =
            SSLContexts.custom().loadTrustMaterial(trustStore, null);
    return sslContextBuilder.build();
  } catch (Exception e) {
      throw new RuntimeException(e);
  }
}

And that’s it. Running Elasticsearch secure by default in your integration tests with TLS and authentication enabled!

Complete code sample

This is the final code snippet, which can be also seen in this sample repo

public class ElasticsearchTests {

  private static final String IMAGE_NAME = 
      "docker.elastic.co/elasticsearch/elasticsearch:8.0.0";

  private static final ElasticsearchContainer container =
      new ElasticsearchContainer(IMAGE_NAME)
          .withExposedPorts(9200)
          .withPassword("s3cret");

  private static ElasticsearchClient client;
  private static RestClient restClient;
  private static ElasticsearchAsyncClient asyncClient;


  @BeforeAll
  public static void startElasticsearchCreateLocalClient() throws Exception {
    // remove from environment to have TLS by default enabled
    container.getEnvMap().remove("xpack.security.enabled");

    // custom wait strategy not requiring any TLS tuning...
    container.setWaitStrategy(
        new LogMessageWaitStrategy().withRegEx(".*\"message\":\"started\".*"));
    container.start();

    // extract the ca cert from the running instance
    byte[] certAsBytes = container.copyFileFromContainer(
        "/usr/share/elasticsearch/config/certs/http_ca.crt",
        InputStream::readAllBytes);

    HttpHost host = new HttpHost("localhost", 
       container.getMappedPort(9200), "https");

    final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
    credentialsProvider.setCredentials(AuthScope.ANY, 
        new UsernamePasswordCredentials("elastic", "s3cret"));
    final RestClientBuilder builder = RestClient.builder(host);

    builder.setHttpClientConfigCallback(clientBuilder -> {
        clientBuilder.setSSLContext(SslUtils.createContextFromCaCert(certAsBytes));
        clientBuilder.setDefaultCredentialsProvider(credentialsProvider);
        return clientBuilder;
    });
    builder.setNodeSelector(INGEST_NODE_SELECTOR);
    restClient = builder.build();

    final ObjectMapper mapper = new ObjectMapper();
    mapper.setPropertyNamingStrategy(PropertyNamingStrategies.SNAKE_CASE);
    mapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);

    ElasticsearchTransport transport = new RestClientTransport(
        restClient,
        new JacksonJsonpMapper(mapper)
    );
    client = new ElasticsearchClient(transport);
    asyncClient = new ElasticsearchAsyncClient(transport);
  }

    // tests go here
}

… aaaaaaand that’s it!

Now this still has required some manual effort on the client code side, so the next steps is to properly support this within Testcontainers, so that users do not need any extra code and have helper methods readily available. If you read this past it’s creation date in February 2022, make sure to check out the testcontainers documentation if jumping through these hoops is still needed.

Final remarks

You can follow or ping me on twitter, GitHub or reach me via Email (just to tell me, you read this whole thing :-).

If there is anything to correct, drop me a note, and I am happy to do so and append to this post!

Same applies for questions. If you have a question, go ahead and ask!

If you want me to speak about this, drop me an email!

References

Back to posts

Backend developer, productivity fan, likes the JVM, full text search, distributed databases & systems