Testing a Kafka consumer with Avro schema messages in your Spring Boot application with Testcontainers

Author: Anja Gruss

The short answer is, as usual: it depends. Let’s assume this situation:

The sample project uses Apache Maven with the avro-maven-plugin to download the schema files and generate the sources, but of course there are plugins for Gradle too.

Now you want to test that your Kafka consumer reads the events, transforms them into your database entities, and saves them.

All these solutions have their valid pros and cons.

In this blog post, I will present a solution that uses a minimum set of Testcontainers to provide the best compromise between control, speed and efficiency. The other solutions I listed above have their relative pros and cons, which I will cover later in this post.

Set up and configure your containers

And create a Kafka producer and consumer with their configurations for the tests.

Here we only add the mock prefixed schema registry url, no special serializers or deserializers. Don’t forget to set the property for the specific.avro.reader to “true” in your Kafka consumer configuration to ensure that the consumer does not fail with a class cast exception.

Of course, runtime is an issue. Testcontainers need to start and network before being fully available. A much faster alternative to my setup is to move all the infrastructure to memory, by using EmbeddedKafka as a broker and replacing PostgreSQL with the H2 database. You can find a working version of the EmbeddedKafka in the branch “embeddedKafka” (my naming skill is highly imaginative).

Admittedly, in terms of timing, the minimalistic setup with EmbeddedKafka and H2 is pretty hard to beat. The test contains the same methods as the main branch, and takes on average 5 seconds to run on my machine.

My version with one Kafka and one Postgres container takes on average 15 seconds to run.

Now imagine how long it would run with 4 Testcontainers (Schema Registry, Kafka and Zookeeper, plus the database container). And with more code, with more complex business logic to test. 😕

So I stuck to the container setup because of version constraints. In the case for this demo application it did not matter at all, but to avoid unexpected behavior I recommend sticking as close to the production versions as possible. And of course using the H2 might also not always be an option, depending on what features (such as constraints, json datatype, some join statements) you use of your production database. So this is a compromise regarding control and test runtime.

Also, did I already mention ‘it depends’?

Related posts:

La scorsa settimana ha fatto di nuovo segnare una diminuzione di contagi, ancor più netta di quella registrata sette giorni prima: poco meno di cinquemila nuovi casi in meno, 4924 per amor di…

After previous successful and efficient of card sorting methods link here. For the filters problem we decided to take tactile approach. There was nothing in particular which was better than digital…