Testcontainers for Spring Boot Integration-Tests with Redis, MariaDB and Gitlab CI/CD

Integration Testing… an often neglected part of software development. It's hard, it's cumbersome, it's flaky, it requires a lot of discipline and maintenance… all in all it's not much fun.

Personally, I'm also strongly team Unit-Test as they are a lot easier to maintain and a hell-a-lot faster to run than Integration Tests, but Integration Tests are still essential to ensure the quality of our software too.

So, we write the year 2022, and by looking at the state of Integration Tests for Spring Boot, we can see that it has come a long way. It's more stable and reliable than ever, partly thanks to Testcontainers.

Testcontainers provide us with a reliable way to bootstrap Docker services alongside our Integration Tests, and easily wire those into the Spring Boot Application Context to be used by our tests.

So this time, let's take a look at how we can integrate Testcontainers in our test setup, and while we are at it also integrate it in our GitLab CI/CD pipeline to have them run fully automatically.

The Application to Test

In order to see how Testcontainers Integration Tests work, we obviously need an application to test.

So we are going to build a Book Service which provides us with the ability to add books, rate them and retrieve a list of already added books. The data will be stored in MariaDB, and the list of books will be cached on a Redis instance utilizing Springs Cache abstraction.

Why exactly this setup? Well, having the DB in an Integration Test is kind of a prime example, even though it was already long before possible to use the in-memory H2 database for Integration Tests. Showcasing how it's now just as comfortable to utilize the real DB implementation is kind of a must.

It's also quite common for applications to use some kind of caching. As this often utilizes a good deal of AOP goodies provided by the Spring Cache abstraction, it is also often one of those parts that makes Integration Tests a bit harder to set up.

Application Structure

The following illustrates the overall structure of the book service we are going to test via Integration Tests.

We'll not go into the details of every component of our application, but focus only on those relevant enough to get a good overview of what we are going to test, and how it's done. The full service implementation can be found in this GitLab repository.

├── pom.xml
└── src
    ├── main
    │   ├── java
    │   │   └── io
    │   │       └── auroria
    │   │           └── book
    │   │               ├── BookApplication.java
    │   │               ├── api
    │   │               │   └── BookApi.java
    │   │               ├── config
    │   │               │   └── CacheConfiguration.java
    │   │               ├── domain
    │   │               │   ├── BookDTO.java
    │   │               │   ├── BookService.java
    │   │               │   ├── NewBookDTO.java
    │   │               │   └── NewBookRatingDTO.java
    │   │               ├── impl
    │   │               │   └── DefaultBookService.java
    │   │               └── persistence
    │   │                   ├── BookRepository.java
    │   │                   ├── RatingRepository.java
    │   │                   └── entity
    │   │                       ├── Book.java
    │   │                       └── Rating.java
    │   └── resources
    │       └── application.yml
    └── test
        ├── java
        │   └── io
        │       └── auroria
        │           └── book
        │               └── BookApplicationTest.java
        └── resources
            └── application.yml

The API Layer

We define 3 endpoints for the service:

1. POST /books adds new books to our service
2. POST /books/{id}/rating allows us to rate a book
3. GET /books retrieves all the books we added, including an average score of each.

@RestController 
@RequestMapping("books")
public class BookApi {
    private final BookService bookService;

    @PostMapping
    @CacheEvict(value = "books", allEntries = true)
    public BookDTO addBook(@RequestBody NewBookDTO newBookDTO) {
        return bookService.addBook(newBookDTO);
    }

    @PostMapping("{bookId}/rating")
    @CacheEvict(value = "books", allEntries = true)
    public BookDTO addRating(@PathVariable("bookId") Long bookId, 
                             @RequestBody NewBookRatingDTO newBookRatingDTO) {
        return bookService.addRating(bookId, newBookRatingDTO);
    }

    @GetMapping
    @Cacheable(value = "books")
    public List<BookDTO> getBooks() {
        return bookService.getBooks();
    }
}

We can see that this is also where our caching logic is going to be triggered. Requesting the book list is going to cache it, while adding new books or rating a book is going to evict the redis cache.

Service Implementation

The service implementation takes the data from our API layer, processes it and passes it on to the persistence layer utilizing Spring Data Repositories.

@Service
public class DefaultBookService implements BookService {

    private final BookRepository bookRepository;
    private final RatingRepository ratingRepository;

    @Override
    public BookDTO addBook(NewBookDTO newBookDTO) {
        var newBook = new Book(null, newBookDTO.getName(),
                newBookDTO.getAuthor());
        var savedBook = bookRepository.save(newBook);

        return convertToBookDTO(savedBook, Collections.emptyList());
    }

    @Override
    public BookDTO addRating(Long bookId, 
    						 NewBookRatingDTO newBookRatingDTO) {
        ratingRepository.save(new Rating(null, bookId,
                newBookRatingDTO.getRating()));

        var book = bookRepository.getReferenceById(bookId);
        var ratings = ratingRepository.findByBookId(bookId);

        return convertToBookDTO(book, ratings);
    }

    @Override
    public List<BookDTO> getBooks() {
        var books = bookRepository.findAll();
        var bookIds = books.stream()
                .map(Book::getId)
                .collect(Collectors.toList());
        
        var ratings = ratingRepository.findByBookIdIn(bookIds);

        return books.stream()
                .map(b -> {
                    var bookRatings = ratings.stream()
                            .filter(r -> r.getBookId().equals(b.getId()))
                            .collect(Collectors.toList());

                    return convertToBookDTO(b, bookRatings);
                })
                .collect(Collectors.toList());
    }

    // ...
}

Persistence Layer

In order to be able to save our books and their accompanying ratings, we create the following entities …

@Entity
public class Book {
    @Id @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String name;
    private String author;
}

@Entity
public class Rating {
    @Id @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private Long bookId;
    private int rating;
}

… along with their repositories to get access to them …

@Repository
public interface BookRepository extends JpaRepository<Book, Long> { }

@Repository
public interface RatingRepository extends JpaRepository<Rating, Long> {
    List<Rating> findByBookId(Long bookId);
    List<Rating> findByBookIdIn(List<Long> bookIds);
}

Adding Testcontainers Dependencies

As our service implementation is now finalized, it's time to add the dependencies needed for Testcontainers to work its magic.

<dependencies>        
	<!-- .... -->
    <dependency>
        <groupId>org.testcontainers</groupId>
        <artifactId>junit-jupiter</artifactId>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.testcontainers</groupId>
        <artifactId>mariadb</artifactId>
        <scope>test</scope>
    </dependency>
    <!-- .... -->
</dependencies>

<properties>
    <testcontainers.version>1.17.6</testcontainers.version>
</properties>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>testcontainers-bom</artifactId>
            <version>${testcontainers.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

Testcontainers comes with its own generic implementation for JUnit Jupiter, as well as with multiple pre-configured modules for services commonly needed in Integration Tests, f.e. mariadb.

Additionally, in order to fetch the correct version of every Testcontainers module, we are leveraging the testcontainers-bom for dependency management.

Configuring MariaDB Container Connection

As we imported all necessary test dependencies, we can go on configuring our test application setup in the application.yml configuration file in the test resources folder.

spring:
  datasource:
    driver-class-name: org.testcontainers.jdbc.ContainerDatabaseDriver
    url: jdbc:tc:mariadb:10.3:///books
  jpa:
    hibernate:
      ddl-auto: create
  cache:
    type: redis

First, we instruct Spring to use the ContainerDatabaseDriver provided by Testcontainers to connect to a database. Second, we define the literal mariadb:10.3 Docker image name in the JDBC connection string, prefixing it with tc: in order to pass the JDBC connection string resolution request on to the Testcontainers driver.

This allows the ContainerDatabaseDriver to determine which version of MariaDB we'd like to utilize. It'll pull the docker image of that container and run it alongside our test runs.

Additionally, we also instruct Spring to specifically use redis for our caching backend. Notice that we don't specify any connectivity property like host or port. These will be specified dynamically in the tests.

Setting up the Tests

Finally, to the main part, setting up the Integration Tests. We start out with a plain simple Spring Integration Test, the only difference being that we additionally annotate the test class with @Testcontainers.

After that, we can write our first Testcontainers Integration Test, which for the moment only utilizes MariaDB.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@Testcontainers
class BookApplicationTest {

    @Autowired
    DefaultBookService bookService;

    @Test
    public void addBookViaService() {
        var author = "Josh Long, Kenny Bastani";
        var book = new NewBookDTO("Cloud Native Java", author);
        var resultBook = bookService.addBook(book);

        assertThat(resultBook.author()).isEqualTo(author);
        assertThat(resultBook.id()).isNotNull();
    }
}

We autowire the DefaultBookService, and access it directly to add a new book. The book gets saved into our database, and the result is being returned.

Great, the MariaDB connection works, we can verify this by checking the logs of our running test.

> 2022-11-24 14:34:58.430  INFO 29952 --- [main] 🐳 [mariadb:10.3]: Creating container for image: mariadb:10.3
> 2022-11-24 14:34:58.567  INFO 29952 --- [main] 🐳 [mariadb:10.3]: Container mariadb:10.3 is starting: bca3b984bb6d05b22c58d9fc6d53470f41e9059c1d0945a7a256d9a5ee70d19c
> 2022-11-24 14:34:58.860  INFO 29952 --- [main] 🐳 [mariadb:10.3]: Waiting for database connection to become available at jdbc:mariadb://localhost:52028/test using query 'SELECT 1'
> 2022-11-24 14:35:05.181  INFO 29952 --- [main] 🐳 [mariadb:10.3]: Container is started (JDBC URL: jdbc:mariadb://localhost:52028/test)
> 2022-11-24 14:35:05.182  INFO 29952 --- [main] 🐳 [mariadb:10.3]: Container mariadb:10.3 started in PT6.7528S

It's great, but it's not a full Integration Test yet, is it? Yes, the Application Context booted correctly, yes, stuff is being stored in the DB, but the direct injection of the service is not going to test the full Request-Response circle.

So let's set up the next layer, the API layer. As the API layer makes use of the Springs Cache abstraction, we need to add redis at this point. Unfortunately, there exists no Testcontainers module yet which could be as easily imported as mariadb.

Nevertheless, as Testcontainers allow us to basically bootstrap any Docker image, we can just create a static GenericContainer field in our test class, annotate it with @Container and instantiate any Docker container we'd like, redis in our case.

Additionally, we dynamically reconfigure the Spring properties via redisProperties(DynamicPropertyRegistry registry) making it aware that alongside the test a redis Docker instance has been started, and it should use the given host and port address to connect to it. (This is why we previously didn't need to specify those properties in the application.yml file.)

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@Testcontainers
class BookApplicationTest {
    // ...
    
    @Container
    static GenericContainer redis =
            new GenericContainer(DockerImageName.parse("redis:7"))
                    .withExposedPorts(6379);

    @DynamicPropertySource
    static void redisProperties(DynamicPropertyRegistry registry) {
        redis.start();
        registry.add("spring.redis.host", redis::getHost);
        registry.add("spring.redis.port", redis::getFirstMappedPort);
    }

    @Autowired
    BookApi bookApi;

    @Test
    public void addBookViaApi() {
        var author = "Betsy Beyer, Chris Jones, Jennifer Petoff " +
                "and Niall Richard Murphy";
        var book = new NewBookDTO("Site Reliability Engineering", author);
        var resultBook = bookApi.addBook(book);

        assertThat(resultBook.author()).isEqualTo(author);
    }
}

This way, we can now inject the BookApi in our tests, and directly call its methods to verify the workings of our application, but also this is still not a full Request-Response circle.

To achieve a full Request-Response style Integration Test, we can inject the TestRestTemplate which is aware of our running web server due to the fact that we added webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT to our @SpringBootTest annotation.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@Testcontainers
class BookApplicationTest {
	//...
    
    @Autowired
    private TestRestTemplate restTemplate;

    @Test
    public void addRatingViaRestTemplate() {
        restTemplate.postForEntity("/books/1/rating", 
                new NewBookRatingDTO(10), BookDTO.class);
        restTemplate.postForEntity("/books/2/rating", 
                new NewBookRatingDTO(8), BookDTO.class);
        var response = restTemplate.postForEntity("/books/1/rating",
                new NewBookRatingDTO(2), BookDTO.class);

        assertThat(response.getBody().averageRating()).isEqualTo(6);
    }

    @Test
    public void getAllBooksREST() {
        var response = restTemplate.exchange("/books", HttpMethod.GET,
                null, new ParameterizedTypeReference<List<BookDTO>>() {
                });

        assertThat(response.getBody().size()).isEqualTo(2);
    }
}

Via that template, we can call the API like a normal end-user would call it, retrieve the result and verify its correctness.

Hand it over to the CI/CD lifecycle

After we've implemented our service, and verified its workings via Integration Tests which utilize Docker containers to bootstrap external services needed by our application to fulfill its duty, we might think we are done.

We are not.

Tests exists to be run, over and over again, which is not hard per se, but hard to memorize to do every time we'd like to deploy a new version of our application.

There is a saying that states “If it's not automated, it's not going to happen”, and that's very true. Tests which are not run frequently tend to rot very, very fast.

In order to not let this happen to us, we integrate the test suite into our GitLab CI/CD lifecycle.

stages:
  - test

build:
  variables:
    DOCKER_HOST: "tcp://docker:2375".
    DOCKER_TLS_CERTDIR: ""
    DOCKER_DRIVER: overlay2
  services:
    - name: docker:dind
      command: ["--tls=false"]
  image: maven:3-openjdk-17-slim
  stage: test
  script:
    - mvn clean verify

We are running our job inside the maven Docker image, but utilize docker:dind(Docker in Docker) at the same time as a service for our container test suite to start up the Testcontainers Docker services needed to run the tests.

We follow the official docs quite closely here.

Now, whenever we commit & push new code, our project is built from the ground up and all integration tests are being executed.

Congratulations

Nicely done, by now you should have a working setup, utilizing Testcontainers to build first class Integration Tests for your Spring Boot application.