Quick look at component testing

Cover image

Component testing for easier developer life.

by Aleksander Piotrowski · 4 min read testing component

Component testing is one of the greatest things for microservices we have recently discovered at Ingrid. Right now, we are introducing them in all of our microservices as it can speed up the development of new features and increase confidence about them properly working.

Component testing is testing one microservice at a time in isolation. With those types of tests, you can make sure all connections are correct, SQL statements are right, and responses from your endpoints are on point.

Testing triangles
Component testing replaced some of our integrations tests

This article aims to show you what component tests are and how you can do them on your own with whatever language you use.

Why are we using component tests at Ingrid?

Shorter inner dev looppermalink

At Ingrid, we are working with many microservices, databases, and pub-subs that are connected with each other. It is hard to be sure that our code works as intended and does not introduce new bugs based only on unit tests. On the other hand, testing everything with integration and end-to-end tests is slow and requires deploying new code to development environment. Deploying code takes only a few minutes so it is not a problem when you are doing it once. However, everybody had that situation, when after deploying code you found one little bug and you have to deploy code again, and again and again.

Standard inner dev loop with kubernetes
That is how I worked before the introduction of component tests

With component tests, we managed to cut off a lot of time from this development loop. Right now, when we want to check if our feature is working as intended, all we have to do is call one command. It is essential as testing should be as painless as possible and as fast as possible. Slow tests will not be used by programmers.

We are able to find regressions and broken features straight away. We don't rely solely on testers and e2e tests. Component testing is great as it is:

  • fast
  • allows to quickly add new tests
  • run against real databases and pub-subs.

Standard inner dev loop with kubernetes and component tests
This is how my workflow looks now

Finding errors before they reach high load on productionpermalink

With component testing we can simulate many different situations that involve a high load of requests or pub-sub errors.

Recently, I found a deadlock in the database while developing a new feature. Normally it would be found by our testers later, or even worse, on production during black Friday. That would be a very nervous day.

Component testing makes it really easy to simulate all sorts of situations like other service failures, pub-sub mistakes, or just high traffic. Thanks to that, we can find bugs we would normally find on production when it could be too late for some of our clients.

Problems with component testing

Although component tests are great, they are not a silver bullet that would solve everything. You have to be aware that it will not allow you to remove all other tests. Unit and end-to-end tests are still a must-have in your projects.

First of all, they may be heavy on resources. If you run the tests locally in Docker, the heavy load may slow down the machine, especially if they utilize databases and pubsub emulators. That's something to keep in mind if you have a not-so-beefy machine.

Another issue is high entry-level. You have to know docker and docker-compose, look for proper images, code a lot of helpers yourself to work with the database migrations, pubsub subscriptions, etc.

How are we writing component tests at Ingrid?

To create your first component test, you will need docker and docker-compose. Tools like journey or migrate might also be handy. In this article, I will only briefly explain how we are doing it at Ingrid. Next time I will dive into details.

TL;DR

Component test in a nutshell:

  • Run docker with external dependencies that can't be mocked (databases, pubsubs, etc.)
  • Run migrations
  • Run other services with mocked responses
  • Build and run service you want to test
  • Run tests

Before running testspermalink

Docker

Check what external services you will need to create component tests. When you gather all information, create docker-compose.yml with needed services.

version: '3'

services:
pubsub-emulator:
image: google/cloud-sdk:339.0.0
ports:
- 8085:8085
command: gcloud beta emulators pubsub start --host-port=0.0.0.0:8085
mysql:
image: mysql:8
environment:
- MYSQL_DATABASE=som_test
- MYSQL_ALLOW_EMPTY_PASSWORD=true
ports:
- 3306:3306

Important note: Always run services with 0.0.0.0. This way it will be available outside docker container

You can run migrations manually before running tests via migrate/journey cli or create functions that will do it for you automatically. We are using migrate library for that.

Mock services

When you use proto buffers, it is effortless to mock service responses just by running those services with mock implementation.

type coolServer struct {
pbcool.UnimplementedCoolServer
}

func NewCoolServer(addr string) *coolServer {
coolServer := &coolServer{}

lis, err := net.Listen("tcp", addr)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}

grpcServer := grpc.NewServer()
pbcool.RegisterCoolServer(grpcServer, coolServer)
go grpcServer.Serve(lis)

return coolServer
}

If you are using good old REST for communication, you can use an application like json-server for mocking.

Writing testspermalink

var mainCLI mainServer

func TestMain(t *testing.M) {
// Component tests will run only when you explicitly tell them to run.
// To run component tests set env variable
if os.Getenv("RUN_COMPONENT_TESTS") != "true" {
return
}

// Initialize all mocks
NewCoolServer(":15444")

// Run main service
NewMainServer(":15445")

// Connect with main service
conn, err := grpc.Dial(":15445", opts...)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
mainCLI := pbmain.NewMainClient(conn)

// Run tests
t.Run()
}

// Write normal test just like you would with unit tests.
// Only difference is that you are calling real endpoint.
func Test_ImportantEndpoint(t *testing.T) {
t.Parallel()

resp, err := mainCLI.ImportantEndpoint()
if err != nil {
...
}

if resp != ... {
...
}
}

That is how our tests looked like when we started with this approach. They have a lot of repetition involved with real-life scenarios.
Right now, we are preparing our internal library created for component testing to go open-source. After that, I will create a new article that will cover doing component tests in Go using our library in a detailed way.

Wrapping up

Writing component tests is not easy. It requires large upfront investment in the beginning when you do not know how to do them properly. Although it is still worth investing this time, as it can reduce the feedback loop and make you more confident about your code.


Cover photo by @nahakiole on Unsplash

Does Ingrid sound like an interesting place to work at? We are always looking for good people! Check out our open positions