A few days ago, I post an article about how to handle +9M request on a small EC2 instance with Laravel and, on Reddit, they said it's nothing. They said that any service is able to manage that amount of traffic easily.

Today, we're going to talk about my experience at Wallbox, where we handled over 455M requests per week in a PHP service done with Symfony, and over 725M requests per week in total.

Before, a bit of context

Wallbox is the leading supplier of smart EV chargers. Our chargers are connected to the internet 24/7, sending data to the backend every 3 seconds while charging and every 9 seconds when idle.

Our mobile and web apps provide users with real-time information, configuration changes and the ability to make any action or request.

Wallbox receives approximately 455 million requests per week in a single service and 725 million requests per week in all their services combined.

How we deal with that huge amount of traffic

1. Monitoring, Logging, and Alerting

You must have the right tools to monitor your application's performance. You need to be able to monitor the performance of your instances, the CPU and memory usage, and disk space. You also need proper logs to know what's happening and to discover and fix errors. Without the proper logs, each error could be a huge problem.

Without proper alerting, you won't be able to identify when something isn't working properly.

In Wallbox, we track and log every request. If we start receiving fewer requests from chargers, for example, an alert is triggered, we'll check the logs to see what's going on.

2. Background processes, and queues

Don't even think about handling that many requests in the foreground. You need background processes, queues, and messages/jobs.

With Laravel, you can use something as simple as a cron job and a MySQL database. However, when you need to scale, you will need something more powerful, such as AWS SES and SQS.

You must be able to trigger events and messages and perform background processes. It could be as simple as sending an email or creating or updating a resource.

Queues and messages are superpowerful because they allow you to scale. You can have as many background processes as you need, and you will see all the errors in your error queues. You can also retry and manage them again.

You must start thinking about how to split your use case into multiple asynchronous background jobs instead of doing everything during the same request.

3. Improve your CD/CI, be able to revert fast

Making your CD/CI fast and efficient is not going to help you improve performance and respond to more requests, but when the traffic you handle is so high, it becomes critical.

Sometimes we make small features or fixes that without realizing it were slightly worsening performance and we would instantly see, thanks to the monitors, that the machines were starting to saturate.

At that moment we could revert the changes and in a matter of seconds, the machines would return to the previous state without affecting our users. Then, thanks to the logs, we could see what had happened, adjust the RM, and bring it back up with more peace of mind.

This was day to day, no matter how many tests we had, it was impossible to predict the behavior of a change when processing +725 requests per week.

4. Forget about Slow Queries

When you have few requests you can afford to have some slow queries, but when you work with high amounts of traffic it is important to optimize them.

One of the things that worked best for us was creating projections.

These projections were something similar to a view. Instead of making complex and slow queries, with several joins, we create and maintain a projection, using events.

This projection is a table where we have the data ready to consume as we need it and queries go from slow to instantaneous.

Of course, before getting to this point, it is important to have a good queuing and event system, as well as logs and monitors. It can become impossible to know why a projection is out of date if you don't have these points previously covered.

← Go back to the blog