Disruption of service
Incident Report for Hello Retail
Postmortem

Sunday between 10.03 and 10.35 GMT+1 we had issues that caused an interruption of the service. We always have a lot of additional server power to catch spikes in traffic, but in this case, a wrongly configured newsletter caused an extremely high additional load. Normally we are aware if servers are running low on resources, but in this case, the spike was instant due to the error. This very high additional load made all the requests queue so we couldn't serve recommendation boxes and search requests.

As soon as we found the culprit we increased the server capacity so it can handle such errors, and at the same time fixed the specific error. We will continue to run at much higher server capacity until we deploy a fix, so an error like this won't be able to cause issues.

We know this has been very inconvenient to you, and we are sorry for the interruption, and we will do everything in our power that this issue won't arise again.

Posted Aug 11, 2020 - 12:48 CEST

Resolved
Services are now operational - we are investigating why the incident occurred. We will provide you with a more detailed explanation when we know more.

We are sorry for the inconvenience it caused you.
Posted Aug 09, 2020 - 10:42 CEST
Investigating
We are currently experiencing downtime with our services, we are investigating the cause.
Posted Aug 09, 2020 - 10:21 CEST
This incident affected: Admin dashboard, Product recommendations, and Search.