Performance issue
Incident Report for Hello Retail
Postmortem

At 10.17 we were rearranging some of the newer servers to increase performance further. Part of this was changing the size of servers from many to fewer, but with much more resources. We are often changing the servers around to better serve you and your customers. However, this time we were going from many smaller servers to fewer larger, to be able to faster deploy hotfixes - if it would become necessary.

The issue we ran into was the number of threads each server could handle. This means that user's requests got queued, and customers experienced slower than normal response time both in rendering the recommendation boxes as well as when searching.

We are truly sorry for the inconvenience it had for you. We will keep monitoring the servers closely to ensure peak performance for you and your customers, especially on this important day.

If you have any further questions, don't hesitate to reach out to support@helloretail.com.

Customers affected: all customers using our recommendations and search solutions.

Type of issue: Degraded performance

Timeframe: The issue was discovered 10.17 GMT+1 and solved 10.39 GMT+1.

Posted Nov 29, 2019 - 11:16 CET

Resolved
The issue has now been solved, and we will later today provide you with a postmortem. We apologize for the inconvenience.
Posted Nov 29, 2019 - 10:45 CET
Investigating
We are currently experiencing performance issues which impact the load time on recommendation boxes and the search speed. We apologize for this, and will let you know when we have an update.
Posted Nov 29, 2019 - 10:31 CET
This incident affected: Admin dashboard, Product recommendations, and Search.