tag:status.helloretail.com,2005:/historyHello Retail Status - Incident History2024-03-28T16:48:02+01:00Hello Retailtag:status.helloretail.com,2005:Incident/198570122024-01-29T13:34:00+01:002024-01-29T14:06:18+01:00Issue with database<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>13:34</var> CET</small><br><strong>Resolved</strong> - We experienced a brief service interruption of 3 minutes during a recent deployment. This was due to a database transaction on a table that unexpectedly became unresponsive. Our team is actively investigating the root cause to prevent such occurrences in the future.<br /><br />We apologize for the inconvenience.</p>tag:status.helloretail.com,2005:Incident/192090062023-11-24T10:30:19+01:002023-11-24T10:46:52+01:00Degraded search performance between 09:17-09:30<p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>10:30</var> CET</small><br><strong>Resolved</strong> - We had degraded search performance with customers experiencing slow response times between 10:17 and 10:30. <br />The issue was resolved at 10:30. <br /><br />Sorry for the inconvenience.</p>tag:status.helloretail.com,2005:Incident/189315642023-10-26T17:10:52+02:002023-10-26T17:10:52+02:00Search analytics dashboard issues<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>17:10</var> CEST</small><br><strong>Resolved</strong> - The issue has now been resolved and the search analytics dashboard is now up to date. <br /><br />Sorry for the inconvenience and thank you for your understanding.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>11:30</var> CEST</small><br><strong>Identified</strong> - We have identified the issue and hope to have the dashboard fully up and running in a couple of hours.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>09:22</var> CEST</small><br><strong>Investigating</strong> - We are currently having issues showing search analytics within the dashboard - we are working on a resolution.,</p>tag:status.helloretail.com,2005:Incident/188757822023-10-21T09:00:00+02:002023-10-21T09:25:38+02:00Search database issue<p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>09:00</var> CEST</small><br><strong>Resolved</strong> - We experience issues with our search database causing an outage.</p>tag:status.helloretail.com,2005:Incident/186008932023-09-25T03:00:00+02:002023-09-26T21:03:53+02:00We are currently experiencing issues with our databases<p><small>Sep <var data-var='date'>25</var>, <var data-var='time'>03:00</var> CEST</small><br><strong>Resolved</strong> - We have had issues with our servers being down from 14.57 to 15.03 - we are looking into the reasons why this brought the service down and will be adding a post-mortem later. We apologize for the inconvenience.</p>tag:status.helloretail.com,2005:Incident/173401022023-05-23T12:55:00+02:002023-05-23T13:42:36+02:00All systems affected by downtime<p><small>May <var data-var='date'>23</var>, <var data-var='time'>12:55</var> CEST</small><br><strong>Resolved</strong> - Moving to a new underlying server system caused unexpected downtime which was resolved at 13:05. <br /><br />We apologize for any inconvenience this might have caused</p>tag:status.helloretail.com,2005:Incident/173401282023-05-23T09:18:00+02:002023-05-23T13:45:42+02:00"New" HelloRetail script unavailable<p><small>May <var data-var='date'>23</var>, <var data-var='time'>09:18</var> CEST</small><br><strong>Resolved</strong> - Moving to a new underlying server system made the new version of our script unavailable to the clients already using this new version. <br />The problem was resolved 09:30. <br /><br />Sorry for any inconvenience this might have caused.</p>tag:status.helloretail.com,2005:Incident/169111532023-04-19T14:30:00+02:002023-04-19T15:04:51+02:00Issues with our Pages solution<p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>14:30</var> CEST</small><br><strong>Resolved</strong> - At 14:28 we made a release that caused our Pages product to no longer be functional. This issue was resolved at 14:57. <br /><br />We apologize for the inconvenience.</p>tag:status.helloretail.com,2005:Incident/162689642023-02-24T10:00:00+01:002023-02-24T17:19:27+01:00Major outage<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>10:00</var> CET</small><br><strong>Resolved</strong> - Due to network issues with our primary database, there was a major outage for 3 minutes affecting all services from 4:57 - 5:00 pm.</p>tag:status.helloretail.com,2005:Incident/161478292023-02-15T01:30:00+01:002023-02-15T13:08:47+01:00Reduced availability<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>01:30</var> CET</small><br><strong>Resolved</strong> - At around 1:30, a combination of maintenance tasks caused our secondary database cluster to run slow enough, that virtually all queries ended up in the slow query log. This meant that all servers quickly ran out of disk space, causing them to crash almost simultaneously at 1:34. This meant that our automated recovery procedure could not keep up, and manual recovery was needed. This was completed on 1:48.<br />In the intermediate time, all newsletter tiles and some recoms (depending on the algorithm) would time out. No data was lost in the incident.<br />We apologize for the inconvenience.</p>tag:status.helloretail.com,2005:Incident/158222442023-02-02T03:35:27+01:002023-02-02T03:35:27+01:00Database Maintenance<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>03:35</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>03:30</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>22:32</var> CET</small><br><strong>Scheduled</strong> - We are doing some database maintenance and will therefore be down for 2-5 minutes at the specified time. <br /><br />We apologize for the inconvenience.</p>tag:status.helloretail.com,2005:Incident/158222582023-01-26T03:40:01+01:002023-01-26T03:40:01+01:00Database maintenance<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>03:40</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>03:30</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>22:35</var> CET</small><br><strong>Scheduled</strong> - We are upgrading our database engine, which sadly requires around 10 minutes of scheduled downtime.<br />All services will be affected by this upgrade for a maximum of 10 minutes.<br /><br />We apologize for the inconvenience.</p>tag:status.helloretail.com,2005:Incident/157890042023-01-07T04:09:21+01:002023-01-07T04:09:21+01:00We are currently experiences issues<p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>04:09</var> CET</small><br><strong>Resolved</strong> - We have solved the issue, and the services are running as normal. The root cause was a mistake in a change to our system for backing up customer data. <br /><br />We apologize for the inconvenience it had you and your customers.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>03:45</var> CET</small><br><strong>Monitoring</strong> - We are currently experiencing issues with our services which causes our products to run slowly, and we are looking into what might be the root cause.<br /><br />Our services started failing around 3:10. Services have been brought back up, and we are currently monitoring the situation.</p>tag:status.helloretail.com,2005:Incident/143189622022-12-04T10:51:22+01:002022-12-04T10:51:22+01:00High response time [Degraded performance]<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>10:51</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>10:45</var> CET</small><br><strong>Update</strong> - The issue has been resolved</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>10:40</var> CET</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.helloretail.com,2005:Incident/127053792022-11-03T14:18:32+01:002022-11-07T11:51:48+01:00We are currently experiences issues<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>14:18</var> CET</small><br><strong>Resolved</strong> - We have solved the issue, and the servers are running as normal. We are investigating more about why it happened and will get back with a post-mortem. <br /><br />We apologize for the inconvenience it had you and your customers.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>14:08</var> CET</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>13:54</var> CET</small><br><strong>Investigating</strong> - We are currently experiencing issues with our services which causes our products to run slowly, and we are looking into what might be the root cause.</p>tag:status.helloretail.com,2005:Incident/108518512022-08-18T11:00:00+02:002022-08-18T12:00:49+02:00Disruption of service<p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>11:00</var> CEST</small><br><strong>Resolved</strong> - Our services were unfortunately disrupted from 11:17 to 11:27 due to an issue caused when releasing improvements to our products.<br /><br />All services are running normally again.<br /><br />We are sorry for any inconvenience it has had to you and your customers</p>tag:status.helloretail.com,2005:Incident/107823792022-08-08T14:28:04+02:002022-08-08T14:30:24+02:00Database issue<p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>14:28</var> CEST</small><br><strong>Resolved</strong> - The issue has been resolved, and all services are working and stable again. We apologize for the inconvenience it had.</p><p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>14:19</var> CEST</small><br><strong>Investigating</strong> - We are currently experiencing a database issue that is causing a disturbance in our service. We have found the error and are deploying a fix.</p>tag:status.helloretail.com,2005:Incident/91841572022-01-28T13:57:47+01:002022-01-28T13:57:47+01:00High response time [Degraded performance]<p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>13:57</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>13:48</var> CET</small><br><strong>Investigating</strong> - Customers might experience degraded performance. We are currently investigating</p>tag:status.helloretail.com,2005:Incident/91677082022-01-26T14:28:20+01:002022-01-26T14:30:30+01:00Degraded search performance<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>14:28</var> CET</small><br><strong>Resolved</strong> - The search is now working at full capacity. We apologize and are working on a solution to prevent this from happening again.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>14:21</var> CET</small><br><strong>Identified</strong> - Unfortunately, we are experiencing degraded search performance - we are working on a solution. We apologize for the inconvenience it has for you and your customers.</p>tag:status.helloretail.com,2005:Incident/91668542022-01-26T12:54:27+01:002022-01-26T12:54:27+01:00Degraded search performance<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>12:54</var> CET</small><br><strong>Resolved</strong> - Search is now running normally again. We will examine the root cause of the issue to prevent it from happening again</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>11:47</var> CET</small><br><strong>Identified</strong> - We are currently experiencing slower than expected performance on searches. The issue has been identified and is being fixed.</p>tag:status.helloretail.com,2005:Incident/77053742021-08-07T13:27:17+02:002021-08-09T15:29:23+02:00Disruption of service<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>13:27</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>13:26</var> CEST</small><br><strong>Update</strong> - The services are now stabilized, and we will continue to monitor the solution. We are looking into ways to prevent issues like this from arising in the future. <br /><br />Thank you very much for your patience, and sorry for any inconvenience it has had to you and your customers.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>12:28</var> CEST</small><br><strong>Investigating</strong> - We are unfortunately experiencing issues with the service. We are working on a fix.<br /><br />Sorry for any inconvenience that it is causing.</p>tag:status.helloretail.com,2005:Incident/69333062021-05-06T09:19:00+02:002021-05-06T09:40:18+02:00Temporary issue with response times<p><small>May <var data-var='date'> 6</var>, <var data-var='time'>09:19</var> CEST</small><br><strong>Resolved</strong> - We had 4 minutes of service interruption from 9.19 to 9.23. We are investigating why the interruption occurred.<br /><br />Sorry for the inconvenience.</p>tag:status.helloretail.com,2005:Incident/69250742021-05-05T11:37:31+02:002021-05-06T11:07:18+02:00Stability issues<p><small>May <var data-var='date'> 5</var>, <var data-var='time'>11:37</var> CEST</small><br><strong>Resolved</strong> - A fix has been deployed, and we are running as usual. <br /><br />Again, sorry for the inconvenience, we are working to prevent this from happening again.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>11:29</var> CEST</small><br><strong>Investigating</strong> - We are currently experiencing issues due to an error in the deployment of a release. We will keep you posted with more updates as soon as we know more.<br /><br />We are really sorry for the inconvenience.</p>tag:status.helloretail.com,2005:Incident/67418522021-04-13T20:15:01+02:002021-04-13T20:15:01+02:00Search issue<p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>20:15</var> CEST</small><br><strong>Resolved</strong> - The final investigation has is done, and the search for all customers should work as expected. We are looking into how we can prevent issues like this in the future, and again, sorry for the inconvenience it might have caused you.</p><p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>17:13</var> CEST</small><br><strong>Monitoring</strong> - We have identified the cause of the issue and released a fix that makes sure all products can be found in search again. We are closely monitoring the system to make sure that there are no further issues. <br />We will resolve this issue when the final investigation is done.<br /><br />We are very sorry any inconvenience you may have had.</p><p><small>Apr <var data-var='date'>13</var>, <var data-var='time'>15:43</var> CEST</small><br><strong>Investigating</strong> - We are currently experiencing an issue where a small number of customers are experiencing problems with their search solutions. For these customers, not all products can be found in the search. We are investigating, and are in the process of rolling out a fix. <br /><br />We are very sorry any inconvenience you may have had.</p>tag:status.helloretail.com,2005:Incident/51657282020-09-24T12:04:00+02:002020-09-24T12:04:00+02:00Limited availability<p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>12:04</var> CEST</small><br><strong>Resolved</strong> - The issue was resolved 11.29. We are continuing to look into the root caues and will provide more information when we know more.<br /><br />We are genuinely sorry for the inconvenience this caused.</p><p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>11:28</var> CEST</small><br><strong>Investigating</strong> - We are currently have limited availability our services, we are investigating.</p>