Degraded Application availability
Incident Report for Kameleoon
Resolved
Availability has remained high for the last 3 hours. This incident has been resolved. Thanks again for our customers’ patience and understanding during this incident.
Posted Oct 22, 2020 - 16:04 CEST
Update
Since Monday 19, servers providing all experiments' results have been returning a high percentage of errors causing intermittent Kameleoon's application (back-office) availability. The root cause has now been fully identified and is due to requests done to access old experiments results (older than 1 year).

The engineering team is currently pushing several fixes in production to definitely solve the issue. Stability will be up in the next few hours. New servers will also be added soon to fully stabilize the situation, so expect a short unavailability on results pages.

To ensure a better service in the future, the different components of our back-office will be more separated so that, if one component is having trouble, it does not affect the others. This will allow customers unaffected by the root issue (which should have only directly concerned about 5% of all data queries) to get normal service as usual.

Please note that the availability of the two mission-critical components of Kameleoon (our main script for your visitors and the data collection parts of our infrastructure) are totally shielded from the issues we encountered, which means that the performance of your website and the accuracy of the data collected were not affected at all. Only getting access to reports has been difficult.

We are grateful for our customers’ patience and understanding during this incident and look forward to continuing to provide the exceptional customer service and support that has been our hallmark and tradition.
Posted Oct 22, 2020 - 11:37 CEST
Monitoring
Our data processing pipeline that reports visit and conversion numbers in both A/B testing and Personalization result pages is running behind and could be delayed by up to 12 hours. The missing data is currently being recovered and all will quickly come back to a normal situation.
Posted Oct 21, 2020 - 12:37 CEST
Identified
The replacement servers allowed for a brief recovery period before availability dropped again this morning. The engineering team is looking to identify the root cause of the repeated failures. We strongly apologize for the inconvenience and will update as soon as we have more visibility.
Posted Oct 21, 2020 - 09:29 CEST
Monitoring
The application availability has been restored and the results pages are now loading as expected.
Posted Oct 20, 2020 - 14:22 CEST
Investigating
We are currently investigating an issue causing Kameleoon back-office application unavailability for all customers. We will post additional updates as soon as we have more information available. All experiments are still active and data is still being collected. We apologize for any impact our customers are experiencing.
Posted Oct 20, 2020 - 11:20 CEST
This incident affected: Main application (Back-office and graphic editor) (Back-office (app.kameleoon.com), Graphic editor).