Some targeting conditions have been removed from segments that can be used in campaigns

Incident Report for Kameleoon

Postmortem

Yesterday, our engineering team deployed an update to our platform to address several minor bugs. One of the fixes required structural changes to our main database, and an update script had to be executed to implement those changes. Unfortunately, a critical bug in the upgrade script went unnoticed during our internal reviews, resulting in some data loss. While the technical details of the bug are intricate, the outcome was mostly random data loss. This affected a small percentage of our customer base, as some segment conditions were modified for customers who had previously deleted certain custom data, particularly if it was their first custom data entry.

To remedy this, we initiated our standard data restoration process to recover the lost data. Given the complexity of the data changes, it took our team approximately 8 hours to develop functional code for data restoration. The code was then run in the staging environment before being deployed to production.

In order to prevent similar issues in the future, our team is actively developing a new process to more effectively categorize the potential impact of seemingly minor releases. We aim to flag and schedule dangerous releases appropriately.

A communication will be released in-app for all affected campaigns. If one of your campaign is affected, we recommend ignoring the date of the 21st of November by using the timerange filter in your analysis to obtain accurate insights for this particular campaign.

Once again, we apologize for the inconvenience.

Posted Nov 22, 2023 - 10:55 CET

Resolved

We have successfully restored the affected segments to the last known state before the bug occurred in production this morning. If you are still experiencing issues with any of your campaigns, please reach out to our team at support@kameleoon.com. We sincerely apologize to impacted customers for any inconvenience caused today. Additionally, we will be conducting a post-mortem and will share the findings with you.

Posted Nov 22, 2023 - 01:00 CET

Update

We are actively working on restoring the affected segments, but the process is taking longer than anticipated. We will provide an update as soon as possible and apologize for any inconvenience caused by the delay.

Posted Nov 21, 2023 - 21:17 CET

Update

We have implemented a fix and are currently testing it in our staging environment before releasing it to all affected customers.

Posted Nov 21, 2023 - 16:28 CET

Update

We've identified the underlying bug in production and are working urgently to restore affected segments. We apologize for any inconvenience and strongly recommend our customers check their campaign settings while we fix the issue.

Posted Nov 21, 2023 - 15:24 CET

Identified

We're investigating an issue impacting live campaigns in production. The issue concerns only customers who had removed custom data in their account. When that's the case, there is a collateral effect on some segments that belong to the same project.

We're urgently working to restore affected segments.

Posted Nov 21, 2023 - 14:40 CET

Investigating

Some targeting conditions seem to have been removed from some segments and clients. We are investigating and seeking a way to resolve this as quickly as possible.
It's possible that some segments end up being empty.

Posted Nov 21, 2023 - 14:10 CET

This incident affected: Web application snippet (Campaign triggering) and Main application (Back-office and graphic editor) (Back-office (app.kameleoon.com)).