On Dec 13th most of our engineers were focused on the critical vulnerability (CVE-2021-44228) named Log4Shell that came out on Dec 10th. As security is one of the core values at Kameleoon, we immediately assessed the risk inherent to the solutions we used vs. the mitigating policies that had been implemented to guard our platform against future threats and deployed a release in production at 11:15am CET to prevent any issue from occurring due to Log4Shell.

However, in the meantime around 11am CET, our service started experiencing a critical outage impacting some of our customers who had running campaigns (A/B test or Personalization) which include an image update/modification. The updated images stopped being displayed in all live campaigns.

Transparency at Kameleoon is in our DNA and as such we want to share with you all the details of this outage and more importantly our action plan to prevent this from occurring again in the future.

We started receiving broken image links from one customer around 11am CET and pushed a temporary fix at 11:20am CET. After further investigations we identified the root cause of the issue: removing an experiment or a variant was wrongly triggering an image folder deletion process. At 10:30am Dec 14th a bug fix release was deployed in production to completely solve the issue.

Even if the outage was not global to all our customers and occurred only in specific cases, the consequences were serious. A very small percentage of all lost images were unable to be recovered by our engineering team and had to be manually re-uploaded (using Kameleoon graphic editor). This was due to some issues in the way the image library was backup-ed. We reviewed all our standard procedures for backups and will be moving the image backup process to the same level of resilience that we use for the backup of collected data (which is at a much higher level).

The quality of our service is our top priority and we strongly apologize to our customers who experienced this issue.

If you're not aware of any issues, that means you were most likely not impacted and all the safety mechanisms worked.
If you're still having issues, please get back to us by sending a request to support@kameleoon.com and we will look at it with top priority.