Root Cause: The incident stemmed from a bug within the service responsible for rebuilding the JS Application File (kameleoon.js). This bug resulted from the utilization of feature flags within the service's infrastructure and an anomaly was identified in the code of the underlying feature flag being used.
Impact: As a consequence of this incident, no new updates were published in production from Saturday, 20th to Monday, 22nd, 10:30 am CEST, as expected. Customers relying on the scheduling feature were particularly affected, experiencing delays in publishing/stopping experiments.
Resolution and Preventive Measures: The development team promptly identified and resolved the bug within the service. Moving forward, stringent testing protocols will be instituted to ensure the stability and reliability of the service. Additionally, proactive monitoring and regular audits will be conducted to detect and address potential issues before they escalate, thus reducing the likelihood of similar incidents occurring in the future.