Backend API

Post-Mortem - Back-end incident 13th of April 2022

Résolu
Problème signalé

Situation:
On April 13th 20:40 CEST Zivver executed a back-end release.
Part of this release was the upgrade of a component related to header information.

Impact:
The change has led to certain messages not being delivered. Zivver messages being sent from a specific server hosted by Amazon Web Service were affected.
This was unfortunately not discovered in our QA testing pre- and post-release.

Solution:
Zivver discovered based on customer feedback the relation between this issue and the release, after which the rollback was initiated.
Affected users were informed and after April 14th 1128 CEST the rollback was deployed.

Root-cause:
Zivver added additional functionality related to sender header information which was not accepted by Amazon for domains you have not claimed as hosted solution provider.
Unfortunately, the QA testing didn’t include this specific Amazon server.

Mitigating Actions:

  1. Extend domains representing all mail delivery servers and send flows of Zivver.
  2. Monitoring increased for send flows and specify threshold for specific alerts.