On Tuesday, September 3rd, at 1:12pm ET (10:12am PT), PaymentExpress staff deployed a routine software change which unexpectedly broke an API contract between our Public Site and our Payment Processing API. As a result, our Public Site was unable to lookup processing fees, which prevented customers from completing checkouts between 1:12 and 1:31 pm ET (10:12 - 10:31am PT). We restored service by reverting the problematic release.

This incident resulted from an unexpected difference in behavior between our production and non-production systems, which prevented our automated and manual testing from identifying the issue in non-production. We have found the bug that resulted in our production environment behaving differently from non-production, and are working to correct it in an upcoming release. Additionally, we will be improving our release monitoring to more easily identify issues with API consumers. This will allow us to more quickly recognize downstream errors and revert a release of our API if those consumers appear to be adversely impacted.