Summary

On April 1, 2026, between 9:32 AM and 10:13 AM Eastern Time, several secondary services that support TaxSys functionality experienced a disruption during a failover of our production environment. The majority of TaxSys functionality remained available, however some workflows were impacted as a result of issues with these secondary services.

The most notable impacted services were those that support login, transmission of emails, transmission of payment information from PaymentExpress to TaxSys, and the storage of payment receipt history. The full impact of these issues is described in the following section.

Impact

Login

• During the period of impact, some staff were unable to log in to TaxSys, and some customers were unable to log in to public sites. Staff or customers who were already logged in still had access to those sites. Customers could still use public sites without logging in.

Email Sending

• Some emails (for example, certain BillExpress or reminder campaigns) did not send or were delayed during the incident. We are working directly with the affected counties to either resend those notices or confirm that no further action is needed.

Post Payment Actions

• For some completed payments, follow-up actions which are typically performed immediately after the successful payment were delayed. Most notably, this impacted posting of successful payment data from PaymentExpress to TaxSys, and the storage of payment receipt history on public sites. This temporarily prevented the systems from remaining in sync and in some cases prevented clerks from completing the checkout process.
• Due to the failure to post payment data back to TaxSys, in some cases clerks may have been unaware of the success of payments, leading to unnecessary duplicate payment attempts.
• These delayed actions were completed later in the day after service was restored, ensuring records ultimately were brought back in sync.

Online Payments

• Online payment workflows requiring login (such as creating and using saved payment methods) were unavailable.
• Due to the post-payment synchronization issue mentioned above, customers paying online may have had confusion about the success of payments, leading to unnecessary duplicate payments.
• All duplicates were identified and corrected via PaymentExpress before the charges were settled.

Root Cause

The incident was triggered by a problem in our data center failover process for these secondary services.

During a change in how traffic was routed between our data centers, an unexpected race condition in our automated failover procedure left the portion of our database powering these services in an unhealthy state, resulting in errors and timeouts for the services that depend on it.

To restore service, our support team:

1. Moved traffic back to the original data center.
2. Repaired and stabilized the affected database cluster.
3. Began cleanup work to resend delayed messages, restore missing receipts, and correct any duplicate or missing records.

Action Items

To address the underlying issue and reduce the risk of similar incidents in the future, we are taking the following steps:

1. Armor our failover process against the race condition which impacted this database. We have identified the race condition, which will be fixed and thoroughly tested prior to any future data center failover.
2. Improve monitoring around failover events. While our monitoring immediately identified issues with the failover, we have identified changes which can provide our team better clarity earlier in the process, and improve our time to resolution for similar issues.
3. Improve resilience and recovery for post-payment integrations. We are refining how we detect and recover from delayed post-payment actions so that any issues are corrected more quickly, reducing how long affected payments remain in a confusing or incomplete state for customers and clients.

‌

We sincerely apologize for the disruption this incident caused to your operations. Ensuring the reliability and availability of TaxSys is our highest priority. We are committed to learning from this event and strengthening our infrastructure and procedures to better protect your services going forward.