Please watch this short video from our CEO, Ismael, before reading the rest of this post:
- Duration: ~10 hours
- Customer data lost: None
- Subscriptions affected: No
- Root cause: Infrastructure provider database upgrade failure
- Major improvements already completed: Replica databases, improved failover, enhanced monitoring, upgraded status page with email/SMS alerts
Full technical details: Below you’ll find our complete postmortem, including what happened, where we fell short, what we’ve already changed, and the improvements we’re making going forward.
What happened
On the morning of July 3, at 08:43 UTC, one of our core production databases went offline during a forced, unscheduled upgrade initiated by our infrastructure provider. This database holds critical checkout configuration, including product settings, pricing plans, and checkout setup.
With it offline, new checkout sessions couldn’t complete, so new purchases were unavailable while our team worked to restore service. Customers and vendors also saw errors on login.
A few important points on what was and wasn’t affected:
- No customer data was lost.
- Existing subscription renewals, subscriptions with a fixed number of payments (limited subscriptions), and split payments continued processing throughout.
- Once systems were restored, all services resumed normal operation.
The part we own
The upgrade that triggered this was outside our control, and we’ll cover exactly what happened below. But the length of the outage, and how we communicated during it, are on us.
Two things in particular.
First, our platform wasn’t resilient enough to this specific class of infrastructure failure. A single database upgrade should never have been capable of disrupting checkouts for this long. That’s the architectural weakness we’ve already begun addressing.
Second, our incident communication was too slow. We had a status update prepared early, but we waited on our provider’s resolution estimates instead of telling you clearly and immediately that the issue was ongoing. You should have heard from us sooner and more often. We’ve changed how we handle this, and the new automated tooling we describe below is built specifically so this doesn’t happen again.
How we responded
Our monitoring detected the issue within a minute, and the team was alerted immediately.
Because our provider initially estimated the upgrade would complete in around 15 minutes, our first decision was to wait for it to finish while, as a precaution, starting a full restore from a database snapshot in parallel at around 09:30 UTC. That precaution turned out to be the thing that brought us back online.
As the provider’s estimates slipped from 15 minutes to three hours, and then past that with no resolution, we stopped waiting and committed fully to our own recovery path. Restoring and validating a production database of this size, and replaying transactions to guarantee no data was lost, takes 8-10 hours for 75,000 ThriveCart customers. That restore was completed at 18:32 UTC, at which point we switched live traffic over to the recovered database.
To reduce confusion for buyers and vendors, we also swapped raw error pages for a clear maintenance page during the restoration.
Outage timeline
08:42 UTC. Our infrastructure provider initiates a forced, unscheduled upgrade of our primary production database (MariaDB 10.11.11 to 10.11.16). We had automatic updates disabled and received no advance notice of this upgrade. ThriveCart also operates a code freeze over major holidays, so this was not related to any deployment on our side.
08:43 UTC. The database goes offline. Checkout and vendor/customer login become unavailable, and users begin seeing gateway timeout errors. The team is alerted.
09:00 UTC. This is the point at which we should have sent an “ongoing incident” email to all customers and updated our status page, and we didn’t do so. That was a mistake.
~09:30 UTC. As a precaution, we begin restoring a database snapshot in parallel while waiting for the provider’s upgrade to complete.
10:24 UTC. The provider’s tooling estimates the upgrade should have finished around 09:35 UTC, already past. We learned the upgrade had actually begun at ~08:55 UTC and was delayed while existing database connections drained.
~10:55 UTC. We reach out to our provider’s account manager(s) and support team. They confirm the forced upgrade was triggered by end-of-support for our database version, and revise the estimate to roughly three hours total, projecting completion around 11:43 UTC (or 48 minutes remaining).
11:00 UTC. Based on the provider’s estimate, we prepare a “resolved” email notification. When that timing isn’t met, we update our status page noting the incident as critical.
12:29 UTC. Status page updated: checkout remains unavailable, restoration in progress.
~12:43 UTC. The three-hour mark passes with no resolution. We begin chasing the provider for a renewed estimate.
~12:50 UTC. We confirm the two additional production databases have been proactively upgraded (using a near-zero-downtime switchover technique) to protect them ahead of the holiday period.
13:32 UTC. The provider acknowledges in writing the upgrade has taken far longer than expected, that the system entered a rollback state after the initial restart failed, and escalates internally to their highest severity.
~14:24 UTC. The provider detects that the upgrade had technically failed during the restart phase and initiates a rollback.
16:00 UTC. The provider begins promoting a secondary database host to primary as a recovery step.
~18:32 UTC. Our own snapshot restore (started at ~09:30 UTC) completes. We switch live traffic to the newly restored database. Checkouts and dashboard start operating normally again.
19:32 UTC. After an hour of testing, Engineering confirms the platform is fully operational. Customers updated.
Root cause
The extended duration was not simply slow validation. The provider confirmed in writing that the upgrade workflow failed during the restart phase, which is not normal behaviour, forcing a rollback. They also confirmed they have no record of notifying us that this forced upgrade would take place, despite our having automatic updates disabled.
That explains the trigger. But the lesson we’re taking isn’t that our provider had a bad day. It’s that our own architecture gave a single database too much power to take checkout down, and we’re fixing that.
What we’ve already changed
Every incident is a chance to make the platform more resilient. These changes are already in place:
- Duplicate database replicas across our production environment for additional redundancy.
- A strengthened failover strategy to cut recovery time if a similar event happens again.
- Improved monitoring and alerting around infrastructure maintenance events, so a forced change like this one surfaces immediately.
- A review of our operational processes so future maintenance can be managed more proactively.
What we’re going to change
These changes are being explored:
- Separating checkouts from the main database (so checkouts will process should there be a database outage).
- Multi-provider server back-ups to minimize future downtime and outages.
The ThriveCart Reliability Initiative
Beyond those immediate fixes, we’re launching a long-term Reliability Initiative focused on continuously improving the performance, resilience, and stability of the platform, and on making reliability a visible, ongoing part of how we build and communicate. We’ll regularly share engineering improvements and reliability milestones as part of it.
A new standard for incident communication. The multi-hour gap in updates during this incident is exactly the kind of thing this addresses. We’ve improved our externally hosted Status Page to include automated real-time updates, plus optional automated email, SMS, and RSS notifications, so if we ever face another significant incident, you’ll know what’s happening as it happens.
A Customer Advisory Council. Some of our best product decisions come directly from customers. In the next few weeks, we will be forming a council of merchants, affiliates, and experienced customers who’ll meet regularly with our leadership, product, and engineering teams to shape priorities. We’ll share what we learn and the actions we take as a result.
We expect to expand and iterate on the initiative over time.
Our commitment
Many businesses depend on ThriveCart every day, whether you’re running a live launch or processing recurring revenue. No cloud platform is immune to infrastructure failure, but our job is to respond fast, communicate clearly, and keep strengthening our systems so that a single failure can’t take you offline.
We’re grateful for your patience during this incident, and thankful to our customers for all the support while our team worked to restore our services. Our commitment is unchanged: building a platform you can trust to power your business.
Expect to hear much more from us over the coming weeks and months about the initiatives outlined above, the progress we’re making, and the improvements we’re continuing to invest in. This is just the beginning, and we look forward to building an even more resilient, more transparent, and more customer-focused ThriveCart with you.