01
The situation
An auto-parts shop with about 100,000 products on a four-core server. The database went down three times a day, server load sat between 10 and 25, and listing a part in the admin panel took long enough to stop the team.
No backup outside that server. Years of undocumented changes inside the shop engine. The usual advice was a bigger server.
02
Data first
Before any fix: the database, the code and 253,599 product photos copied from the live server straight to off-site storage, with no maintenance mode and no table locks. The shop kept taking orders during the copy.
The backup counted as done only after it was restored into a working clone and its products, orders and customers matched production.
03
Measure, don't guess
The slow-query log was switched on in the live database without a restart. Search was using 93% of the database: one missing index made every search read the whole word list.
Four of my own first hypotheses were wrong, and the report says so. Measuring on production, not on the much faster clone, is what found the real cause.
04
Small, reversible fixes on the live shop
Two indexes, one keyword in the search query, a cap on PHP workers so the operating system stopped killing the database, and a quotation module that no longer failed the page when its licence server said no.
Each change had a backup and a one-line rollback written before it went in. Indexes went in with a short lock timeout, so a failed attempt blocked nothing.
05
The five-hour incident
Mid-engagement every product page returned an error. A setting changed ten months earlier took effect the first time something cleared the cache. The home page, categories and search kept working; the product pages were back in 5 h 19 min with the cause in writing.
06
What was left, honestly
The shop still runs on PHP 7.4, out of security support since 2022. Upgrading it would wipe 822 undocumented lines in the engine, so they were extracted and saved first. That became the start of the next project: a new platform built next to the old shop instead of a risky upgrade.