Skip to main content
Back to blog
Engineering9 min read

Black Friday: a data architecture that can handle the load

Massive price updates, stock levels changing by the minute, channels limiting throughput: what goes wrong during sales peaks and how to prepare for them in advance.

Pixee PIM Team · November 15, 2026

With about ten days to go before a peak in sales, nobody is overhauling their architecture. The question is no longer what should have been built, but what needs to be checked now and what we are prepared to postpone. Yet the breaking point is rarely where we’re monitoring it. It’s almost never the shop’s server that crashes; it’s the catalogue update chain — prices being changed en masse, stock levels shifting by the minute, channels hitting their write-through limits, queues growing faster than they can be cleared.

What really breaks things: reading is not the same as writing

The read load is the best-handled aspect of e-commerce: cached pages, content served by a CDN, read replicas, load testing. It is elastic, under your control, and you know how to measure it.

The write load on the catalogue side has none of these properties. It cannot be cached — a stock update must reach its destination, or it serves no purpose. It is subject to quotas imposed by third-party systems over which you have no control. And it is rarely monitored with the same care as incoming traffic.

The time lag makes it virtually invisible. The update peak precedes the sales peak.On the night before the shop opens, the entire promotional catalogue changes all at once: discounted prices, crossed-out prices, dedicated visuals, availability updates across all channels. Traffic hasn’t started yet, the dashboards are quiet, and yet this is when the queue begins to fill up. The second peak—that of stock reductions—arrives a few hours later, on a channel that hasn’t yet finished processing the first.

The synchronisation architecture under normal conditions is described in our article onmulti-channel cataloguesynchronisation. What follows focuses on this same chain when it is under pressure: it is not the design that changes, but the margins that disappear.

The four breaking points

1. Bulk price updates

A promotional price fed up from the ERP affects a significant portion of the catalogue in a single operation, and each modified line triggers a publication per channel: the volume of outgoing updates is the number of lines multiplied by the number of destinations. Added to this are the knock-on effects — recalculation of crossed-out reference prices, tiered discounts, negotiated B2B rates, and prices by currency.

The risk lies not only in the time taken, but also in the import being partially applied : a price import interrupted halfway through leaves the catalogue in two states simultaneously, and the recovery must be carried out manually, under pressure. A price import must be transactional at batch level and reproducible exactly.

2. Stock propagation

The profile is exactly the opposite: a vast number of tiny entries whose value plummets within minutes. Sending an update for every unit sold clogs up the pipeline for no reason. Two mechanisms are required: a trigger threshold — we publish when the change exceeds a given increment or when a lower threshold is crossed — and queue compaction, where only the latest known value for a product reference is actually sent.

Practical implication: during a peak, we do not chase latency; we absorb it using a buffer. A safety stock for SKUs with low order depth costs a marginal loss of revenue; an oversell costs a cancellation and a degraded seller rating.

3. Channel throughput limits

Each destination imposes its own limits: requests per second, hourly or daily quotas, and sometimes distinct limits depending on the type of object being written. Exceeding these limits results in a 429, and the wrong response — retrying immediately, increasing parallelism — prolongs the blockage rather than shortening it. The correct response is an increasing delay with an element of randomness, adherence to the headerRetry-After when the channel provides one, and a queue that holds the work in the queue.

The least anticipated pitfall: the quota is linked to the seller’s account, not to the integration. PIM, repricing tools, ERP and manual back-office processes all share the same quota — a team manually correcting product records during the event uses up the call budget for stock synchronisation.

A publishing pipeline must therefore self-limit before it is restricted. Pixee PIM applies 23 levels of throughput limiting depending on the nature of the operation, across 530 API entry points: costly calls do not share the budget with critical updates. The principle applies regardless of the tool — throughput is regulated by flow type, not globally. The destinations supported natively are listed on the integrationspage; a connector such as Mirakl adds a nuance, as the limit depends on each marketplace operator rather than on the underlying technology.

4. Batch processes that exceed their time window

A scheduled job is sized for a daily volume of modified rows. When a significant portion of the catalogue changes overnight, the same task takes longer and may exceed its time window. If nothing prevents it, the next execution starts whilst the previous one is still running: duplicate work, contention on the same rows, and a queue that doubles in size instead of emptying.

Three safeguards are sufficient: a single execution lock per task, division into bounded batches with resumption from the last processed batch, and strict separation of queues. Pixee PIM distributes its scheduling across 22 scheduled tasks, executed by 12 workers on 14 separate queues, to achieve this isolation: a large media export does not tie up the workers assigned to stock management. The distribution across 8 PostgreSQL databases follows the same logic on the persistence side.

What needs to be prepared in advance

The timeline below outlines the full preparation process. Around ten days before the event, only the last two lines remain actionable; the first few apply to next year and are worth noting now, whilst the context is still fresh.

DeadlineActionReason
30 days before the eventFreeze on structural changesNo new attributes, no tree structure reorganisation, no new channels. A change to the structure invalidates the mappings and requires a complete republish.
D-21Preparation of promotional prices with activation datePrices entered, checked and published in advance with an effective date. The launch becomes a simple date rollover, rather than a mass import at the busiest time.
D-14Checking quotas with the channelsCompile a list of limits, identify integrations sharing an account, request temporary increases — several channels grant these with advance notice, not as a matter of urgency.
D-7Warm up queues and clear errorsResend a representative volume of posts outside peak hours, then clear the error queue. An error queue that’s already full before the event is a guaranteed blind spot.
D-2Freeze non-essential content and rehearse the failover planNo more supplier imports or recategorisation. On-call rota, access rights and the procedure for pausing a feed must be known to more than one person.

Setting a promotional price in advance requires knowing your market position before the event, not discovering it on the day itself — that is the role of monitoring competitors’ prices carried out in advance. Adjustments decided during the peak are the most costly: they trigger mass reposts at the worst possible moment, and are made without hindsight.

During the event: prioritise, downgrade, defer

In a high-pressure situation, not everything can go out at the same time: the issue is not about going faster, but about explicitly choosing the order. One criterion is enough to make the call: if a delay in this data could result in an order that cannot be fulfilled, it is a priority; otherwise, it can be deferred.

  • High priority— stock levels, closing an offer on an out-of-stock item, actual selling price. The only data where obsolescence costs money by the minute.
  • Normal priority— creation of new offers, delivery times, corrections to attributes that block a sales channel.
  • Deferrable— descriptions, titles, marketing attributes, media, translations, recategorisations, supplier imports. None of these causes overselling simply because they are published three days later.
  • Suspendable— automatic enrichment, completeness recalculations, image variants, analytics exports, full re-indexing. Pausing these is the quickest way to free up capacity for stock propagation.

Graceful degradation involves deciding all this beforehand, not whilst it’s happening: queues separated by criticality and not just by channel, a switch per processing, and a procedure for pausing consumption from a queue without emptying it — a purged stream loses updates, a paused stream replays them.

Finally, the on-call person does not look at the same screens as the web infrastructure team. Three metrics are sufficient: the depth of each queue and its trend, the 429 and 5xx per channel, and the error queue. A queue depth that rises steadily, even slowly, indicates an outgoing throughput lower than the incoming throughput: this will not correct itself.

Next: what to measure

A peak is the only time of year when the system reveals its true limits. Record these measurements in the days that follow, whilst the logs are still available — not in January.

  • Maximum queue depth per channel, and the time it was reached— the measurement that determines capacity for the following year. The time indicates whether the point of strain was the launch of promotions or the sale itself.
  • Propagation delay, observed at peak values— the average tells us nothing; it is the slowest cases that cause incidents. To be recorded by data type and by channel.
  • 429 response volume by channel and by hour — the factual basis for requesting a quota increase before the next event, and the means of identifying the channel that is actually the limiting factor.
  • Over-sales and cancellations, compared against observed lead times— enough to turn a commercial incident into a technical adjustment: trigger threshold to be reviewed, safety buffer to be adjusted.
  • Actual duration of scheduled processes compared to their time slots— the remaining margin indicates which will be affected next year if volume increases further.

Frequently asked questions

Should the catalogue be completely frozen during the event?

No: prices and stock levels must continue to be updated. The freeze applies to the structure— data model, tree structure, flow mappings, adding channels, major updates — and to non-urgent content. It is these changes that trigger full republishing, and are therefore the most unpredictable in terms of cost.

How much of a delay in stock updates is acceptable?

There is no universal figure: the setting depends on the depth of stock and the turnover rate for each product family. An item available in large quantities can tolerate a significant delay without risk; an item with only a few units in stock is not protected by latency but by a safety buffer and the offer being closed when stock falls below a low threshold. Measure the actual delay in your data feeds, then set the buffers accordingly.

What should you do if a channel continuously returns 429 errors?

Deliberately slow down the process, in this order: reduce outbound concurrency on that channel, adhere to the delay specified in the response, enable batching, then suspend any deferrable flows to that destination. Retrying more quickly or opening additional connections prolongs the block and, on some channels, triggers a longer restriction.

Is the PIM sufficient, or does the ERP also need to be enhanced?

A chain is only as strong as its weakest link, and this link is frequently upstream: price retrieval, calculation of multi-warehouse available stock, and the number of simultaneous connections permitted by the management system. A PIM that absorbs the peak is of no use if it receives its data several hours late. Check the ERP’s export window and how it behaves when the number of modified lines changes by an order of magnitude.

A publishing chain that absorbs peaks

Queues separated by criticality, rate limiting per operation type, error recovery and a publication log: the architecture is the same all year, it just shows on the busiest days.

See the plans

Stay up to date

Receive our PIM insights every month

Best practices, compliance updates, integration guides — directly in your inbox.

Subscribe for free

30-day trial · No credit card

Ready to structure your product catalogue?

Try Pixee PIM for 30 days — up to 2,000 products, no commitment.