Multi-vendor standardisation: 5 challenges and how to overcome them

The supplier file has arrived; it is complete and opens without any errors. This is where the work really begins. Twenty suppliers means twenty different ways of writing a weight, twenty different sets of attribute terms and twenty different levels of text quality. This article reviews the five standardisation challenges that recur at all retailers, and how to address them so that the result is reproducible from one week to the next.

A clarification of scope: we are not discussing here how to retrieve the files. FTP, SFTP, PRICAT EDI and the scheduling of data collections are the subject of a separate article, ‘Automating your supplier imports via FTP and EDI’. Here, the transfer has already been completed. Everything that follows takes place between reading the first line and writing to your product repository.

Challenge 1 — Units and value formats

This is the most common and most costly challenge. One supplier expresses lengths in millimetres, another in centimetres, and a third in inches: each exports in the unit used by their own system. The problem arises when all three files end up in the same column.

• Lengths— 45 can mean either 45 mm or 45 cm. Without a unit specified in the file, the only reliable source is the supplier’s specification sheet, which should be kept alongside the import rule.
• Gross weight and net weight— two distinct concepts that many catalogues merge into a single column POIDS. Gross weight includes packaging; net weight does not.
• The decimal separator— 1,250 is 1.25 kg in a French export and 1250 in an Anglo-Saxon export. A single column may contain both conventions if the supplier has aggregated several sources themselves.
• Currencies— a price without an explicit currency is unusable: store it next to the amount, along with the date the price list was applied.
• VAT— excludingVATor including VAT, at what rate, with or without an eco-levy. A price list received as ‘including VAT’ and imported as ‘excluding VAT’ shifts your entire margin without triggering the slightest technical error.

The consequences are evident in delivery charges, calculated by actual weight and volumetric weight — a dimension interpreted in millimetres instead of centimetres changes the price bracket — and in published product listings: a television advertised as having a 55 mm diagonal screen triggers product returns and marketplace rejections.

The solution is to never convert implicitly. Every numeric field in your target template has a single standard unit — lengths in millimetres, weights in grams, prices excluding VAT in a base currency — and every connector specifies the source unit to be converted. A value with an unknown unit is not accepted: it is flagged as an anomaly.

Challenge 2 — Encoding and text integrity

The second challenge doesn’t break anything: it just makes a mess. The lines are imported without error, and six months later you discover hundreds of listings published with broken accents.

• UTF-8 versus Latin-1— a Windows-1252 file read as UTF-8 produces the infamous é and °. The worst-case scenario is double transcoding: the file has been opened and saved once, and the original value can no longer be recovered.
• Unwanted spaces— non-breaking spaces, spaces at the end of a value, tabs within a label. Two visually identical labels no longer appear next to each other, and your facets become duplicated.
• All-caps text— PERCEUSE SANS FIL 18V COFFRET is readable in an ERP system, but not on a product page. Set up a line-breaking rule with a dictionary of exceptions for brands and acronyms.
• Escaped HTML— you receive <p>Texte</p> instead of actual tags, or the reverse: raw HTML with inline styles that break your site’s styling guidelines. Decide whether your description field stores rich text or plain text, and strip out style attributes on input.
• Barcodes corrupted by spreadsheets— an EAN entered into Excel without text formatting returns as 3,60421E+12, truncated and irrecoverable in its original form. The topic is covered in detail in our article on the automatic validation of GS1 barcodes.

In practical terms, here is the process of converting a raw line into a clean line.

Challenge 3 — Missing or differently named attributes

One supplier calls REF_FAB what the next one calls MPN and the third Code constructeur. The mapping brings these columns into line with a single target attribute model, which is your own and remains unchanged when a supplier changes their export.

Establish the target model before mapping

Order matters. First, you define the attributes your catalogue must contain, family by family, then you link each supplier to it. Building the model as and when files are received results in a repository with two hundred attributes, half of which are synonyms.

Reconciling list values

Mapping does not stop at column names. For a list-type attribute, Noir, NOIR, black and RAL 9005 denote the same thing and must map to the same entry. The key rule is: any unknown value is flagged; it is never created silently. Without this safeguard, your navigation filters become unusable.

What to do with missing data

For each product family, declare the attributes without which a product cannot be sent to a sales channel — a drill without a power supply is not a record that can be published. You then have an objective criterion for choosing between three responses.

• Rejection— the record is not accepted. To be used only for fields that render the record unusable: ID, description, purchase price.
• Default value— useful when the value is predictable. Never used for a field involving a binding commitment, such as weight or a warranty.
• Enrichment— the data is missing from this supplier but exists elsewhere, with another supplier of the same product or in a reference database. The product is entered with an ‘incomplete’ status and theenrichmentmodule completes the attributes before publication.

Challenge 4 — Duplicates

As soon as two suppliers list the same brand, you receive the same product twice . Without reconciliation, your consolidated stock figures are incorrect and two competing product listings from your own company end up online. Reconciliation is carried out in a cascade, from the most reliable to the least certain.

• By barcode— this is the only identifier genuinely shared amongst stakeholders. However, it must be valid: a match based on an EAN with an incorrect checksum creates nonsensical merges. Validation always precedes the match.
• By brand and manufacturer’s part number (MPN) — when the EAN is missing, the brand + MPN pair is the best alternative, provided both terms are standardised before comparison: hyphens, spaces and suffixes vary from one source to another.
• By similarity, as a last resort— similar product names and technical attributes . This level never triggers automatic merging: it feeds into a queue of matches awaiting confirmation.

Beware of the most common false positive: batches and packaging. A bottle sold individually and the same bottle sold in a box of twelve share the brand and often a reference root, but they are two distinct items. Packaging is therefore a key criterion for matching, on a par with the reference number.

It remains to be decided which value takes precedence once the merge has been confirmed: a source priority defined field by field — the manufacturer is the authoritative source for the technical data sheet, the most recent source for the purchase price — rather than the most recent entry overriding everything else.

Challenge 5 — Partial updates

The fifth challenge is the most dangerous: it destroys data rather than adding incorrect data. Some suppliers send their complete catalogue, others only the modified lines, and others an extract by product range. Treating an extract as a complete catalogue wipes out half your database.

• Differential by default— a file updates only the SKUs it contains and leaves the others untouched. This is the safe approach as long as the supplier has not confirmed that their export is exhaustive.
• Full replacement, subject to conditions— reserved for feeds declared to be comprehensive and subject to a variation safeguard: if today’s file contains significantly fewer lines than the previous one, the import is suspended and requires human validation rather than propagating an incomplete set.
• Reference removals— the correct signal is an explicit deletion row. Failing that, a reference absent from several consecutive comprehensive imports is marked as end-of-life. In both cases, the status is updated; the record is not deleted: the order history depends on it.
• Fields controlled by you— a revised description or a visual chosen by your teams must not be overwritten during the next import. Mark these fields as locked; otherwise, you will have to redo the same work every cycle.

Making the whole process repeatable

These five challenges can be tackled initially with a bit of patience. What matters in the long run is being able to repeat the same process the following month, when the person who set it up is on holiday. Three elements are all that’s needed.

Versioned standardisation rules

Unit conversions, value tables, source priorities and field locks are configuration settings, not disposable scripts in a shared folder. Every change must be dated, attributed and reversible: on the day a price changes, the first useful question is to find out which rule has changed since the last correct import.

A simulation before the actual import

No file should go into production without a dry run. The simulation applies the rules and produces a report of what would happen — new records, updates, rejections, values that vary abnormally — without writing anything to production. This is the only time when making corrections takes minutes rather than days.

An error log readable by non-technical staff

A log that simply displays a parser exception is of no use to anyone in a catalogue team. A usable log specifies the row, the reference, the faulty field, the value received and the expected action; it can be filtered, exported and replayed once the corrections have been made. A manager can then resolve these issues themselves rather than opening a ticket. This is the approach adopted in the import and data quality modules of Pixee PIM: visual mapping, simulation, line-by-line log, and rules maintained by the supplier.

Frequently asked questions

Should the supplier’s raw value be retained after normalisation?

Yes, without fail. Keeping the received value alongside the standardised value allows you to rerun an import if a rule turns out to be incorrect, and to respond to a supplier who disputes a piece of data. Without this history, you would have to request the file again, which often no longer exists.

Should the entire file be rejected if a few rows are invalid?

No, in the vast majority of cases. Rejection must be granular: compliant rows are accepted, whilst the others are flagged as anomalies with the reason stated. Rejecting the entire file is only justified if the file itself is suspect — missing columns, unreadable encoding, or an abnormally low volume.

What should you do when a supplier changes their headers without warning?

Validate the structure before the content: the import checks that the expected columns are present and stops if they aren’t, rather than mapping by position and writing prices into a weight field. Case-insensitive column-name matching accommodates cosmetic variations without masking genuine changes in format.

How long does it take to get a new supplier up to speed?

According to our internal estimates, most of the work is concentrated on the first file: understanding the units, writing the mapping, and reconciling list values. Subsequent imports require only monitoring of the error log, and the determining factor is not the volume of SKUs but the number of attributes to be mapped. Pixee PIM’s 30-day free trial, limited to 2,000 products and 2 suppliers, is sufficient to set up an initial data feed; details of volumes per plan can be found on thepricingpage .