When a UTF-8 CSV is valid but the text is still wrong

Tell a useful byte-order mark from saved mojibake, including a mark stuck inside a field.

A valid decode can still be the wrong text

The companion guide on fixing garbled CSV characters starts from bytes that are not valid UTF-8. This page is the next failure, and it is easy to miss because the file opens cleanly. The bytes form legal UTF-8. Auto mode does not throw. Accented-looking marks are already characters in the file. Changing the encoding menu cannot put the original letter back, because that letter is no longer what the file stores.

Two mechanisms get mixed up in the same complaint: “there is a weird character at the start” and “the names look translated.” A byte order mark is a signature at the start of a byte stream. Mojibake is text that was decoded with the wrong encoding and then saved again, so the wrong characters became the new source. A signature can be stripped without changing any field. Saved mojibake is the field.

Unicode assigns U+FEFF two roles. At the very beginning of a text stream it can be a byte order mark, a signature for the encoding form. Anywhere else, the same code point is a zero-width no-break space. The Unicode FAQ on BOMs warns that prefixing every field with a signature wastes space and makes two equal strings compare unequal. Microsoft documents the UTF-8 signature bytes as EF BB BF. Those three bytes are not the three characters you see when a Latin-1 reader has already saved the misread signature.

sku,product_name,marker
NM-14,Café tray,double-encoded
NM-27,Naïve lamp,double-encoded
NM-08,Plain tray,saved-signature
NM-03,Plain tray,code + U+FEFF + join

The first three data rows above match the downloadable UTF-8 sample character for character. The fourth marker is abbreviated on purpose. The sample file does not contain the words “code + U+FEFF + join”. It contains the letters c, o, d, e, then the single character U+FEFF, then j, o, i, n. A proportional preview can make that field look like codejoin. The character is still there.

Two files, two different failures

Keep the files separate in your mind. mojibake-utf8.csv is valid UTF-8 and has no leading signature. Its product names are already damaged. leading-bom-sku.csv is a different teaching file: the same kind of catalog, correct letters, and the three signature bytes EF BB BF before the header. It is not a repaired copy of the first file. TableMender cannot turn the damaged names into the correct names. Nothing in the byte stream of the first file still encodes U+00E9, the letter é in Café.

Here is the usual path into the damaged file. The exporter wrote Café with UTF-8 bytes C3 A9 for é. Some other program read those two bytes as Windows-1252. Byte C3 is U+00C3, Latin capital A with tilde. Byte A9 is U+00A9, the copyright sign, not an e. The program then saved that text as UTF-8. The new file is well-formed Unicode. It simply says something else. The same path turns Naïve into N, a, U+00C3, U+00AF (a macron), v, e. People describe both results as “é” or “ï” because the glyphs sit close together. The code points are the check, not the nickname.

The third row is the signature after it has been saved as characters. Bytes EF BB BF, read as Windows-1252 and saved as text, are U+00EF, U+00BB and U+00BF: the letters people write as . They are ordinary visible characters. They are not a byte order mark anymore. Stripping a leading EF BB BF from some other file will not remove them.

Read the damaged characters as code points

For NM-14 in the damaged file, the product name is not “Café tray”. It is C, a, f, U+00C3, U+00A9, space, t, r, a, y. For NM-27 it is N, a, U+00C3, U+00AF, v, e, space, l, a, m, p. For NM-08 the name begins with U+00EF U+00BB U+00BF and then “Plain tray”. For NM-03 the marker has nine code points and the fifth is U+FEFF.

If you instead open that same damaged file with Windows-1252, the tool will decode it. Windows-1252 maps almost every byte, so “not valid” is not the error you get. You get a second layer of damage. NM-14’s name becomes C, a, f, U+00C3, U+0192, U+00C2, U+00A9, then “ tray”. That is not Café either. Switching encodings until the preview “looks less wrong” is how the second layer gets saved. Stop when the preview matches a source you trust, not when the symbols look decorative.

The leading-signature file is the contrast. After a strict UTF-8 decode, one leading U+FEFF is removed and the header field is exactly sku. The names are Café tray (with U+00E9) and Naïve lamp (with U+00EF). The status line still says utf-8. This version does not print a separate “UTF-8 BOM found” flag. A quiet header is consistent with a stripped signature, and it is also consistent with a file that never had one. Only the bytes, or a hex view outside the tool, distinguish those cases.

Steps in CSV check and repair

1. Keep an unchanged copy of the original export. Do not re-save it from a spreadsheet first. A spreadsheet save can add a signature, drop one, or freeze mojibake into a new UTF-8 file.
2. Open CSV check and repair. Use the file picker for both samples. Pasted text has already been decoded by the browser, so it cannot prove which bytes were on disk.
3. Load mojibake-utf8.csv with Auto, Comma and the header on. Preview four data records and three columns. Compare the code points above. Do not download yet.
4. Change the encoding to Windows-1252 and preview again. Confirm the names get longer and still are not Café or Naïve. Switch back to Auto. The original bytes are still the file you selected; the menu does not edit the disk file.
5. Clear the input. Load leading-bom-sku.csv with Auto. Confirm two data records, the header sku with no extra character, and the correct accented names.
6. Download UTF-8 only after the preview matches the source system. The CSV BOM checkbox is on by default because a signature often helps current Excel recognize UTF-8. Leave it on for Excel, and turn it off when the receiver treats EF BB BF as data. Adding that signature does not repair NM-14.

Inspect encoding without guessing →

Common errors compared

Edge cases that look similar

A leading signature is removed once by decoding and, if a leftover U+FEFF is still the first character of the text, the parser removes one leading U+FEFF as well. A signature after a blank line, inside quotes, or in the middle of a header is data. NM-03 is that case: U+FEFF sits between code and join, so both the decoder and the parser keep it. Replacement with a normal space would be an edit. This tool does not make that edit.

UTF-16 is a different signature. Auto mode checks the first two bytes for FF FE or FE FF and then decodes UTF-16 little-endian or big-endian. A UTF-8 file that merely contains the characters  will not take that path. Do not rename a UTF-8 file to force UTF-16.

Some exporters write a signature and also repeat U+FEFF inside the first field because an earlier tool stored the signature as text and a later tool added a real one. Stripping the leading bytes still leaves the inner character. Look at the first field’s length after a successful decode.

If the file already contains U+FFFD, the replacement character, an earlier program discarded the original bytes. No encoding on this menu reconstructs them. The same is true when the mojibake was saved years ago and the pre-damage export is gone. You need that earlier file, or the source system, not a cleverer selector.

Very large files do not change the rule. The limits are 5 MiB, 20,000 records including the header, 256 columns and 250,000 cells. A truncated download can look like an encoding error because the last quoted field never closes. That is a broken file, not mojibake. The quoting error names a record, a physical line and a field.

What this tool changes, and what it leaves alone

CSV check and repair decodes the bytes you selected and can write a new UTF-8 file with or without a leading signature. It can also change the delimiter. It does not search for “Ô and replace it with the letter you hoped for. A global replace would also hit a legitimate Ã, a copyright sign that belongs in a legal line, or the macron in a correctly stored name. Those letters exist in real words.

The text-preserving CSV to Excel workbook will store the damaged names as text exactly as previewed. That is preservation, not a repair. Choose it when you need the wrong characters to stay stable while you compare them with a source system. It will not detect that U+00C3 U+00A9 “should have been” é.

Excel’s own Open behavior is a separate program. Microsoft documents that a UTF-8 CSV with a signature often opens with the intended characters, and that import tools are the fallback. Interface names differ by Excel version. TableMender does not remote-control Excel, and a workbook Excel has already saved is a new file with new bytes.

Limits of an encoding selector

Auto mode is a strict UTF-8 decode after a short check for a UTF-16 signature. It is not a detector for Windows-1252, Shift JIS, GB18030, or “whatever the spreadsheet used last Tuesday.” Those encodings appear in the menu because you can ask for them. A preview that is valid under two encodings is ambiguous. Prefer the encoding the exporter documents.

The samples use invented catalog rows. They are not customer extracts, and they are not a claim that every mojibake string on the web follows C3 A9. Other letters produce other pairs. The method is the same: identify the code points, compare them with a known-good row, and do not save a second misread.

This page does not prove that your receiving system will honor a signature. Some loaders strip EF BB BF, some keep it as U+FEFF on the first header, and some reject the file. Download once, open that download in the receiver, and compare sku, the accented names, and the record count. If the first header gains a character only in the receiver, the receiver kept the signature as data. Turn the BOM checkbox off and export again. If the names were already wrong in the TableMender preview, the receiver is not the cause.

← All guides