Skip to content
DevLab

Encoding Tests

Free CSV test files in UTF-8 without BOM, UTF-8 with BOM, Shift_JIS and CP932, containing Japanese data — for testing mojibake and CSV import.

A CSV test file encoded in UTF-8 without BOM

utf8.csv / 659 B
A CSV test file encoded in UTF-8 with BOM

utf8-bom.csv / 662 B
A CSV test file encoded in Shift_JIS

sjis.csv / 514 B
A CSV test file encoded in CP932, including vendor-specific characters

cp932.csv / 518 B

Why character encoding tests matter

CSV files containing Japanese (or other non-ASCII) text often suffer mojibake due to encoding mismatches. Excel, for example, prefers UTF-8 with BOM, and tool support varies widely.

Use these test files to verify that your CSV importers and text-processing libraries handle each encoding correctly.

Key characteristics of common encodings

📖 Where people get stuck

The same content written out in UTF-8 with and without a BOM, Shift_JIS, CP932 and more, so one set tells you which of them your reader handles. Note that "it did not garble" is not the same as "it was detected". Guessing right by accident happens.

Case What happens What to do
Opening it in Excel garbles the text Excel reads a UTF-8 CSV with no BOM as the OS locale encoding — CP932 on Japanese Windows. The file is fine; the reading is wrong. Write CSVs destined for Excel as UTF-8 with BOM (EF BB BF). Try both variants from this list and see which one garbles.
You meant Shift_JIS and it was CP932 CP932 is the Microsoft extension of Shift_JIS with vendor characters such as ①, ㈱ and Ⅲ added. A strict Shift_JIS decoder throws when it meets them. Treat CSVs from Japanese Windows as CP932 in practice, always, and name that codec explicitly — SJIS-win in PHP, cp932 in Python.
Auto-detection got a short line wrong mb_detect_encoding and chardet are statistical guesses. On a line of a few dozen bytes, or one that is pure ASCII, there is nothing to decide from. Where you can, let the user choose instead of guessing. For text that is already garbled, listing candidates in the mojibake fixer and picking one is the surer route.

Text merely read with the wrong encoding can be recovered; text saved again in that state cannot. Once characters have become U+FFFD or ?, the original bytes are gone. Keeping the raw data somewhere before conversion is what makes an import recoverable.