UTF-8, CRLF vs LF: A Practical Guide to Text Encoding

Garbled characters, files that show one giant line, diffs where “nothing changed” but everything did — almost all of it comes down to text encoding and line endings. These are invisible by design, which is exactly why they’re so confusing when they break. Here’s what’s actually going on, in plain English, and how to stop it from happening.

Encoding: what UTF-8 really means

A text file is just a sequence of bytes. Encoding is the agreement about which bytes mean which characters. Decades ago, everyone used different, incompatible encodings, which is why old files sometimes turn to gibberish today.

UTF-8 is the modern standard — the agreement the whole web now shares. It can represent every character, from English to Japanese to emoji, and it’s backward-compatible with old ASCII, so plain English text looks identical. If you standardize on one thing and never think about it again, make it UTF-8.

When you see mojibake — é where é should be, or æ–‡å— where Japanese should be — it almost always means the file was written in one encoding and read in another. The bytes are fine; the interpretation is wrong. The fix is to open the file with the correct encoding, then save it as UTF-8.

A word on the BOM

You may run into the “byte order mark,” a few invisible bytes some tools add to the start of a UTF-8 file. Most modern tools handle it silently, but it occasionally causes odd bugs — a stray character before <?php, or a script that won’t run. When in doubt, save as “UTF-8 without BOM” unless something specifically requires it.

Line endings: CRLF vs LF

Every line in a text file ends with an invisible marker. There are two common ones:

• LF (\n) — used by macOS and Linux.
• CRLF (\r\n) — used by Windows.

They both mean “start a new line” — they just use different bytes to say it. Neither is wrong; they’re conventions from different operating systems that now have to coexist.

Why the difference bites you

• A file with LF endings can appear as one long line in older tools that only expect CRLF.
• Version control can flag an entire file as changed when only the line endings were converted — turning a one-line edit into an unreadable diff.
• Shell scripts can fail with cryptic errors when a shebang line (#!/bin/sh) carries a stray \r from being saved on Windows.
• Copy-pasting between apps can silently mix both styles in one file, which some tools handle badly.

None of this is your logic being wrong — it’s an invisible byte mismatch masquerading as a “weird bug.”

How to stay out of trouble

• Pick UTF-8 for everything and don’t look back.
• Be consistent with line endings within a project. On cross-platform teams, the common convention is to agree on LF and configure your tools to keep it.
• Configure version control to normalize line endings so they stop polluting diffs (many teams use a .gitattributes rule for this).
• Use an editor that shows and converts both. Seeing the current encoding and line ending in the status bar — and switching them with a click — turns a mystery into a two-second fix.

Editure reads and writes UTF-8 and detects CRLF/LF automatically, so files behave the same whether they came from Windows or macOS. When you need to convert, it’s a visible, deliberate choice rather than a silent surprise.

A quick troubleshooting checklist

When text looks wrong, run through this in order:

1. Is it mojibake? Wrong encoding on read. Reopen as UTF-8 (or the file’s original encoding), then save as UTF-8.
2. Is the whole file “changed” in your diff? Line endings were converted. Normalize them and commit once.
3. Is everything on one line? LF endings in a tool expecting CRLF. Open it in an editor that understands both.
4. Does a script fail mysteriously? Check for stray \r. Convert the file to LF.

Why this still matters in 2026

You’d think encoding would be a solved, invisible problem by now, and for greenfield projects it mostly is — everything defaults to UTF-8. But the real world is full of files from elsewhere: a CSV exported by a decades-old system, a document from a colleague on a different OS, a log from a legacy server, data scraped from a page with a mislabeled charset. The moment text crosses a boundary between tools, encoding and line endings can drift. Knowing the two variables turns those inevitable encounters from a lost hour into a quick fix.

Encoding and the wider world

A few practical notes that come up often:

• CSV files are a classic trouble spot, especially with non-English characters. If names or symbols arrive garbled, it’s almost always an encoding mismatch on import or export — set both ends to UTF-8.
• Web pages declare their encoding in a header or meta tag; if that declaration lies, browsers guess, and you get mojibake. UTF-8 is the safe declaration.
• Databases have their own encoding settings; a UTF-8 app talking to a non-UTF-8 table produces the same garbling one layer down.

The pattern is always the same: something wrote bytes with one agreement and something else read them with another.

A mental model that sticks

Picture two people exchanging a sealed letter. Encoding is the language it’s written in; line endings are how they mark the end of each sentence. If the reader assumes the wrong language, every word looks like nonsense (mojibake). If they expect a different end-of-sentence mark, the whole letter reads as one endless run-on (the “single giant line” problem). Get both agreements to match — UTF-8, and a consistent line ending — and the letter reads exactly as written. That’s the entire game.

Manage line endings with Git

Teams should record line-ending policy in the repository instead of relying on each person’s editor. A .gitattributes file can declare text handling and exceptions, for example:

* text=auto
*.sh text eol=lf
*.bat text eol=crlf

Agree on the policy before normalizing an existing repository. Changing every line ending at once can make every line appear modified, obscuring meaningful history and creating conflicts with active branches. Put normalization in a dedicated commit, ask everyone to finish or save ongoing work, and inspect the diff before merging.

Git configuration and .gitattributes serve different scopes. Personal settings affect a developer’s checkout, while repository attributes travel with the project. For a shared project, keep the essential rule in the repository and use personal settings only as compatible defaults.

CSV files and older Japanese encodings

CSV is especially troublesome because its consumer may expect a locale-specific encoding. A file that displays correctly as UTF-8 in an editor may open incorrectly in older spreadsheet software, while a Shift_JIS export may corrupt characters that encoding cannot represent. Confirm the required encoding, delimiter, quotation rules, and line endings with the importing system rather than guessing from the .csv extension.

Do not overwrite the only copy when converting a legacy Japanese file. Keep the original bytes, convert a copy, and check names, symbols, circled numbers, wave dashes, and other characters prone to incompatibility. If a replacement character such as � has already appeared, saving again usually cannot reconstruct the lost byte sequence; return to the original.

A safe conversion procedure

1. Make a byte-for-byte copy of the original
2. Identify the expected encoding from the producing system or specification
3. Open it by explicitly selecting that encoding
4. Inspect representative Japanese text and symbols
5. Save to a new file in the target encoding and line-ending style
6. Reopen that file and test it in the destination application
7. Compare record and line counts before replacing anything

Encoding conversion and line-ending conversion are separate operations. When diagnosing a problem, change one at a time so you know which solved it. For source code, run tests; for data, validate row counts and required fields.

What to inspect in your editor

A dependable editor should visibly report the current encoding and line ending, allow reopening with a specified encoding without immediately saving, and distinguish “reopen” from “convert and save.” It should also preserve an existing format unless you deliberately change it. Before mass conversion, verify the target shown in the save dialog or status bar.

Character encoding on the web and in HTTP

Web pages normally use UTF-8, but the declaration must agree at every layer. The file bytes, HTML <meta charset="utf-8">, and HTTP Content-Type header should not contradict one another. HTTP headers generally take precedence, so changing only the HTML may not repair a server that declares another charset.

JSON is conventionally exchanged as UTF-8. For APIs, inspect raw response headers and bytes rather than trusting a browser’s corrected display. If form submissions or downloads fail only for Japanese input, trace the encoding at input, storage, response generation, and client decoding.

Information that speeds up troubleshooting

When reporting corrupted text, record the original file, expected and detected encodings, whether a BOM exists, line-ending style, application and version, operating system, and the exact step that first produced the corruption. Include a small nonconfidential sample or hexadecimal bytes when possible. “Japanese is garbled” is hard to reproduce; “a UTF-8 file without BOM is interpreted as Shift_JIS on import” gives investigators a concrete boundary.

Frequently asked questions

Should every UTF-8 file include a BOM? No. Use it only where the consuming tool requires or benefits from it. Many modern code and web workflows prefer UTF-8 without BOM.

Does changing CRLF to LF alter the visible text? Usually not, but it changes the bytes and can affect scripts, checksums, diffs, and tools with strict expectations.

Can garbled text always be repaired by choosing another encoding? Only if the original bytes remain intact. Once incorrectly decoded text is saved and replaces unrepresentable characters, information may be lost.

The bottom line

Encoding problems feel like dark magic until you know the two variables involved: which encoding, and which line ending. Standardize on UTF-8, stay consistent with line endings, configure your tools to respect both, and use an editor that surfaces them — and mojibake becomes a memory instead of a recurring mystery.