Cleaning up messy text: the five operations that solve most of it

Messy text is what you get from a PDF, an email or a spreadsheet: it almost never arrives clean. The failures are boringly consistent, which is good news: five problems cover nearly everything.

1. Hard line breaks from a PDF

The symptom. Every line ends where it ended on the page, so the paragraph will not reflow and looks like poetry.

Why. A PDF has no paragraphs. It has text positioned on a page, and copying reconstructs lines from position. The line break is invented at copy time.

The fix. Replace every single newline with a space, and every double newline (a real paragraph break) with itself. Done in the right order:

1. Replace \n\n with a placeholder such as @@PARA@@.
2. Replace \n with a space.
3. Replace @@PARA@@ with \n\n.

Then look for hyphenated words split across the original lines — informa- tion — and remove those.

2. Non-breaking spaces and other invisible characters

The symptom. Text looks correct and behaves oddly: a search fails, a column will not sort, a value that is clearly a number is treated as text.

Why. Web pages and word processors insert characters that render as a space but are not one — the non-breaking space U+00A0 most often. Zero-width spaces and byte-order marks are rarer and more confusing, because they are invisible entirely.

The fix. Most editors can show invisible characters; turn that on first so you can see what you are dealing with. Then replace U+00A0 with an ordinary space, and strip zero-width characters. In a regex-capable editor, \u00A0 and \u200B match the two common ones.

3. Smart quotes and dashes

The symptom. Code will not run, a CSV parses wrongly, or a search for a word with an apostrophe finds nothing.

Why. Word processors and many messaging apps automatically convert ' to ' and -- to —. Excellent in prose, fatal in anything a machine has to parse.

The fix. Replace the curly forms with straight ones — ' ' to ', " " to ", — and – to - where appropriate. Keep the typographic versions in text meant for humans; remove them from anything meant for a parser.

4. Encoding damage

The symptom. café appears as café, or as caf?, or as caf□.

Why. Text encoded as UTF-8 was read as if it were Latin-1, or the reverse. Each is a distinct signature: é for the first, replacement characters for the second.

The fix. Prevention rather than repair — open the file specifying the correct encoding, and save as UTF-8. Repairing damaged text is possible but lossy, and the original file is almost always still available and still correct.

For CSV specifically, saving from a spreadsheet as "CSV UTF-8" rather than plain CSV avoids most of this.

5. Rich text that will not stop being rich

The symptom. Pasted text arrives with someone else's font, size and colour, and reformatting it never quite works.

The fix. Paste as plain text — Cmd+Shift+V or Ctrl+Shift+V in nearly everything. When an application ignores that, round-trip the text through a plain text editor, which discards all formatting by definition.

Doing it repeatedly

For a one-off, an editor's find-and-replace is enough. When the same cleanup happens weekly, the choice is between a saved sequence and a small script.

The consideration that decides where to do it is privacy rather than convenience: pasting a document into an online text tool sends it to a server, and text is the category where people paste the most sensitive material with the least thought — contracts, credentials, customer lists. A local editor has no such question attached.

Every one of these problems is invisible until it breaks something downstream. Turning on "show invisible characters" once, in whatever editor you use, catches most of them before they do.

What makes text messy?

Messy text is text carrying formatting or characters from wherever it came from — a PDF's line positions, a word processor's typographic substitutions, a web page's invisible spacing. It looks correct and behaves wrongly, which is why the problems are only discovered downstream.

Which problem do you have?

Naming which of the 6 you have is most of the work; each fix is a find-and-replace once you know.

What to turn on once

Enable "show invisible characters" in whatever editor you use. Most of the problems above are invisible by definition, and seeing them is the difference between a two-minute fix and an afternoon of confusion.

Where to do it

For a one-off, your editor's find-and-replace is enough. For a weekly job, a saved sequence or a short script. The consideration that decides where is privacy rather than convenience: text is the category where people paste the most sensitive material with the least thought — contracts, credentials, customer lists — into whichever web tool ranked first. A local editor has no such question attached. See text tools, online tools and free utilities.

The 5 fixes, in order

• Hard line breaks — replace double newlines with a placeholder, single newlines with a space, then restore the placeholder. Then join words hyphenated across the original lines.
• Non-breaking spaces — turn on invisible characters, then replace U+00A0 with a normal space and strip zero-width characters.
• Smart quotes — replace curly quotes and long dashes with straight equivalents in anything a machine will parse.
• Encoding damage — do not repair, re-open. Read the original with the correct encoding and save as UTF-8.
• Rich text — paste as plain text with Cmd+Shift+V or Ctrl+Shift+V, or round-trip through a plain editor.

Messy text is almost always one of those 5. The Unicode character database is where to look when it is not — usually a character that renders as something familiar and is not it.

Every one of these is invisible until it breaks something downstream, which is why the single most useful change is turning on invisible characters once and leaving it on.

Doing it repeatedly

Messy text arriving weekly from the same source deserves a saved sequence rather than a fresh find-and-replace each time. Most editors save a macro; a short script handles the rest.

Where you run it matters more than how. Text is the category where people paste the most sensitive material into whichever tool ranked first — contracts, credentials, customer lists — because a paste feels lighter than a file upload. It is not lighter, and a local editor removes the question entirely. Five problems, five fixes, one setting turned on permanently. Messy text stops being mysterious the moment you can see what is in it.