Invisible Unicode Characters Explained

Some Unicode characters have a code point, count toward the length of a string and take part in every search and comparison, yet draw nothing on screen. Most of these invisible Unicode characters have a job: marking where a line may wrap, identifying a file’s encoding, controlling how letters join, or setting the direction of mixed left-to-right and right-to-left text. They cause trouble when they ride along with copied text into a place that expects only what you can see. The usual signs are a search that misses a word you can read, two identical-looking strings that compare as different, a character count that is too high, or a pasted URL, email address or identifier that is rejected.

Whether you can delete one depends on which one it is. Zero-width spaces, byte order marks, soft hyphens and stray control characters are almost always safe to strip from copied text. Zero width joiners, zero width non-joiners and bidirectional controls are not: they change how emoji and several writing systems are displayed.

Quick reference

Character	Code point	What it is for	Safe to remove?
Zero width space	U+200B	Allows a line break without showing a space	Yes, in most copied text
Byte order mark (zero width no-break space)	U+FEFF	Encoding signature at the start of a file	Yes, in pasted text
Soft hyphen	U+00AD	Marks where a word may be hyphenated at a line end	Yes, unless the text is going into a typeset layout
Word joiner	U+2060	Prevents a line break between two characters	Yes, in most copied text
Zero width joiner (ZWJ)	U+200D	Builds emoji sequences; requests joined letter forms	No, unless you are sure it is stray
Zero width non-joiner (ZWNJ)	U+200C	Keeps two letters from joining; part of normal spelling in some languages	No, unless you are sure it is stray
Left-to-right and right-to-left marks	U+200E, U+200F	Fix the direction of nearby punctuation and numbers	Only deliberately
Embeddings and overrides	U+202A–U+202E	Force a direction on a stretch of text	Only deliberately
Isolates	U+2066–U+2069	Set a direction on a stretch of text without affecting its surroundings	Only deliberately
C0 and C1 control characters	U+0000–U+001F, U+0080–U+009F	Old device and layout codes; tab and line break are the useful ones	Yes, except tab and line breaks

Characters that are usually safe to remove

The Text Cleaner removes these with the switches in the Characters card under the Input Text box. The default General Clean preset turns none of them on, because it only tidies whitespace; the presets that do are named below.

Zero width space (U+200B)

A zero width space marks a point where a line may break, without displaying anything. It is meant for languages written without spaces between words, such as Thai and Khmer, and it is also used to let long strings such as URLs and identifiers wrap. Copy such a string and the break opportunities come with it: the URL returns “not found”, the identifier is not recognised, or a search fails because the word is really data[ZWSP]base.

Removing it from copied text is safe in nearly every case; in a language that relies on it you lose line-breaking hints but no letters. Remove zero-width spaces is on in the PDF Copy, Word / Docs and Plain Text presets.

Byte order mark (U+FEFF)

At the very start of a file, U+FEFF is a signature that tells the reading program which Unicode encoding follows and, for UTF-16 and UTF-32, the byte order. In UTF-8 it is the three bytes EF BB BF and serves only as a signature. Anywhere else in a text it should not normally occur.

It becomes a problem once it is no longer the first thing in a file, for instance after files are joined or their contents are copied and pasted. Then the first column header of a CSV is [BOM]id instead of id, a parser rejects the very first character, or  appears where the bytes were read with the wrong encoding. In pasted text it has no function, so removing it is safe. Remove byte order marks is on in the Word / Docs and Plain Text presets.

Soft hyphen (U+00AD)

A soft hyphen marks a place where a word may be hyphenated at the end of a line; it is invisible unless the break is used. It arrives from web pages that use ­ and from typeset documents and PDFs. In the middle of a line it still splits the word as far as software is concerned, so a search for cooperation does not find co[SHY]operation.

Removing soft hyphens is safe when you are reusing text as plain text; the only loss is hyphenation hints for a layout that would have used them. Remove soft hyphens is on in the PDF Copy and Word / Docs presets.

Word joiner (U+2060)

The word joiner is the opposite of the zero width space: it has no width and prevents a line break between its neighbours. It is rare, causes the same mismatches, and deleting it only discards a no-break instruction. Remove zero-width spaces removes it together with U+200B.

C0 and C1 control characters

Unicode keeps 65 control codes from older standards: U+0000–U+001F, U+007F and U+0080–U+009F. Tab, line feed and carriage return are everyday text. The others have no place in ordinary writing, but a few still turn up in copied text: the vertical tab (U+000B), which Word uses for a manual line break, the form feed (U+000C), and the occasional null or escape character from a log or terminal. Parsers for strict data formats tend to reject them.

Remove control characters deletes them, leaving tab and line break alone, and turns vertical tab, form feed, line separator (U+2028) and paragraph separator (U+2029) into ordinary line breaks so that the break is not lost. It is on in the Word / Docs preset.

Characters to leave in place: ZWJ and ZWNJ

A zero width joiner (U+200D) asks for the characters on either side of it to be displayed as a connected unit. Its most visible use is in emoji: many are sequences of several emoji glued together with ZWJ, which a supporting font draws as one image.

👩 [ZWJ] 💻             drawn as one "woman technologist" emoji
👨 [ZWJ] 👩 [ZWJ] 👧    drawn as one family emoji

Take the joiners out and each sequence is shown as separate emoji. A zero width non-joiner (U+200C) does the reverse: it stops two letters from connecting where they normally would. Both belong to correct text in several writing systems. In Arabic-script languages they control whether neighbouring letters join; in Persian, for instance, a ZWNJ keeps two parts of a word from joining without putting a space between them, and it is part of normal spelling. In Indic scripts the two characters request particular forms of consonant clusters.

Software cannot tell a joiner that matters from one that got there by accident, so no Text Cleaner preset removes them. The option Remove ZWJ / ZWNJ is off everywhere and marked as risky. Tick it only when you know the text should contain neither, such as a list of ASCII product codes where a stray joiner is breaking a match.

Bidirectional controls

Text that mixes a left-to-right script such as Latin with a right-to-left script such as Arabic or Hebrew is ordered for display by the Unicode Bidirectional Algorithm. The algorithm cannot always know what the writer meant, particularly around punctuation, numbers and file paths, so invisible controls let the writer settle it:

• Marks. The left-to-right mark (U+200E) and right-to-left mark (U+200F) act like an invisible letter of that direction.
• Embeddings and overrides. U+202A, U+202B, U+202D and U+202E open a stretch of text with a stated direction, and U+202C closes it.
• Isolates. U+2066, U+2067 and U+2068 do the same while keeping the enclosed text from affecting its surroundings, and U+2069 closes them. Unicode encourages isolates for new documents.

In right-to-left text these characters are legitimate, and deleting them can move punctuation or digits to the wrong side on screen. In an identifier, a file path or a number they are one more cause of a failed match. In source code they are also a security concern: the Trojan Source research showed that controls placed in comments and strings can make code display in a different order from the one the compiler reads, so a reviewer sees logic that is not what runs.

No preset or checkbox in the Text Cleaner removes bidirectional controls. To remove them on purpose, open Advanced Cleaning Options and, under Find & Replace, tick Use regular expression, leave Replace with empty and put this in Find:

[\u200E\u200F\u202A-\u202E\u2066-\u2069]

How to find invisible Unicode characters in your text

1. Compare counts. Each box in the Text Cleaner shows a character count. A ten-letter word that reports eleven characters has something invisible in it.
2. Clean and read the summary. Paste the text, choose the Word / Docs preset and press Clean Text. The summary lists only changes that were actually made, so an entry for zero-width spaces, soft hyphens or control characters confirms they were there.
3. Use an editor that shows them. VS Code, for example, highlights uncommon invisible characters in source code by default.
4. Print the code points in a programming language and look for the values in the table above.

In the example below, [BOM], [ZWSP] and [SHY] stand for a byte order mark, zero width spaces and a soft hyphen, and the result comes from the Word / Docs preset.

Before

[BOM]Invoice INV-2041[ZWSP] is over[SHY]due.
See https://example.com/pay[ZWSP]ments/2041
Contact: accounts@example[ZWSP].com

After

Invoice INV-2041 is overdue.
See https://example.com/payments/2041
Contact: accounts@example.com

Nothing visible has changed, but the text is five characters shorter, and the summary reports 1 byte order mark removed, 3 zero-width spaces removed and 1 soft hyphen removed.

Related problems that look the same

A non-breaking space has width, so it is not strictly invisible, but it looks exactly like an ordinary space and defeats searches and trims in the same way. Plain doubled spaces and tabs are a separate matter, covered in removing extra spaces and tabs.

The Normalize Unicode (NFC) option, which sits in the same Characters group, does a different job: it makes an accented letter stored as a base letter plus a separate accent identical to the single-character form. It removes none of the characters described here.

Hidden characters are one entry in a longer list of copy-paste formatting problems, and they often arrive together with the visible ones.