Text Diff Checker

Diff tokens

Step-by-step calculation

1. 1Tokenize both texts by lines.
2. 2Compare tokens case-sensitively.
3. 3Build the longest-common-subsequence table and backtrack to classify each token as unchanged, added or removed.
4. 4Similarity is the unchanged share of all compared tokens.

Quick check

Best for: Everyday conversions, debugging and code reviews

• Method: Paste the text or bytes into the text measurement tool and read the headline result.
• Precision: UTF-8 byte counts and Unicode code points are exact; display-width estimates are approximate.
• Safeguards: Cross-check one ASCII value, one accented value and one emoji before trusting a length rule.

Production implementation

Best for: Shipping code that reads, stores or transmits text

• Method: Use an explicit encoding, decode with errors="strict", and keep the byte and character layers separate.
• Precision: Match the exact encoding and normalization form used by the storage layer or wire protocol.
• Safeguards: Add tests for ASCII, CJK, emoji, combining marks, embedded NULs and lone surrogates.

Security review

Best for: Input validation, identifiers and URL handling

• Method: Normalize before comparing, reject unexpected control characters, and encode at the correct output context.
• Precision: Apply NFC for identifiers and the output-context encoder (HTML, URL, JSON) for the destination.
• Safeguards: Test homoglyphs, zero-width characters, double encoding and mixed-script input.

Diff granularity

Lengths for the same string

Core relationship

diff = LCS(A, B); similarity = 2 × LCS / (|A| + |B|)

Verification

1. Decode the output with the stated encoding. 2. Round-trip it back to the original and compare. 3. Check the code-point, UTF-16 and UTF-8 counts independently. 4. Repeat with ASCII, accented text, CJK and one emoji.

One word changed

Inputs

brown → red

Math

Remove “brown”, add “red”

Result

−brown +red

Line added

Inputs

Two lines → three lines

Math

One unchanged, one added

Result

+ new line

Character diff

Inputs

color → colour

Math

Insert u

Result

+u

Python

Python 3 strings are sequences of Unicode code points. Use len(text) for code points, len(text.encode('utf-8')) for UTF-8 bytes, and the unicodedata module for normalization. For text diff checker, prefer str.encode/decode with an explicit codec and errors='strict' so invalid input fails loudly. The standard library handles surrogate pairs and most legacy codecs; do not hand-roll UTF-8 parsing.

JavaScript / TypeScript

JavaScript strings are UTF-16 code units. Use Array.from(text) for code points, TextEncoder/TextDecoder for UTF-8, and Intl.Segmenter for grapheme clusters where available. For text diff checker, avoid split('') on text that may contain emoji; it splits surrogate pairs. Buffer and btoa are not available in browsers, so use TextEncoder plus a small Base64 helper.

Java / Kotlin

Java String.length() counts UTF-16 code units; codePointCount() counts code points. Use StandardCharsets.UTF_8 with getBytes and new String for byte conversion, and Normalizer for NFC/NFD. Kotlin’s String is the same UTF-16 representation. For text diff checker, specify the charset explicitly instead of relying on the platform default.

Go

Go strings are immutable byte sequences, usually UTF-8. Ranging over a string yields runes (code points); len(s) yields bytes. Use utf8.RuneCountInString for code points, []rune(s) for rune slices, and golang.org/x/text/unicode/norm for normalization. For text diff checker, validate with utf8.ValidString when the input may be malformed.

Rust

Rust strings are guaranteed UTF-8. Use .chars() for code points, .bytes() for bytes, and the unicode-segmentation crate for grapheme clusters. Slicing a String by byte index can panic if it lands inside a code point. For text diff checker, work with char_indices() when you need both position and character, and use encoding_rs for legacy encodings.

C / C++

C and C++ distinguish char, wchar_t, char16_t and char32_t. UTF-8 is usually stored in char arrays; convert with codecvt, ICU or a dedicated library rather than manual byte surgery. For text diff checker, keep the byte length separate from the display width and validate the input before conversion. Do not assume wchar_t is 32-bit on every platform.

Mistakes that break conversions

• Calling String.length a character count when it counts UTF-16 code units.
• Comparing byte, code-point and grapheme counts without stating which one the limit uses.
• Counting UTF-16 code units and calling them characters; JavaScript and Java lengths are not character counts.
• Decoding bytes with the wrong encoding, which produces mojibake instead of an error.
• Treating Base64, percent-encoding or HTML escaping as encryption; all three are reversible by anyone.
• Truncating between surrogate halves or inside a multi-byte UTF-8 sequence.

Pro tips for production code

• Use grapheme clusters for UI limits and UTF-8 bytes for storage limits.
• Normalize text before comparing identifiers or lengths.
• Specify the encoding at every boundary: file, network, database and display.
• Normalize identifiers with NFC before comparing them, and reject control or zero-width characters where they serve no purpose.
• Use byte length for storage limits and grapheme clusters for user-facing character limits.
• Prefer the platform encoder or serializer over hand-built escaping; it handles the edge cases you will forget.