Home / Knowledge hub / Unicode normalisation attack
Knowledge hub
Unicode normalisation attack
Characters that look different but compare equal after conversion, slipping past filters.
What it means
Unicode lets the same text be written in more than one way. A fullwidth letter, a letter with a combining accent and a single accented letter can all be converted into one standard form. That conversion is called normalisation, and it comes in several forms, such as NFC and NFKC.
A normalisation attack exploits the order of events. Your filter checks the raw text and finds nothing wrong. A later step converts it to a standard form, and the harmless-looking characters turn into the dangerous ones the filter was meant to block.
Why it matters
A filter that blocks admin, a script tag or a path separator can be bypassed by characters that only become those things after conversion. Another effect is account clashes: two usernames or email addresses that look different to your database but are the same after normalisation, which can open an account takeover route.
Nothing is broken in any one component. The flaw is in the gap between them, which is why it survives code review so easily. It affects any check that compares text, from login names to file paths and input validation rules.
How it shows up
You meet it where text is compared after being transformed: usernames, email addresses, URLs, file names, search terms and security filters. Frameworks, databases and operating systems each normalise at different moments, and some do not do it at all. Even a small mismatch is enough.
The rule is to normalise first and validate second, using one agreed form across the whole app. Store and compare usernames and emails in that form, and reject unexpected characters with an allowlist. Test by registering lookalike names and submitting fullwidth or combined characters to every filter.
Questions people ask
Unicode normalisation attack, answered
What is Unicode normalisation?
It is the process of converting text that can be written several ways into one standard form, so equal text compares as equal. The Unicode standard defines four forms: NFC, NFD, NFKC and NFKD.
Compatibility forms such as NFKC fold more characters together.
Is this the same as a homoglyph attack?
No, though they are cousins. A homoglyph attack uses characters that look alike but stay different, such as a Cyrillic letter that resembles a Latin one. Normalisation attacks use characters that become the same after conversion.
Both exploit the gap between what you see and what is stored.
Should we normalise usernames and emails?
Yes, pick one form and apply it before storing and before comparing. Otherwise two accounts can end up with names that are the same to a person and different to your database.
Document which form you chose.
Can a WAF catch this?
Only if it normalises in the same way and at the same point as your app. If it checks before conversion and the app acts after, it sees something different.
The order of operations is the thing to fix.
How do we test for it?
Submit fullwidth, combined and compatibility characters in login names, search fields, file names and any field with a filter. Compare what the filter sees with what the app finally uses.
Look for lookalike accounts that your system treats as separate.
Sources
Where this comes from
Related
Terms that sit next to this one
Canonicalisation
Reducing input to one standard form before checking it, so tricks cannot slip by.
Input validation
Checking that what arrives is the shape and range you expected.
Allowlist
A list of what you permit, with everything else refused by default.
Account enumeration
Learning which emails have accounts because your app answers differently.
XSS
An attacker's script running in your pages, in your customers' sessions.
Does the filter see what the app uses?
We try lookalike and converted characters against logins, filters and file handling, and show the request and the effect. Retests replay the same input. Book a pilot on one app.
Or start with a $199 pilot on one application: thirty days, success criteria agreed before day one, credited against the annual if you convert.