Progrider

Programming discussion

Board index
Findings · Open discussion · unicode

String length is not what you think it is, again

Posted by ines (5 posts) #1 Quote
Edit Report

Third bug report this year from a form that "limits names to 20 characters".

'é'.length is 2 in JS (e + combining accent), while 'é' typed as one code point is 1. They look identical. Most emoji are 2, because length counts UTF-16 units. The family emoji with zero-width joiners is 11.

What to use depends on why you're counting:

  • bytes for a storage limit: Buffer.byteLength or TextEncoder
  • what a person calls a character: Intl.Segmenter with granularity 'grapheme'
  • comparing strings: normalize('NFC') both sides first

[...s].length gets you code points, which is the one people reach for, and it's still not what a person would count.

3 replies

ferrous (11 posts) #2 Quote
Edit Report

Rust's s.len() is bytes, chars().count() is code points, and graphemes need a crate. At least it's obvious you're choosing.

sam (19 posts) #3 Quote
(3)
Edit Report

So my maxlength on the HTML form is counting UTF-16 units?

ines (5 posts) #4 Quote
Edit Report

Yes. maxlength and JS length both. For most text you'd never notice. It's names, and anyone using emoji, that hit it.

Add a reply

Formatting help. Public, and removed when the thread expires.

Type name#secret for a tripcode, an identity others can't copy.

Posts and requests are logged for research. Privacy notice.

Related topics