You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
decode() reads past the end of a short string on truncated UTF-8, and decodes overlong sequences #305
Strings of 200 bytes or less (and all map keys) are decoded by utf8DecodeJs. It does not check the bytes it reads, so invalid UTF-8 gives a different result than longer strings, which go through TextDecoder.
The worst case: a truncated sequence at the end of a string reads bytes that belong to the next value.
import{decode}from"@msgpack/msgpack";// ["\xe2\x82", "A"] — first string is 2 bytes, a truncated 3-byte sequencedecode(Uint8Array.from([0x92,0xa2,0xe2,0x82,0xa1,0x41]));// → ["₡", "A"] (0xa1, the header of "A", was read into the first string)// expected ["�", "A"] (what TextDecoder gives)// overlong encodings decode to real charactersdecode(Uint8Array.from([0xa2,0xc0,0x80]));// → "\u0000" (TextDecoder: "��")decode(Uint8Array.from([0xa3,0xe0,0x81,0x9c]));// → "\\" (TextDecoder: "���")// other casesdecode(Uint8Array.from([0xa1,0xff]));// → "ÿ" (TextDecoder: "�")decode(Uint8Array.from([0xa1,0x80]));// → "\u0080" (TextDecoder: "�")decode(Uint8Array.from([0xa4,0xf4,0x90,0x80,0x80]));// → "𐀀" U+10000 (TextDecoder: "����")
So the same bytes decode one way in a short string and another way in a long one. Overlong NUL and \ can also get past checks done on the decoded string.
On 20k random byte strings made of valid and broken sequences, utf8DecodeJs differs from new TextDecoder() on 17,795 of them.
I have a fix that makes utf8DecodeJs follow the WHATWG decoder (one U+FFFD per maximal subpart, never read past byteLength) with the same speed on valid input. I can open a PR.
Activity