Skip to content

decode() reads past the end of a short string on truncated UTF-8, and decodes overlong sequences #305

Open

Description

@bardrop

Strings of 200 bytes or less (and all map keys) are decoded by utf8DecodeJs. It does not check the bytes it reads, so invalid UTF-8 gives a different result than longer strings, which go through TextDecoder.

The worst case: a truncated sequence at the end of a string reads bytes that belong to the next value.

import { decode } from "@msgpack/msgpack";

// ["\xe2\x82", "A"] — first string is 2 bytes, a truncated 3-byte sequence
decode(Uint8Array.from([0x92, 0xa2, 0xe2, 0x82, 0xa1, 0x41]));
// → ["₡", "A"]   (0xa1, the header of "A", was read into the first string)
// expected ["�", "A"] (what TextDecoder gives)

// overlong encodings decode to real characters
decode(Uint8Array.from([0xa2, 0xc0, 0x80]));       // → "\u0000"  (TextDecoder: "��")
decode(Uint8Array.from([0xa3, 0xe0, 0x81, 0x9c])); // → "\\"      (TextDecoder: "���")

// other cases
decode(Uint8Array.from([0xa1, 0xff]));                   // → "ÿ"        (TextDecoder: "�")
decode(Uint8Array.from([0xa1, 0x80]));                   // → "\u0080"   (TextDecoder: "�")
decode(Uint8Array.from([0xa4, 0xf4, 0x90, 0x80, 0x80])); // → "𐀀" U+10000 (TextDecoder: "����")

So the same bytes decode one way in a short string and another way in a long one. Overlong NUL and \ can also get past checks done on the decoded string.

On 20k random byte strings made of valid and broken sequences, utf8DecodeJs differs from new TextDecoder() on 17,795 of them.

Version: 3.1.3 (main 6d4b666).

I have a fix that makes utf8DecodeJs follow the WHATWG decoder (one U+FFFD per maximal subpart, never read past byteLength) with the same speed on valid input. I can open a PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions