guofangandClaude Sonnet 5.5 2f59bddba0 Show readable tokens for byte-level BPE tokenizers ...

Qwen tokens were printed as raw byte characters (e.g. 你好 as ä½łå¥½).
Decode each token for byte-level tokenizers, keeping the raw form for
tokens that hold only part of a multi-byte character.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

2026-11-10 22:19:56 +08:00

.gitignore

Initial commit

2026-11-10 11:07:18 +08:00

embedlab.py

Show readable tokens for byte-level BPE tokenizers

2026-11-10 22:19:56 +08:00

requirements.txt

Complete while typing with prompt_toolkit

2026-11-10 22:16:50 +08:00

setup.sh

Initial commit

2026-11-10 11:07:18 +08:00

S

Description

No description provided

91 KiB

0 Stars 1 Watchers 0 Forks

Languages

Python 98.5%

Shell 1.5%