Commit d92e39a
committed
Add Cow-based internal tokenizer functions for zero-copy preparation
Add internal _borrowed() functions that return Cow<\'a, str> to prepare for
zero-copy tokenization. When the source string needs no transformation
(no escaping), return Cow::Borrowed. When transformation is required,
return Cow::Owned.
The Token enum still uses String, so borrowed values are converted via
to_owned() for now. This maintains API compatibility while preparing the
codebase for a future refactor where Token can hold borrowed strings.
Optimized: comments, quoted strings, dollar-quoted strings, quoted identifiers.1 parent 82c6657 commit d92e39a
1 file changed
+315
-95
lines changed
0 commit comments