pub fn tokenize_legacy(source: &str) -> Result<Vec<Token>, CompileError>Expand description
#2203: tokenise source as the lexer read it before // comments, for
the formatter’s migration of a file that still uses the legacy spellings.
A -- opens a comment wherever it starts the input or follows whitespace
(ADR 0188 D1), so --note and a --- Helpers --- divider are comments, as
they were; --| first on its line is a doc line. // comments and ---
blocks lex as in tokenize, so a half-migrated file reads too. Every
legacy token keeps its legacy spelling, so comment_body and
doc_line_text read it and the formatter prints the new one.