Article URL: https://astgrep.com/blog/tree-sitter-rust-rewrite Comments URL: https://news.ycombinator.com/item?id=49060509 Points: 30 # Comments: 2

ast-grep rewrote Tree-sitter's C core in Rust, with AI writing the code. The new core is faster at parsing, faster at reading the completed tree, and faster in ast-grep itself. (The title's “30%” is the parser-only number; end-to-end, ast-grep runs about 22% faster.) Two quick introductions before the numbers. ast-grep — the structural code-search tool this blog belongs to — searches code by syntax rather than by text, so every file it touches must first become a syntax tree. Tree-sitter is the parser framework that builds that tree: you give it a grammar definition, and it generates a fast parser for that language. Born in the editor world, it now powers an enormous ecosystem of grammars and tools. Performance and peak RSS. Throughput is normalized so the unmodified C build (“C / normal”) scores 100; higher is better. RSS is peak resident memory, and the raw-parsing row shows it as a range because it varies across the benchmark's language fixtures. The outline row is ast-grep's real workload: parse every file in a repository, then walk each completed tree to extract a structural outline. Rust won every parser and traversal fixture, and ast-grep produced exactly the same outline. Memory is the tradeoff: the Rust build uses about 8 MiB more in the ast-grep run. On the much larger TypeScript stress corpus — the TypeScript compiler repository's test-baseline tree, this project's memory torture test — it peaks at 91.2 MiB. That figure is a triumph, not a confession: earlier in the project, the same corpus peaked above 1 GiB. The result is not a 1:1 replacement for upstream Tree-sitter. It is a narrower runtime built for AI coding agents that analyze complete file snapshots: That boundary keeps the grammar ecosystem useful for agentic coding while removing editor-specific machinery the target workload does not need. Every serious ast-grep performance investigation eventually arrived at the same place: Tree-sitter. ast-grep could make its rules faster. It could prune work, cache configuration, and avoid visiting irrelevant syntax. But every file still had to become a syntax tree first, and Tree-sitter built that tree. The parser was both the foundation and, increasingly, the ceiling.