Parse input files

I need a parser for our input file format that reports line and column on syntax errors, supports comments and include directives, and produces an AST for downstream processing. Implement a streaming lexer and a recursive-descent parser that limits recursion depth to prevent stack blowup. Run the parser on the 500 existing config files and fix any failures before I hand it to the tools team on Monday.

Before I replace the old parser, make it simpler for reviewers and for future maintainers. Put the grammar summary and the error-reporting policy at the top of the file, collapse tricky expressions into small helper functions, surface the common parsing errors and their suggested fixes, and call out any constructs that would make a reviewer hesitate (ambiguous precedence, silent ignores of unknown directives).

I replaced the legacy parser and some CI jobs show diffs: the new parser rejects several config files the old parser accepted. I don't know whether the old behavior tolerated undefined directives or the new parser has a stricter grammar. I'm worried a silent change will break many users. What are the likely differences to check first, what test inputs will isolate them quickly, and what's the least-disruptive compatibility policy to adopt while we reconcile parsers?

Each time we refactor parsing code, obscure config files start failing in production. We lose credibility and spend days triaging. Where should we change habits: add a corpus of real-world configs in tests, define a clear compatibility mode, or reduce grammar complexity? Recommend the one habit that will eliminate most regressions and a pragmatic plan to implement it without blocking feature work.