<p>I'm rewriting one of my applications and don't want to reuse my old, clunky, hand-written lexer. Is there a good library out there for writing some fairly complicated grammars?</p>
<p>I've found <a href="https://github.com/alecthomas/participle">https://github.com/alecthomas/participle</a>, but am having trouble figuring out how to use it. Also not a big fan of using tags/reflection.</p>
<p>Are there any other libraries out there for lexing and generating tokens? The grammar doesn't really matter.</p>
<hr/>**评论：**<br/><br/>dgryski: <pre><p>There are multiple implementations of lex, as well as ragel and lexmachine.</p></pre>Flowchartsman: <pre><p>I always upvote Ragel</p></pre>Badu_Ro: <pre><p>Antlr (<a href="https://github.com/antlr/antlr4" rel="nofollow">https://github.com/antlr/antlr4</a>) might help you. Has a generator for Go and a repo with a huge number of grammars.</p></pre>nairb774: <pre><p>Doesn't solve the complicated use case, but "text/scanner" is great for the simpler side of things. One of those packages that I find can be easily missed in the standard library.</p>
<p>Edit: s/cane/can/</p></pre>swimmer91: <pre><p>Woah, cool! I had no idea that existed</p></pre>meta_leap: <pre><p>It's pretty great. I have a hard time imagining a language-lexing use-case needing anything more, as you can handle the emitted primitives however suits you. It does apply Go-the-language's own semantics of string/number/ident/comment lexing but especially the former 3 are perfectly fine for near-all scenarios.</p>
<p>When I used it for a side-project (parsing a 'Haskell Core'-like mini-lang) I evolved this little wrapper pkg over <code>text/scanner</code> for <em>my</em> usage (ie. not particularly general-purpose!) --- has a few helper funcs and might further illustrate how to fit slightly-custom-needs with that brilliant stdlib pkg: <a href="https://github.com/go-leap/dev/tree/master/lex">https://github.com/go-leap/dev/tree/master/lex</a></p></pre>Bake_Jailey: <pre><p>To give you more things to look at:</p>
<ul>
<li>Rob Pike's lexer talk: <a href="https://www.youtube.com/watch?v=HxaD_trXwRE" rel="nofollow">https://www.youtube.com/watch?v=HxaD_trXwRE</a></li>
<li>Some parser/lexer libraries:
<ul>
<li><a href="https://github.com/goccmack/gocc" rel="nofollow">https://github.com/goccmack/gocc</a></li>
<li><a href="https://github.com/pointlander/peg" rel="nofollow">https://github.com/pointlander/peg</a></li>
<li><a href="https://github.com/prataprc/goparsec" rel="nofollow">https://github.com/prataprc/goparsec</a></li>
</ul></li>
</ul>
<p>Some of that might be useful, but some not. gocc is great, and does let you generate just a lexer.</p>
<p>I was writing my own parser-combinator without codegen, but never released it. Maybe I should finish it, someday...</p></pre>ChrisTrenkamp: <pre><p>Rob Pike's talk is what I based my old lexer on. It's great when you're implementing your own grammars, but when you're implementing a spec, you need something to verify you've implemented it correctly.</p></pre>Dummies102: <pre><p>I have to recommend a <a href="https://en.wikipedia.org/wiki/Parsing_expression_grammar" rel="nofollow">PEG</a> parser. I've used <a href="https://github.com/mna/pigeon" rel="nofollow">https://github.com/mna/pigeon</a> with some success. Very easy to use. It's not quite what you're asking for, but unless you need left-recursion, it should probably work. </p></pre>FatFingerHelperBot: <pre><p>It seems that your comment contains 1 or more links that are hard to tap for mobile users.
I will extend those so they're easier for our sausage fingers to click!</p>
<p><a href="https://en.wikipedia.org/wiki/Parsing_expression_grammar" rel="nofollow">Here is link number 1</a> - Previous text "PEG"</p>
<hr/>
<p><sup>Please</sup> <sup>PM</sup> <sup><a href="/u/eganwall" rel="nofollow">/u/eganwall</a></sup> <sup>with</sup> <sup>issues</sup> <sup>or</sup> <sup>feedback!</sup> <sup>|</sup> <sup><a href="https://reddit.com/message/compose/?to=FatFingerHelperBot&subject=delete&message=delete%20dyb4el2" rel="nofollow">Delete</a></sup></p></pre>knotdjb: <pre><p>Also interested.</p></pre>0xjnml: <pre><p><a href="https://github.com/cznic/golex" rel="nofollow">https://github.com/cznic/golex</a></p></pre>adamcolton: <pre><p>I'm going to plug my own project <a href="https://github.com/adamcolton/parlex" rel="nofollow">https://github.com/adamcolton/parlex</a>. <a href="https://github.com/AdamColton/parlex/tree/master/lexer/stacklexer" rel="nofollow">This lexer</a> may be useful to you. And if you have any questions, or find any issues, I'll do what I can to help.</p></pre>
