Parsing text based content / Large dictionnary

<p>Hey guys,</p> <p>Hope you are all well !</p> <p>I was wondering what are the keywords/concepts used in order to detect concurrently some known patterns into a file or list of files.</p> <p>Use case: - Generate alpine docker files based on all included/required dependencies by a Github/Gitlab repo by scanning the source code. - Enable the task through git hooks</p> <p>Context: - 10 000 github starred repositories - 10 000 know patterns like c++/go/nodejs frameworks or packages, editors or companies (eg: <a href="https://github.com/porter-io/tagg-python/blob/master/tagg/default_defs.json" rel="nofollow">https://github.com/porter-io/tagg-python/blob/master/tagg/default_defs.json</a>)</p> <p>Goals: - Distributed and recursive files scanning of github repository - Large dictionaries of words and combinations - Create a report of matched patterns - Generate a dockerfile based on detection and mapping the correspondent apk package by binding alpine package repository (eg. <a href="https://github.com/jessfraz/apk-file" rel="nofollow">https://github.com/jessfraz/apk-file</a>)</p> <p>Questions: - Is there any frameworks, golang based, in order to detect and tag in a fast and concurrently manner those know entities ? - Is there a better alternative than using regex rules ? like runes ? - What are the proper concept I should watch/check to understand the best approach to such task ?</p> <p>References found: <a href="https://github.com/svent/sift" rel="nofollow">https://github.com/svent/sift</a> <a href="https://github.com/jessfraz/apk-file" rel="nofollow">https://github.com/jessfraz/apk-file</a> <a href="https://github.com/DavidGamba/ffind" rel="nofollow">https://github.com/DavidGamba/ffind</a></p> <p>Thanks for any insights and advices !</p> <p>Cheers, Richard</p> <hr/>**评论：**<br/><br/>epiris: <pre><p>Both your posts have way too high level of problems, you&#39;re asking how to write a project rather than how to solve a problem you are facing while writing the project. Effectively your &#34;questions&#34; is how to perform software engineering using Go, it doesn&#39;t have a single answer.</p> <p>If you are just starting with Go I suggest <a href="https://golang.org/doc/install" rel="nofollow">getting started</a> and <a href="https://golang.org/doc/effective_go.html" rel="nofollow">effective Go</a> which may paint a partial picture of some of your questions. Then I would begin the project yourself and if you get stuck then feel free to post here or maybe join the gophers slack channel for a lower latency feedback loop. Make sure not to provide an <a href="https://www.google.com/search?q=xy+problem" rel="nofollow">XY</a>, the best way to do this is to post a working playground example showing something as close as possible to what you want to do.</p></pre>dgryski: <pre><p>For the string matching part, you haven&#39;t quite given enough details but it sounds like you might want <a href="https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm" rel="nofollow">https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm</a> . There&#39;s an implementation from Cloudflare: <a href="https://github.com/cloudflare/ahocorasick" rel="nofollow">https://github.com/cloudflare/ahocorasick</a></p></pre>HowardTheGrum: <pre><p>Distributed and recursive files scanning:</p> <p>For the file scanning: my suggestion here would be to look at the Sift project: <a href="https://github.com/svent/sift" rel="nofollow">https://github.com/svent/sift</a></p> <p>It is a code-focused grep alternative written in Go, it is a project that already knows about things like reading .gitignore files, using parallelism in searching, ignoring irrelevant files, skipping binary files, etc. </p> <p>It should give you at least an idea of how to handle concurrently consuming a file set while matching regular expressions. (Edited to add: just saw you do list sift in your references found... don&#39;t <em>think</em> that was there when I started writing this, but maybe I just missed it.)</p> <p>As to the &#39;distributed&#39; part, this seems relatively trivial - basically, the problem space is already chunked for you, by being divided into files. So you would have something like one app that receives the notification from a git hook that a new check-in has occurred (I assume - not sure how else git hooks would be relevant), which would go into a queue of apks to be produced, I assume, and it would read the repository file list, and then hand those files off to your distributed workers, in the form of a JSON or grpc query with a list of files and a git URL+ref combo or something like that to rejoin the returned results. Distributed workers submit results, either to that same master, or to some other worker that collates them into a master list of dependencies for that repository. Cache that somewhere. Have the first, master service recognize the git url/ref pair and pull from the cache instead of streaming to the workers when it gets the same request again.</p> <p>Take that and match those dependencies up to your apks, find the ones that are missing from your list of already generated apks, and submit them back to the first service to be scanned for dependencies. If there are NO missing items, then this project is ready to be wrapped up into an apk, so either do that, or submit that to a service that does that.</p> <p>Now, go through that and add something to detect and break import loops so you don&#39;t end up with infinite requests flowing between your services.</p></pre>