Help to improve new tool that receives traffic logs from F5 load balancers for indexing in Elasticsearch

<p>Hi, I am currently developing a tool that we use to monitor nearly live traffic logs coming from our F5 Load Balancers inside Elasticsearch.</p> <p>The tool is a simple syslog server that receives messages for every request passing through our LB and creates nice json objects and adds some extra metadata like geoip location. We are actually using in production for a short while and pushing over 400k/req per minute on a single VM, which in itself is pretty impressive and shows the performance Go can deliver.</p> <p>I am mostly a Devops kind of guy who loves programming and Go so I am pretty sure the tool can be improved a bit and follow the more idiomatic Go way of doing things. If anyone feels like taking a look at the source to point out improvements it would be awesome, if not feel free to clone and use.</p> <p><a href="https://github.com/martensson/f5elastic">https://github.com/martensson/f5elastic</a></p> <hr/>**评论：**<br/><br/>bubaflub: <pre><p>A few suggestions:</p> <p>A minor nit, but in the NewWorker function on lines 66-72 I would just return the struct:</p> <pre><code>return Worker{ ... } </code></pre> <p>I would extract lines 96-120 into a separate function that takes a []string and returns a request or an error. That should make the loop a little clearer and also would make it easy to unit test that.</p> <p>I would not throw away the err from json.Marshall on line 130.</p> <p>The continue on line 134 doesn&#39;t look necessary.</p> <p>Sometimes you use log and other times you use fmt.Println. Perhaps just use one?</p> <p>I would pull some of the functionality out of main - that might make it a little clearer.</p> <p>Why do workers have IDs? It doesn&#39;t look like you access that anywhere.</p></pre>42wim: <pre><p>I&#39;ve also made something like your tool, but instead of F5 logs it are netfilter logs (<a href="https://github.com/42wim/ulog2queue" rel="nofollow">https://github.com/42wim/ulog2queue</a>). Which can do 70k req/sec depending on number of cores ;-)</p> <p>Some performance tips you could find useful:</p> <ul> <li>use ffjson (<a href="https://github.com/pquerna/ffjson" rel="nofollow">https://github.com/pquerna/ffjson</a>) - makes marshall/unmarshall about 2x faster.</li> <li>use LRU (<a href="https://github.com/hashicorp/golang-lru" rel="nofollow">https://github.com/hashicorp/golang-lru</a>) - cache your geoip lookups</li> <li>use multiple connections to ES</li> </ul></pre>