Resource Handling in Goroutines

<p>Hey all, I&#39;m working on converting a Python script into Go because I&#39;ve heard a lot of really great things about its overall performance. The Python script essentially telnets into a few thousand devices and collects and parses some diagnostic data. Python has been incredibly slow at this scale and has been burdensome. The act of data collection takes several hours. I&#39;ve rewritten the functional part of the script in Golang and everything&#39;s up and running fine. I&#39;m using <a href="https://github.com/ThomasRooney/gexpect" rel="nofollow">Gexpect</a> to automate the commands and collection since we&#39;re using Pexpect in the original script. The Golang script runs great, it&#39;s collecting the data accurately, and it&#39;s pretty fast. </p> <p>The problem I&#39;m running into is that around 400-500 devices, it&#39;s unable to fork. I believe I&#39;m closing out of the Telnet calls and killing the child process properly in addition to using a <a href="https://github.com/remeh/sizedwaitgroup" rel="nofollow">SizedWaitGroup</a> as a buffer to populate the Goroutines. I was wondering if anyone here has had some experience in this and what I could try to handle this issue. Here&#39;s a <a href="https://gist.github.com/anonymous/417de81094d5fcc81bb6fd6be16a9efd" rel="nofollow">Gist</a> of what the script currently looks like. If anyone has any advice or suggestions, it would be greatly appreciated.</p> <hr/>**评论：**<br/><br/>adzeitor: <pre><p>When <a href="https://github.com/ThomasRooney/gexpect/blob/master/gexpect.go#L150" rel="nofollow">device.child.Close()</a> program sends SIGKILL to telnet process, but not waiting for result and telnet becomes zombie. </p> <p><a href="https://stackoverflow.com/a/36050638/962956" rel="nofollow">Stackoverflow answer</a></p> <p>I think it&#39;s bug in gexpect, but you can quickfix it now, insert this before <a href="https://gist.github.com/anonymous/417de81094d5fcc81bb6fd6be16a9efd#file-gistfile1-txt-L74" rel="nofollow">74 line</a>:</p> <pre><code>device.child.Cmd.Process.Wait() </code></pre></pre>ImageOfInsanity: <pre><p>Holy shit, thank you so much.</p></pre>Redundancy_: <pre><p>I think that interacting with telnet through spawning a process is just strange. It&#39;s not that I haven&#39;t done something similar for other problems, but you&#39;d be far better if you could bring that functionality into Go itself. Maybe something like <a href="https://github.com/reiver/go-telnet" rel="nofollow">https://github.com/reiver/go-telnet</a> would help with that.</p> <p>Note that <a href="https://gist.github.com/anonymous/417de81094d5fcc81bb6fd6be16a9efd#file-gistfile1-txt-L107" rel="nofollow">https://gist.github.com/anonymous/417de81094d5fcc81bb6fd6be16a9efd#file-gistfile1-txt-L107</a> potentially exits without closing the device. You could use defer to prevent that if it isn&#39;t intentional.</p></pre>