Can you help me figure out why some files are corrupted when downloading with http.Get?

<p>Hello, Basically I have a few hundred data entry that contain links to images that I want to queue up and download in parallel.</p> <p>However, I always end up with one of them being corrupted. My guess is that there&#39;s some flaw in my logic and maybe the last one in the queue is getting the hammer.</p> <p>Here&#39;s some code snippet: <a href="https://play.golang.org/p/E0Dd4bm4Vt">https://play.golang.org/p/E0Dd4bm4Vt</a> Do keep in mind that I actually queue up hundred of different links.</p> <p>I also have another problem that maybe you can help me with. For a few of them I always get a tls handshake timeout error. I tried to get rid of it by defining some timeout with my own http.Client instead of using the default one. However, I ended up with lots of error about sockets being already closed and what not.</p> <p>As per jerf comment, I came up with this awful code that limit to 50 downloads (the const says 25 but the code does 50): <a href="https://play.golang.org/p/N7zujDQ-Xy">https://play.golang.org/p/N7zujDQ-Xy</a> it doesn&#39;t seems to corrupt so far.</p> <hr/>**评论：**<br/><br/>jerf: <pre><p>How are they corrupted?</p> <p>If they are cut off early, have you considered that you may be slamming the server so hard it starts falling over? The static files in your sample code probably wouldn&#39;t have that problem, but if you are asking for dynamic resources, possibly from someone less well provisioned than the Google urls you showed in the example, it&#39;s at least a thought. </p></pre>Kraigius: <pre><p>Here&#39;s a corrupted example: <a href="https://i.imgur.com/R4SYOzN.png" rel="nofollow">https://i.imgur.com/R4SYOzN.png</a></p> <p>They are stored on google butt storage. :(</p> <p><strong>edit</strong>: I&#39;ll see if I can make some sort of pool and limit the number of simultaneous downloads. I&#39;m not sure how to fill it all up sequentially without blocking and then blocking x at a time when reading from a channel or something :/</p></pre>Kraigius: <pre><p>Turns out, it might have been a mix of too much downloads at the same time and an error deeper in my business logic where I ended up with a race condition of trying to download 2 different things under the same file name at the same time. Also, if you look at my second link where I first implemented a download guard, I changed the <code>for</code> and <code>if ok</code> check over the channel to a <code>for range</code> on the channel. It simplifies the code and it&#39;s much more readable.</p> <p><a href="https://play.golang.org/p/OeapCYh08Z" rel="nofollow">https://play.golang.org/p/OeapCYh08Z</a></p></pre>kjk: <pre><p>Use race detector (-race, <a href="https://blog.golang.org/race-detector" rel="nofollow">https://blog.golang.org/race-detector</a>) to find out about potential issues due to multi-threading/races.</p></pre>