Speeding Up Multiple Gets from a third part API

<p>Hey all, </p> <p>I&#39;m fairly new to go and have hacked together an app that needs to hit a third party API multiple time (little over a hundred times) to get the data I need to make a calculation. Each time I send a request I need to change out one variable in the request URL. Therefore, I am sending a request for each variable as I iterate over the slice they are stored in. Currently, it can take anywhere from a minute to three to get back all of the data I need. </p> <p>I&#39;m wondering if there is a good way to break up the variables I&#39;m iterating over so I can make multiple requests to the third party API? Is there any good way to improve the performance of my app, is it already being handled by the net/http class, or is it a simple matter of being rate limited by the network and I&#39;m already as performant as I could be?</p> <hr/>**评论：**<br/><br/>kpurdon: <pre><p>Without seeing the real code or having more details it&#39;s a bit hard to say, but you will likely be able to make all of your requests concurrently.</p> <p>Googling &#34;golang concurrent http requests&#34; should get you some good examples of doing this.</p></pre>raff99: <pre><p>If the 3rd party API can &#34;scale&#34; and you can collect the responses out of order, you should get some benefit in sending multiple (2 to 10?) requests in parallel. With the sequential requests you have a lot of &#34;idle time&#34; waiting for the external service to reply.</p> <p>To do this you should &#34;spawn&#34; you can change your current code (where you iterate over the slice) to put the request info on a channel, and have a pool of goroutines that read from the channel and send the request to the 3rd party service.</p></pre>piratelax40: <pre><p>This is quite simple <em>given requests are independent</em>, that is you don&#39;t need the results of a previous request as part of the next. Given it seems like you already have a slice of data, I do not think that is the case.</p> <p>The key is to do the requests in goroutines, such that while waiting for the response it is not blocking.</p> <p>Here is an example:</p> <p><a href="http://blog.narenarya.in/concurrent-http-in-go.html" rel="nofollow">http://blog.narenarya.in/concurrent-http-in-go.html</a></p> <p>Likewise, please make sure you don&#39;t go too crazy with an api. They will often have certain rate limits in place. </p> <p>Take a look at: <a href="http://jmoiron.net/blog/limiting-concurrency-in-go/" rel="nofollow">http://jmoiron.net/blog/limiting-concurrency-in-go/</a></p> <p>You can use a semaphore (that is the term you can look up) to only allow a certain number of requests to be actively done at a time. This can be thought of like a worker pool.</p> <p>Eg if you had 100 urls to request, and set the limit at like 10, it would request 1-10 immediately, then as each finished, another would be kicked off, so no more then 10 were done at a given time.</p> <p>I&#39;d suggest not doing more than 20 concurrent requests.</p></pre>