Web Scraper Help!

<p>Hello Reddit, Here&#39;s our problem: I have been assigned the task of creating a .csv file that automatically updates itself. I have to download a .csv file and add the information from it onto a master list. This update must take place every day at a specific time. I have to complete this task in Golang, and my knowledge of Go is limited. So, the thing that I need help with is that I need to use a web scraper in order to obtain the download link for the .csv file. How should I go about this? Thanks!</p> <hr/>**评论：**<br/><br/>Dummies102: <pre><p>you probably want to use something like this <a href="https://github.com/PuerkitoBio/goquery" rel="nofollow">https://github.com/PuerkitoBio/goquery</a></p></pre>tclineks: <pre><p>it sounds like he&#39;s consuming csv (not html scraping)</p></pre>jamra06: <pre><p>I think OP is under the impression that he needs to scrape the web page to find the CSV download link.</p> <p>I personally would take the route of trying to find the direct download link to the CSV manually and then use that -- unless of course it is not consistent.</p></pre>steakholder69420: <pre><p>Thanks, that really helps! </p></pre>tclineks: <pre><p>I&#39;d recommend reading introductory material and then look into how to fetch resources over http and how to read and write csv files. There are packages in the standard library that allow you to accomplish this (specifically, net/http and encoding/csv).</p></pre>steakholder69420: <pre><p>Thanks so much! Sounds like a great plan! </p></pre>raff99: <pre><p>Does the download link changes every day ? Is it something you can guess without having to parse the container page ?</p> <p>Maybe you want to prototype it using a couple of curl commands (fetch the container page, fetch the CSV), then write down the exact steps you need to execute (you mention a &#34;master list&#34;. Is this another CSV file, a database or what ?) and then ask how you can implement these steps in Go (the first two would be basic HTTP requests, for which you can see examples in net/http as suggested by others)</p></pre>steakholder69420: <pre><p>Thanks a lot for your help! I&#39;m pretty sure that the download link is constant. As well, the master list is another csv file that has to be updated based on changes in other csv files from the Internet. </p></pre>nyoungman: <pre><p>If that is the case, you won&#39;t need goquery or golang.org/x/net/html which underlies goquery. Just hard code the urls to the csv files or pass them as command line arguments.</p> <p>Just download the file using http.Client. You will likely want to os.Open a file and then io.Copy the request Body to the file. </p> <p>Are you updating the existing CSV file in place?</p> <p>To make it a little more robust, you can download the file to a temporary location (use ioutil.TempFile) and only overwrite the previous CSV if there were no errors (use os.Rename with the Name() of the tempfile and the final location).</p></pre>