scrawl website with golang

<p>Hi, I have been working with scrapy (python) for scrawling website . Now i want to try wiith Go . I want to know can we scrawl authentication website with golang ? What is advantages between goland and scrapy.org ?</p> <hr/>**评论：**<br/><br/>ishanjain28: <pre><p>Hi, My friend and I run a bunch of websites that track different stock market websites and then alert users(I wrote the backend of all the websites). We were using scrapy+flask earlier because I was not very good at go. Then I learned go and started using it everywhere and really like it now.</p> <p>The scrapy scraper took about 7-10 seconds from scrapping the website to alerting the user. Then I rewrote the whole thing in go and now It takes about 2 seconds to do the same thing.</p> <p>The code is much cleaner and I really absolutely like goquery over scrapy. </p></pre>Morgahl: <pre><p>Main advantage in this situation will be Goroutines to drastically speed up the process.</p></pre>bjwschaap: <pre><p>Yes you could. Logging into a site with plain golang code would be something in the lines of:</p> <pre><code> // Store session cookies after login jar, _ := cookiejar.New(nil) // Setup HTTP client, and give it access to the cookiejar client := &amp;http.Client{ Jar: jar, } // Create POST data to send to login form log.Println(&#34;About to login user&#34;, *username) urlData := url.Values{} urlData.Set(&#34;sForm&#34;, &#34;login&#34;) urlData.Add(&#34;username&#34;, *username) urlData.Add(&#34;password&#34;, *password) // Prepare http POST request req, _ := http.NewRequest(&#34;POST&#34;, &#34;http://www.mysite.com/login&#34;, strings.NewReader(urlData.Encode())) req.Header.Add(&#34;Content-Type&#34;, &#34;application/x-www-form-urlencoded&#34;) // Do the http POST to login resp, err := client.Do(req) if err != nil { log.Fatal(err) } // ..and we&#39;re logged in log.Println(*username, &#34;logged in succesfully&#34;) resp.Body.Close() </code></pre> <p>In conjunction with <a href="https://github.com/PuerkitoBio/goquery" rel="nofollow">https://github.com/PuerkitoBio/goquery</a> this is very powerfull, especially if you use Go routines like Morgahl said. </p> <p>I don&#39;t know Scrapy, but from quickly looking at the Scrapy documentation the overall experience/usage looks quite similar. Using goquery might be somewhat more verbose, but you could easily create convenience functions for that..</p></pre>lasizoillo: <pre><p>Reading comments... no one advantages. Many of them talks about velocity of golang without check default scrapy conf to be a good citizen. Their are unfair (and wrong) metrics.</p> <p>Scrapy works with event programming, Go with n:m threading system. So probably go consume more ram (2k or 4k of stack size by each connection) but you don&#39;t worry about if your code is blocking or not. If your are doing complex things in your spider, probably go consume less cpu. But if your bottleneck its net your language is not important.</p> <p>Scrapy its a complex and mature framework with a lot of side projects (for example frontera). Go its a language without a comparable framework. Maybe the answer is: use scrapy for complex parser and go for toy or specific ones. Maybe in a future something comparable to scrapy exists in go, but not for now.</p> <p>And no, no and no, scrapy is not comparable with http+goquery.</p> <p>PD: I was working with a modified scrapy with prioritized queues and a custom scheduler. Make it in go will be easier than in scrapy (program with twisted its hard), but i need anti-thotling stuff, xpath and css selectors, ... and many things with came with scrapy. Sometimes I dream with a scrago ;-)</p></pre>hell_0n_wheel: <pre><blockquote> <p>Now i want to try wiith Go</p> </blockquote> <p>Then try, and get back to us when you have something to share.</p></pre>