<p>Hi Everyone,</p>
<p>A requirement of my new startup was eventually building our own web crawler. I've been reading about it for quite awhile now, seeing how others have solved the problem of performing extremely broad web crawls. Very recently too, I even tried using the popular <a href="https://github.com/scrapy/scrapy">Scrapy</a> crawler, but it just didn't meet our goals. Apache Nutch is widely known, but is complicated to set up and <a href="https://wiki.apache.org/nutch/HardwareRequirements">requires a crazy amount of disk space and memory</a>, among other issues. <a href="https://github.com/PuerkitoBio/gocrawl">Gocrawl</a> was another option and it's coded in Go, but it's suited more for depth-first crawls.</p>
<p>We wanted a crawler that could:</p>
<ul>
<li>Perform a true <a href="https://en.wikipedia.org/wiki/Breadth-first_search">BFS</a> (Breadth-First Search) crawl of the web</li>
<li>Crawl billions of web pages on a single machine, with 1 (or more) CPUs, using only a few GB of memory</li>
<li>Use the disk for the entire URL queue while remaining performant, so the only limitation of crawl reach is disk space (much cheaper than memory)</li>
<li>Crawl concurrently</li>
<li>Crawl politely</li>
</ul>
<p>So, after spending a long time learning Go and doing some groundwork like <a href="https://github.com/beeker1121/goque">Goque</a>, we now have our own crawler built 100% in Go that meets these goals.</p>
<p>Here are the current stats of it running on a machine with 1 vCPU, 3.75GB of RAM, 1TB of disk space, running ~10 regular expression on 512KB of data or less: <a href="http://i.imgur.com/H33JYCD.png">http://i.imgur.com/H33JYCD.png</a></p>
<p>My question is, how much interest is there in us open sourcing this crawler? I'm also thinking of writing a post about this on our blog, just detailing how it works.</p>
<p>If this is something you would be interested, let us know by upvoting, commenting, or both!</p>
<hr/>**评论：**<br/><br/>Patatatarte: <pre><p>The community will always benefit from your work and the inverse is likely to be true. </p></pre>bkeroack: <pre><p>I'm a huge FOSS advocate, but this is clearly an address-harvesting crawler for spammers. I don't see how supporting this will benefit the community.</p></pre>Veonik: <pre><p>So would you say that we should censor knowledge because it might be misused?</p></pre>mwholt: <pre><p>I'd like to see the code powering email harvesters, they're going to run either way.</p></pre>unimportant1234567: <pre><p>I would use something like this for general web archiving and curation purposes. I bookmark sites and they die and I lose information because the site is gone. This may be a great tool for this use case.</p></pre>jerf: <pre><p>The web crawler is the hard part. Extracting email addresses out is like three lines attached to the crawler at the end. Even if that is what they happen to using it for, the crawler itself may still be useful.</p></pre>morethanaprogrammer: <pre><p>Would enjoy seeing the process as well</p></pre>nycalibjj: <pre><p>"Emails found" implies that this is an email address harvesting spambot. </p></pre>Yojihito: <pre><p>Which would explain the regex stuff in the description.</p></pre>Mr_Psmith: <pre><p>Yes, by all means! Would love to read your blog post as well.</p></pre>Traim: <pre><p>Blog post would really be appreciated. Currently I personally will not use the crawler but maybe in the future. </p></pre>lluad: <pre><p>I don't need a web crawler, but I'm still interested in seeing what approaches you used.</p></pre>unimportant1234567: <pre><p>Very interested in this type of work and article. Please let us see it :)</p></pre>bateller: <pre><p>Please write this up. I'd find this very useful.</p></pre>thesilentwitness: <pre><p>I'm curious why you would ask if you should open source it instead of just open sourcing it and then linking to it?</p></pre>pandawithpie: <pre><p>Does your crawler deal with client-side rendered pages (e.g. react, angular)? I've found that there's a lot of complexity related to rendering these pages before saving them to disk. </p>
<p>Curious to know if you managed to solve this! </p></pre>Yojihito: <pre><p>A crawler in pure Go can't deal with those, no Javascript = no client-side rendering.</p>
<p>You only get the html code.</p></pre>mwholt: <pre><p>That's not <em>entirely</em> true: <a href="https://github.com/robertkrimen/otto" rel="nofollow">https://github.com/robertkrimen/otto</a></p></pre>robertmeta: <pre><p>There are multiple javascript engines for Go. </p></pre>Yojihito: <pre><p>Do you have a list? I need a replacement for PhantomJS to execute JS on crawled pages.</p></pre>iamafuckingrobot: <pre><p>Yes I'm very interested! Please keep us updated.</p>
<p>Edit: I've been working on a very similar solution, essentially an alternative to Nutch, for over a year as my FT job. Unfortunately not in Go, but you can see why I'm interested.</p></pre>UnchartedFr: <pre><p>I'm interested !
Since i'm a heavy user of scrapy, I feel that it has too many limitations because of our needs and was thinking to migrate to golang too.
Do you handle Xpath like scrapy ?</p></pre>dAnjou: <pre><p>What are those limitations of Scrapy? Last time I used it it seemed very flexible.</p></pre>Asti_: <pre><p>Please write it up. I love learning from people who use Go to solve actual problems and not just the basic "We are going to create a tasks app..." form. Great job.</p></pre>BestKarmaEver: <pre><p>I know i would be very interested in the way it works</p></pre>Southclaw: <pre><p>I was actually thinking of writing a crawler in Go recently, this looks interesting so I'd say definitely open source it!</p></pre>itsamemmario: <pre><p>Please do!</p></pre>chmouelb: <pre><p>I'll definitely be interested to look how this is be implemented </p></pre>matiasbaruch: <pre><p>Sounds interesting!</p></pre>shazow: <pre><p>Sounds like a great codebase with lots of interesting things to learn from. I'd love to read through it, and would definitely contribute back if I used it. Please open source it sooner than later. :)</p></pre>Yojihito: <pre><p>I've build my own crawler as a side project in the office in go but it's limited (by choice) to crawl a single domain and get all links, status codes and link sources in a .csv file.</p>
<p>I would be very interested in your crawler.</p>
<p>What is your crawler crawling? What do you need 1TB full of links for?</p></pre>sarcasmkills: <pre><p>Open source! Sounds awesome!</p></pre>Asdayasman: <pre><p>Not being super comfortable with reading Go code yet, I'm always down to read a writeup.</p></pre>
