Crawling - Recent Bookmarks - Page 1:

My thesis - building blocks of a scalable webcrawler - Marc's Blog
In my last semester as a student, I had the chance of working for an awesome company (Acquia) on a very interesting project. It all started with a post over at Dries Buytaert's blog. He is the CTO and... Read more
http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blocks-of-a-scalable-webcrawler
Tags: nosql, scalability, crawling, io, programming, webcrawler, asynchronous, thesis, searchengine, web Saved by: admin at 20 Dec 2010

Getting Started - Making AJAX Applications Crawlable - Google Code
Your server, on the other hand, needs to know that it has to return an HTML snapshot, rather than the normal page sent to the browser. Remember: an HTML snapshot is all the content that appears on the... Read more
http://code.google.com/web/ajaxcrawling/docs/getting-started.html
Tags: ajax, seo, google, crawler, javascript, search, accessibility, indexing, reference, crawling Saved by: admin at 23 Sep 2010

Grub's Distributed Web Crawling Project
distributed web crawling engine
http://www.grub.org
Tags: search, crawling Saved by: admin at 29 Jun 2009

BotSeer: Robots.txt and Web Crawler Search Engine
http://botseer.ist.psu.edu
Tags: crawling, crawler Saved by: admin at 29 Jun 2009