Webcrawler - Recent Bookmarks - Page 1:

My thesis - building blocks of a scalable webcrawler - Marc's Blog
In my last semester as a student, I had the chance of working for an awesome company (Acquia) on a very interesting project. It all started with a post over at Dries Buytaert's blog. He is the CTO and... Read more
http://blog.marc-seeger.de/2010/12/09/my-thesis-building-blocks-of-a-scalable-webcrawler
Tags: nosql, scalability, crawling, io, programming, webcrawler, asynchronous, thesis, searchengine, web Saved by: admin at 20 Dec 2010

Web Crawler
is a program or automated script which browses the in a methodical, automated manner. Other less frequently used names for web crawlers are ants, automatic indexers, bots, and worms (Kobayashi and Tak... Read more
http://www.noviway.com/Code/Web-Crawler.aspx
Tags: webcrawler, search Saved by: admin at 29 Jun 2009

Nutch - Webcrawler for Lucene
http://lucene.apache.org/nutch
Tags: web, tmv, webcrawler, spider, searchengine, search, opensource, lucene, library, apache Saved by: admin at 29 Jun 2009

Heritrix - Home Page
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/ hereti... Read more
http://crawler.archive.org
Tags: webcrawler, tmv, spider, crawler Saved by: admin at 29 Jun 2009