May have to try this out, cool how Mozilla can be extended. I wonder if it works with Firefox too?

Mozilla-based Spiders Version 0.0.1.7

Author Bob Clary
Created 2004-07-10, Modified 2005-12-14

Web Spider applications have a multitude of possible applications ranging from search indexes, web site quality assurance testing to testing browser implementations. This article introduces an improved mozilla-based Web Spider application framework which can be adapted for many different uses.

* What’s New
* CSpider.js – A New and Improved Spider Framework
* Spider – A Mozilla Application based upon CSpider.js
* Real World Applications of Spider
* Links and Stuff

In Web Site Quality Assurance Testing using Mozilla, I introduced the use of a mozilla-based web-spider application called CSpider to perform web site quality assurance. CSpider was based upon work I originally published on DevEdge. While the application did provide useful information for quality assurance testing, it did have a number of limitations which became more and more apparent to me as I used it to test web sites for compatibility with Mozilla-based browsers. Some of the limitations in the older versions are an inability to handle web pages which use “frame-busting” javascript to break out of html framesets, overly complicated and buggy logic for handling page loading and a lack of clarity in how the user-defined spider “event handlers” could be used to extend the basic framework.

Spider eliminates the limitations of the earlier version by separating the cross-browser JavaScript Class CSpider and implementing a basic framework for implementing spider applications in Mozilla for HTML, Remote XUL and Chrome applications.

Mozilla-based Spiders Version 0.0.1.7