Run Your Own Spider

A spider program collects and filters information from the Internet, looking for specific keywords or other indicators that match its assigned task or topic. Probably the best known spider program is Google, which omits the part about looking for matching topics and simply attempts to index the entire World Wide Web.

Why, then, with Google available and indexing everything, would you want to run your own spider? One obvious disadvantage of Google is that it does, indeed, index EVERYTHING, which means it probably finds sites with keywords that, while matching what you're looking for, don't really dwell significantly on the topic in which you're interested. While Google attempts to rank pages based its analysis of the pertinence of the keyword(s), those page rankings can also be affected by other factors that Google uses to rank the page, such as the number of other sites linking to it.

If you know your customers are interested in one or more particular topics, and the best information on those topics is consistently found on some well-defined list of web sites, then providing a spider to track those sites and highlight the pertinent information for them (either as a search or an RSS feed) will not only perform a valuable service for them, but give them a powerful incentive for regularly visiting your site. PCV Software can help you write the spider program and set up your web site to incorporate the data it gathers to best serve your customers and your business goals.