Tired of surfing the Internet searching for relevant web pages? Wouldn’t it be nice if you had a personal, customizable web crawler that could automatically index or collect specific pages without you having to click on page after page? That’s exactly what Visual Web Spider does with very little effort on your part!

Visual Web Spider is a fully automated, multithreaded web crawler that allows you to index and collect specific web pages on the internet. Once installed, it enables you to browse the Web in an automated manner, indexing pages that contain specific keywords and phrases and exporting the indexed data to a database on your local computer in the format of your choice. No special knowledge or skills are required to get started with this crawler.

Let’s say, for example, you are a shareware developer interested in internet marketing. You need to find all web pages at www.marketingexperiments.com that contain such keywords as “Google Adwords” and “PPC marketing”. Or you need to crawl all the pages of the website, index and download document files (pdf, doc, xls) or audio files (mp3, wma) or video files (mpeg, avi) to your computer's hard drive. Or you may want to collect website links to build your own specialized web directory. You can configure Visual Web Spider to automatically do this for you.

This program’s friendly, wizard-driven interface lets you customize your search in a step-by-step manner. To index relevant web pages, just follow this simple sequence of steps. After you open the wizard, enter the starting web page URL. Or let the program generate URL links based on specific keywords or phrases. Then set the crawling rules and depth according to your search strategy. Finally, specify the data you want to index and your project filename. That’s pretty much it. Clicking on ‘Start’ sets the crawler to work. Crawling is fast, thanks to multithreading that allows up to 50 simultaneous threads.

Another nice touch is that Visual Web Spider can index the content of any HTML tag such as: page title (TITLE tag), page text (BODY tag), HTML code (HTML tag), header text (H1-H6 tags), bold text (B tags), anchor text (A tags), alt text (IMG tag, ALT attribute), keywords, description (META tags) and others. This program can also list each page size and last modified date.

Once the web pages have been indexed, Visual Web Spider can export the indexed data to any of the following formats: Microsoft Access, Excel (CSV), TXT, HTML, and MySQL script. This variety of allowable export formats lets you to process and analyze data in a format convenient for you.

Many search engines use web robots to gather web pages for indexing. They utilize this technology to update their content on a regular basis. Researchers use it to widen their perspective of the internet and its vast collection of data. Now you can have your own web crawler and use it to be more productive in your home or office.

Key Features:

• A Personal, Customizable Web crawler. Crawling rules. Multithreaded technology (up to 50 threads). Support for the robots exclusion protocol/standard (Robots.txt file and Robots META tags);
• Index the contents of any HTML tag. Indexing rules;
• Export the indexed data into Microsoft Access database, TEXT file, Excel file (CSV), HTML file, MySQL script file;
• Start crawling from a list of the URLs specified by user;
• Start crawling from a historical list of the URLs;
• Start crawling using keywords and phrases;
• Store web pages on your local disk;
• Auto-resolve URL of redirected links;
• Auto-remove duplicate or invalid syntax URLs;
• Filter the indexed data;
• Command line options;
• Generate and export map of the visited links;
• Very simple to use, quick learning curve and right to the point.
  

Screenshot: