6. Crawling Search Engines and Why You Want Them
There is some truth to the old adage about learning to crawl before you run. In the case of your website, the more you get hit by search engine crawlers, the better your chances at increasing your traffic (and your sales!)
What a Crawler does:
Google and Teoma are good examples of crawler-based search engines. They send out 'bots that "crawl" (also called "spidering") the web, reading the pages on your site, adding those pages to that search engine's database (note that they might not crawl ALL pages n your site, so make sure that you have the Really Important Information fairly close to the top level of your pages and not buried deep in sub-sub-subpages.) As we write this, the battle is on between Google and Teoma to see which can spider the most pages the quickest, but they are not the ONLY search engines on the web using spidering technology, just the biggest.
Here's how web crawling search engines work to your advantage.
Once your site is listed in a crawling site (either by a free submission - you'll have to wait a while to get listed and there's no guarantee you'll get listed then - or by a paid submission) your site will first get read by that engine's web crawler to index the content of your site. The more text-based content you have HIGH up on your site (first and second level pages,) the more info you'll have on the search engine (and the easier you'll be to find.)
Second, as you change the data on your site (add new content or take away content that no longer interests you, or change the links going to other sites) the information will change as the search engines refresh their databases by re-indexing your pages.
We don't know an exact schedule for WHEN your site would be crawled, but they need to keep their information as fresh as possible to keep people using their search engine. Unless you are using paid submissions with guaranteed timely updates, plan a 4-6 week cycle before the information you have on a search engine would change. As the battle for the #1 slot continues between the two big players, that timeframe will narrow, so that changes to your pages will be reflected sooner in the search results. AND - the more "relevant" your page is (we discussed this in #2 - "Content is KING") the more often your site will be re-indexed (do you see a connection here between high quality content and better placement in the search engines?)
The moral of the story is that getting listed in the search engines is not the end of your work. Once listed, you have the opportunity to IMPROVE your placement in the search engines by making changes to your pages which the crawlers will pick up the next pass through. Make sure the content on the page matches the key search engine terms you are focusing on for your site.
Keep in mind that the average search engine user won't go through more than the first three pages of a search results before they give up and try another search, so you want to do everything that you can to make sure YOUR pages are in those first 3 pages. Once listed in a search engine, you'll switch gears from submissions to optimization, and start working the content on the pages (see #2: "Content is KING") to move yourself higher on the search results for your site's top search terms.
