Top Reasons Why Your website May Not Be Indexed and listed with search engines

At one time or another you may have used a submission tool, or submitted by hand and then wondered why you had not been indexed. Unfortunately, there are many reasons that may delayor prevent you from being indexed by a search engine. There's rarely one simple answer for why you're not being found. Fortunately, there is generally an explanation and a way to correct the problem if you know what to look for.

Search engine traffic is a win-win situation for any online business. It doesn't take much to improve on most web site's search engine traffic - it just takes optimization. Did you know that as of January 1 2002, there were 160,000,000 domain hosts in use worldwide? Did you know that 88% of the web pages worldwide are not indexed by the largest search engines? 88% of web pages are not optimized. How can any business survive on the Internet if it is not optimized for the search engines, and thus can never be found? How can they exist if over 83% of first time visitors never find their web site? How much more money could a web site operator earn if they ensured their web site was even partially visible?

Below are the Top 21 reasons we've compiled over the years as to why you may not be finding your Web site or Web page in one or more search engines:

1. INDEX TIME: First, make sure you've allowed enough time to become indexed. The amount of time to allow is sometimes listed on the search engine's submission page. Unfortunately, the engine's own advertised times are often inaccurate or out of date. WebPosition's Submitter report and WebPosition's URL Verification report will both tell you how much time you should allow before being concerned about not being indexed. Average index times often range from one to eight weeks depending on the engine. Some engines like AltaVista and Inktomi offer paid options if you wish to be indexed more quickly.

2. ALREADY INDEXED: Be sure you're not already indexed but just don't know it. Unfortunately, none of the major engines are kind enough to e-mail or notify you as to if and when you've been indexed.

In addition, you cannot simply do a search on a keyword that applies to your Web site and expect it to pop up at the top. In fact, you must take proactive steps to optimize your pages for each search engine. If you don't, it's very unlikely you'll find your Web site except on the most specific of searches.

The method to determine if a page or domain has been indexed varies from one engine to another, and in many cases, it's difficult to tell for sure whether your pages are in fact in there. Never assume that you're not indexed just because you searched for a bunch of keywords and you never came up in the first few pages of results. You could be there (i.e., indexed) but be buried near the bottom.

In addition, it's not very practical to check the status of a number of pages on each major engine each week. Fortunately, WebPosition Gold has a URL verification feature in the Reporter that makes this process much easier. Each time you run a mission, it will report which URLs exist and do not exist in each engine. If you're using WebPosition and are not finding your URLs after submitting, be sure to see this page for common pitfalls to watch out for:

http://www.webposition.com/urlnotfoundhelp.htm

3. MISSING PAGE: Make sure you have uploaded the pages to your site before submitting them. This one will seem obvious to many people, but submitting a page that does not exist or submitting with a subtle typo in the URL is a goof we might all make at one time or another. If you're using WebPosition's Submitter, there's a checkbox on tab 2 labeled "Verify that each page exists on Web site before submitting." This option defaults so that WebPosition will verify that all your URLs are valid and actually exist before submitting them. This is important since not all search engines will notify you if the URL does not exist when you submit.

4. ROADMAP FROM HOME PAGE: Some engines have been known to drop pages that cannot be traveled to from the home page. HotBot has been rumored to do this. You may want to consider submitting your home page that links either directly or indirectly to your doorway pages. Think of your Web site as a series of roads (i.e., links) from one page to another. If there's no road from your home page to the page you want indexed, a search engine may decide the page is unimportant or of low-quality. You could submit the page directly, but the engine may reject it or may drop it at a later date when it finds no "road."

5. EXTERNAL LINKS: Some search engines such as Google and HotBot have been known to refuse to index Web sites that do not have any other Web sites linking to them. Or, they may index your home page but refuse to index any other pages until you achieve at least one or more links from another domain. Or, they may index you for awhile but then "prune" their database later of all Web sites that did not achieve any external links within a certain period of time. However, do not worry! You simply need to establish some links and when that's done, resubmit both your pages and the pages that link to you. Once you have links to your Web site, it becomes much easier to get indexed, stay indexed, and to achieve top rankings.

To check your link popularity and for tips on how to increase the links to your Web site see our free service at: http://www.linkpopularitycheck.com

6. FRAMES: If you have content inside HTML frames, this can cause problems with submissions. For example, the search engine may index the main content of the page, but not the surrounding menu frame. Visitors to your site will then find some information but may not see the associated menu! It's generally best if you can create non-framed versions of your pages. You should then submit the non-frames versions of your pages, which can of course link to your framed Web site. Alternatively, you can enter your relevant text within the NOFRAMES area of a framed page that most search engine spiders will read. However, don't expect to achieve high rankings while optimizing the NOFRAMES area. Optimizing a NON-framed page will often achieve better results.

7. SPIDER BLOCKS: Search engine spiders cannot index sites that require any kind of registration or password. A spider cannot fill out a form of any kind. The same rule applies regarding indexing of content from a searchable database. That's because the spider cannot fill out a form to query that database. The solution is to create static pages that the engines will be able to find and index without performing a special action on your site. Depending on the database system you have, there are utility programs out there that help you do this, as well as companies that can assist you.

8. FREE SITES: Many engines no longer index pages from free web sites or they limit the number of pages they will index from these hosts. Sometimes they will get too many "junk" submissions from free web site domains such as Geocities or others. Therefore, some engines choose not to index anyone from some of these domains. Or, more commonly, they limit the number of pages they will accept.

It's always best to buy your own domain name (very important) and place it on a respected, paid service to avoid being discriminated against. The free traffic you can generate from the search engines is just too valuable to be sacrificed for the small savings a free hosting service provides. In addition, free hosts are often unreliable or force you to display banners that send valuable visitors away from your Web site soon after arriving. That can cost you sales.

9. GUILT THROUGH ASSOCIATION: If your Web site shares the same IP address as many other Web sites on your host's Web server, then you may find your IP quietly banned from something another Web site on the same server did! It's always best to ask your hosting service if your domain name has its own unique IP assigned to it. If not, ask them to move it to its own IP to avoid the potential of having your submissions ignored because of something that a site sharing your IP did. We've heard from many people who tried everything to be indexed only to find it was a snap once they changed hosting services.

10. SUBMISSION LIMITS: Make sure you're submitting within the recommended limits. Some engines do not like more than a certain number of submissions per day for the same domain. If you exceed the limit, you may find that all your submissions are ignored. Fortunately, WebPosition's submitter will warn you regarding current limits and help keep you within them. Some submission consultants feel it is dangerous to submit more than ONE page a day to an engine for a given Web site. For those who wish to be ultra-conservative in their approach, the WebPosition Submitter includes a checkbox to limit submissions to one URL per day per engine.

11. DYNAMIC PAGES: Dynamic pages are often ignored by the search engine spiders. In fact, any URL containing special symbols like a question mark (?) or an ampersand (&) will be ignored by many engines. Pages generated on the fly from a database often contain these symbols. In this situation, it's important to generate "static" versions of each page you wish to be indexed. In regard to the search engines, the simpler the page is, the better. Does this mean, for example, having a javascript to count visits to the page will prevent you from being indexed, or lower your rankings? No. It simply means that the search engine will most likely ignore the javascript and index the remaining areas of the page. There is evidence that going too far with fancy scripts and code on a page can hurt your rankings if the bulk of your page consists of java or VB scripts.

12. NON-INDEXABLE CONTENT: It's important to know the types of content that the average search engine cannot index. Most engines cannot index text that is embedded in images. Text that appears in multimedia files (audio and video) will not be indexed. Most engines cannot index information that is generated by Java applets or in XML coding. 13. LARGE PAGES: If your site has a slow connection or the pages are very complex and take a long time to load, it might time out before the spider can index all the text. For the benefit of your visitors and the search engines, limit your page size to 50K or less. In fact, most Webmasters recommend that your page size PLUS the size of all your images on the page should not exceed 50K-70K total. If it does, many people on dial up connections will leave before the page fully loads.

14. DEEP LINKS: If you submit just your home page, don't expect a search engine to travel more than one or two links away from the home page or from the page that you submitted. Over time they may venture deeper into your site, but don't count on it happening quickly. You'll often need to submit pages individually that appear further down into your site or create more direct links from the home page (either visible links or hidden links). This way the search engines can find them. Visible links are preferable, but when that's not practical, there are methods to create links that won't be seen by the average visitor:

http://www.webposition.com/hiddenlinks.htm

The technique of submitting one page that then links to multiple other pages you want found is called creating a "hallway page." In many cases, this will not only get you indexed in cases where they are ignoring you, but it will often improve your rankings. That's because many engines assign "bonus points" to pages they find on their own versus pages that were specifically submitted to them.

15. UNRELIABLE HOSTS: If your Web site fails to respond when the search engine spider pays a visit, you will not be indexed. Even worse, if you are indexed and they pay a visit when your site is down, you could be removed from their database! Consequently, it pays to have a reliable hosting service that is up 99.5% of the time. However, at some point a spider is going to hit that other 0.5% and end up yanking your pages by mistake. Therefore, it pays to keep a close eye on your listings and resubmit when needed.

16. SPAM: If you have ever used any questionable techniques that might be considered an overt attempt at spamming (i.e., excessive repetition of keywords, same color text as background, or other things that the WebPosition Page Critic warns you about), an engine may ignore or reject your submissions. If you're having trouble getting indexed in the expected amount of time, make sure your site is spam-free.

17. REDIRECTS: If your site contains redirects or meta refresh tags these things can sometimes cause the engines to have trouble indexing your site. Generally they will index the page that it is redirecting TO, but if it thinks you are trying to "trick" the engine by using "cloaking" or IP redirection technology that it can detect, there is a chance that it may not index the site at all.

18. PROPER DIRECTORY SUBMISSIONS: If you're submitting to a directory site like Yahoo, Open Directory, Looksmart, or others, then a human being will review your site. They must decide if the site is of sufficient "quality" before they will list it. I recommend you read the submission guide on the directory tab of the WebPosition Submitter. It contains tips on how to improve your chances of obtaining a good listing in these directories. Getting listed in major directories first can help you get listed elsewhere.

19. INDEX TIMES CAN FLUCTUATE: WebPosition will tell you the average index time of each search engine. However, this is only an average. Sometimes engines will index sites every 30 days fairly consistently and then suddenly stop indexing most sites for several months. This can be frustrating, but it does happen. Generally a major engine will not go more than three to four months without refreshing its index. If you're wondering if others are experiencing trouble getting indexed on a particular engine, try asking around. You can often find valuable real-world feedback on a search engine discussion forum such as the one at:

http://www.marketposition.com

20. PAGE LIMITS: If you have many pages indexed but are having a hard time getting new ones recognized, be aware that there are limits. Each search engine will only spider so many pages of your Web site. This may range from a few dozen or three or four hundred depending on the engine. Some people have even been successful in getting far more pages indexed depending on the engine. Google is one engine that tends to crawl deeper into your site. However, how deep they go may depend on factors like your link popularity. Sites with higher link popularity are deemed "worthier" of more thorough indexing.

21. RANDOM ERRORS: Last but not least, sometimes the engines just lose submissions at random through technical errors and bugs. After all, they are managing a database of hundreds of millions of pages. Therefore, some people like to resubmit once or twice a month for good merit in case they do lose a submission. Certainly if you've followed all the "rules" and are still not listed, by all means, re-submit! Sometimes a little persistence is all that's needed.

TIP: Once your page achieves a desirable ranking, it's best not to continue submitting it. You risk the engine re-evaluating the page and possibly reducing your ranking.

If any of the above scenarios apply to your submission, you should take the appropriate actions and then re-submit. If that still does not work, you should consider e-mailing or calling the search engine and asking them politely why you have not been indexed yet. Sometimes they will reply back with "Sorry, there was a problem with our system and I've now made sure you'll be indexed within the next couple days." Or, sometimes they'll tell you why you were not indexed so you might correct it. In other cases, they will ignore your e-mail and you'll have to keep e-mailing or calling them until they respond. Still, it's definitely worth the effort to get your site listed with the major engines. This assumes you also take the time to optimize your pages so you'll achieve top rankings.

Consider the following testimonial we received last month in response to an offer to renew a WebPosition knowledge base subscription. Sometimes you can have too much of a good thing I guess...

======================================= Date: Thu, 18 Dec 2001 From: webmaster@centralhome.com To: jim@firstpalcesoftware.com Subject: WebPosition

Hi Jim,

At this moment, I am going to have to pass on your most generous offer. Why? It works too well! I did have a 1 year subscription and optimized a lot of pages. So, what happened - sales started to increase 10% to 20% month over month. Now I am up to my ears in work! I had to get a better server to handle [the] increased hits. I am afraid if I did this again, sales would increase 20% or 30% and create more work.

Best regards, John Barendrecht http://www.centralhome.com http://www.musicridge.com =======================================

If you'd like to have the same problem John is faced with and have not yet purchase your own copy of WebPosition Gold for just $149, see:

https://www.webposition.com/wpg-easyorder.htm

The product is backed by a money back guarantee because we know it works when used properly. We also want you to be 100% happy with your purchase.

Yahoo Pricing Change ~~~~~~~~~~~~~~~~~~~~

By now readers of the MarketPosition newsletter are well acquainted with iProspect, the leading Boston-based search engine positioning services firm. Their CEO, Fredrick Marckini is an occasional contributor to this newsletter and author of the book "Search Engine Positioning." Fredrick alerted us to this important development in Yahoo's Express Service pricing, so we wanted to give credit where credit was due. Here is what we found:

Remember just a few short years ago when Yahoo was eager for your submissions and encouraged you to add your URL to their directory? And just as soon as we all became accustomed to Yahoo editors nipping and tucking our carefully crafted descriptions, Yahoo began charging a "Business Express" submission fee - to cover the cost of their editing and to reduce submission times from eight weeks to seven days. Now, Yahoo has announced their intent to make the $299 submission fee an annual recurring charge instead of a one-time fee. Now, you must pay not just for editorial review, but for continued inclusion, year-after-year.

As this excerpt below taken from Yahoo's new submission page shows, Yahoo! has indeed put a "recurring annual fee" in place for any site submitted after December 28th 2001. What this means is that not only do you have to pay the $299 submission fee to get your listing into Yahoo!, but if you want to keep it in their directory, you have to pay $299 every year after that. If your site was included in Yahoo prior to December 28th, then these reoccurring fees are waived. To quote from Yahoo "your payment only guarantees that Yahoo will consider and respond to your request within seven business days, by either accepting or not accepting your site. You expressly agree to pay to Yahoo such fee whether or not your site is accepted or denied inclusion in the Directory. If your web site is accepted for inclusion in the Directory as part of Yahoo Express on or after December 28, 2001, then your web site's continued inclusion in the Directory will be subject to additional review each year and you agree that your credit card will be charged the Recurring Annual Fee" (http://docs.yahoo.com/info/suggest/terms.html).

Your Yahoo listing is now like your domain name purchase with an annual recurring cost. Note that for your annual fee you do not have the option of making annual changes to your listing. This annual fee is just for continuation in the directory at Yahoo's discretion. Be advised that if you choose to submit a change request using the Yahoo! Express, you will be immediately charged the current initial consideration fee, and the changes may or may not be made.

If you've kept up with the news, you know there's been a significant decline in online advertising revenues for Web-based companies. However, the awareness of the power of search engine listings as an effective advertising channel has grown in recent years. Therefore, the new fee structure arguably makes good business sense for Yahoo. Given the huge number of sites added to the directory each year, Yahoo will have established a substantial reoccurring revenue stream helping to ensure its economic health and continuance. Given the seismic shifts that are occurring in the search engine landscape (i.e., Excite's recent bankruptcy), this annual fee seems less onerous than some of the other models that are appearing.

Unfortunately, what may make financial sense for Yahoo is not equally good news for Webmasters trying to advertise new Web sites on a budget. Yahoo knows that the advertising exposure from a good listing on Yahoo is worth more than $299/year because of their huge traffic volume. That's why they're charging the annual fee. In reality, it has nothing to do with them being able to afford to "review" your site as they imply. The cost of the actual "review" does not come close to $299. However, the other overhead of running the site and attracting millions of visitors does cost some serious change.

Since the stakes have risen, it's all that much more important to get your Yahoo submission right the first time. There's many things you can do to stack the deck in your favor. Unfortunately, if you don't do it right, you can get stuck with a listing that ranks poorly. For tips on submitting to Yahoo (along with other major directories), see the Directory Submission Guide in the WebPosition Submitter.

WebPosition Free Trial Download: http://www.webposition.com/download.htm

To Order WebPosition Gold: https://www.webposition.com/wpg-easyorder.htm

Excite Becomes Clone of Overture ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In last month's newsletter, I mentioned that Excite.com after declaring bankruptcy was purchased by InfoSpace. InfoSpace has now converted Excite's search engine so that it simply searches the Overture database. This has been a trend for search engines that find themselves in financial trouble.

Since Overture is already supported by WebPosition Gold, maintaining separate rank reporting for Excite is now redundant. Therefore, although visitors will still be broken out separately in the WebPosition Traffic Analyzer, Excite.com has been dropped from the Reporter module.

The good news is that this should speed up your WebPosition missions and allow you to focus more effort on the remaining major search engines. You should use the extra time-savings to create additional optimized pages for the other major engines and thus increase your traffic from those sites. If other engines grow in popularity, we'll look at adding them to WebPosition Gold. However, as always, it's important that you do not waste your time on engines that are not large enough to bring reasonable amounts of traffic to your Web site.

Many of you may be wondering whether Overture (formerly Goto.com) will eventually replace all the major search engines? Don't worry. Although their influence may continue to grow, they will not replace major search engines like Google. Although the pay per bid (PPC) model on Overture makes their stockholders money, it is not a model that fulfills the needs of many Web surfers. Many types of searches do not yield good results on a PPC based engine.

Therefore, as we have seen with Go/Infoseek and NBCi when they converted to an Overture clone, many regular visitors will go elsewhere to meet their search needs. This can make some of the remaining players like Google, Lycos, AllTheWeb, and others more popular. However, it might also help open the door for new engines to rise in popularity if market gaps are left open. The search engine landscape changes continually as it has since day one. The key is to stay educated so you can use the current environment to your advantage.

Advertise in MarketPosition and Reach 430,000 People ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you're interested in advertising in MarketPosition(tm), the most popular search engine marketing newsletter in the world e-mail adam@firstplacesoftware.com for current competitive rates and information, or call 1-800-962-4855 x702.

Webmasters, marketers, and business owners around the globe read this high quality newsletter and depend on its advice for promoting their businesses online. Whereas many low quality newsletters tend to get deleted as soon as they arrive in a person's inbox, MarketPosition subscribers look forward to reading the in-depth and invaluable content offered each month.

LAST MONTH: ~~~~~~~~~~ Last month we talked about several important topics including: - Financial Times Of London Recognizes The Importance Of Search Engine Positioning - Company Lands $1.5 Million Dollar Deal Due to Search Engine Positioning Effort - Free Update: WebPosition Gold 1.60.4 - Even Faster and More Robust - AllTheWeb Becomes The World's "Freshest" Search Engine - Excite@Home Bankrupt -- InfoSpace Snatches Up Excite.com

If you missed these or other key discussions, you can find the back issues at:

http://www.webposition.com/newsletters.htm

ABOUT THE AUTHORS ~~~~~~~~~~~~~~~~~

MarketPosition is written by Brent Winters, President of FirstPlace Software, with occasional editing and contributions by Fredrick Marckini, CEO of iProspect.com, Inc.

OTHER RESOURCES: ~~~~~~~~~~~~~~~

FirstPlace Software, Inc. produces several products, including WebPosition, the first software program to report your search positions on the major search engines and to help you improve those positions.

You may download a FREE trial of WebPosition at:

http://www.webposition.com

You may call us at 1-800-962-4855 if you have questions not addressed on our site. You will also find an array of additional tips and techniques for improving your search positions in both the WebPosition Help File and the Reports it generates.

(c) copyright 2002 FirstPlace Software, Inc.