DevPub
Free Mousepad

How Google Works

How Google WorksA look at the myths and facts about Google, and what you should know about Google's inner workings. Continue Reading

Search Engines and You: Getting Crawled

Search Engines and You: Getting CrawledHow to get your newly launched web site to be indexed by search engines and how directories can help you. Continue Reading

Search Engines and You: Getting Crawled

Understand how search engine crawling works, how to get your website indexed by search engines and consequently crawled and a list of the directories you should submit your links to.

On Sunday, March 2nd 2008 at 12:24 AM
By Louis Fernandez (View Profile)
-----   (Rated 0 with 0 votes)

Most pages that get submitted to search engines are spam. There is no guarantee that your site will get included for free just by submitting it. The best way to get listed is to build relevant inbound linkage data pointing at your site.

Yahoo! is currently the only majore search engine to offer a paid inclusion program through Overture. The Overture paid service powers all of the Yahoo search properties. In addition to a $49 inclusion fee, they also charge a category based price per click. Usually paid inclusion is not worth it for most sites.

The best way to submit your site to search engines is by having them find links into your site from other sites. There is no need to submit or resubmit your site to search engines.

 

Directories vs. Search Engines

Search engines are operated by scripts and machine code. Directories are human compiled lists of sites organized by categories. Since directories are entirely human edited they take a ton of time and effort to maintain. Whenever I create a new site and I am happy with it I submit it to many of the directories. When submitting to directories it is worth it to spend the extra time to ensure you are in the correct category and are following the directory guidelines.

 

Submitting Your Site

Submitting to Search Engines:

You may want to pay to submit your site, but most search engines will list your site free. The best way to get your site indexed is through having a search engine follow a link from another site. Toward the end of this book I will also address another option called paid inclusion.

Where to Get Links:

• Directories may list your site after you submit it.
• You can exchange links with similar web sites.
• Writing articles about your topic and placing them on other websites can give you inbound links via the article signature.
• Writing press releases can give you inbound links.
• People interested in your site may eventually link to you without you asking.
• You can participate in forums which provide signature links.
• A somewhat recent move which has been exploited is posting comments directed toward your website in weblogs.

How often do Search Engines Crawl?

Search engines constantly crawl the web. Pages that frequently update with strong link popularity may get crawled many times each day. Pages that do not change that often and / or have little link popularity may get crawled only once or twice a month. Since search engines are constantly adding content to their index they are in a constant state of flux.

How Search Engines Evaluate Links:

In the eyes of a search engine, you usually can not control who links to you, but you can control who you link to. In most cases if bad sites link to you it does not hurt you. If you link back, it does. So in essence it usually does not hurt you to get inbound links from anywhere. You should be rather selective with who you are willing to link out to though.

Blogs and Weblog Comment Spam:

I recommend finding a weblog with posts about a recent news item you are interested in and posting a comment linking to your site with your real name in it. The whole point of weblogs is community discussion, so it is not spam to add something useful and link to your web site from it.

What people talk about when they say comment spam is when people post crap like "nice site" with "BUY VIAGRA" in the link. Already software has been made to slow the effectiveness of this technique.

Some weblogs have inline comments (or comments which appear on the same page as the original post). Posting comments on Movable type weblogs can get your website indexed in no time so long as the links are static links. Static links are links that do not go through redirects.

Again I am not stating that I suggest spamming weblogs, but you can get your site quickly indexed by leaving relevant comments and participating in the web as a community. That is what the web is about anyway, right?

Rel=”NoFollow”:

Many of the major search engines and blog software vendors came together to make a nofollow tag. The nofollow tag allows people to leave static links in the comments and trackbacks which search engines will not count for relevancy.

Essentially the tag is designed to be used when allowing others to post unverified links into your site. You also can use it if you are linking out to shady stuff as an example but do not want to parse any link credit to the destination URL.

Many webmasters will likely be a bit sneaky and create fake blogs and then spam their own blog with links off to high margin website affiliate programs.

The nofollow feature looks as follows:

<a href=“http://www.fgfgsgqf.com” rel=“nofollow”>Link Text</a>

People will still continue to run spam bots to spam blog comments. Many blog owers will see their rankings drop hard since many of their old comment links will no longer help boost their own search relevancy scores.

The rel=”nofollow” tag may make it easier for many webmasters to cheat out reciprocal link partners.

The WikiPedia was the first major non blog site to adopt the nofollow tag and search engines may find in due time that the tag has many unintended negative consequences.

Chat Forums:

In chat forums people asking and answering questions create free content for the person who owns the site. This automated page creation allows the forum owner to sell a ton of advertising space.

In exchange for the posts many SEO forums allow signature links that point at your website. Since forums change rapidly they often get indexed frequently. This will help your site get indexed quickly if you ask a few questions at a few of the various SEO forums.

Of course the goal of chat forums is to have meaningful conversations, but if you are reading this ebook odds are that you may still have some questions.

Forums have many links on the pages though, so the links probably do not have a large effect on SEO. Forum sig links may have more direct value in driving sales than in effecting search results.

 

Directories worth Getting Links In:

In general I think it is worth submitting your site to directories, but submitting your site to search engines is a waste of time. The two most popular directories are DMOZ and the Yahoo! Directory.

The Open Directory Project:

The Open Directory Project (DMOZ) is free, but sometimes it can take a while to get listed. DMOZ editors work free of charge and are under no obligation to list your website.

Ensure you take the time to submit your site to the right category and follow their directory guidelines. If your site is not in English make sure you submit it to the world category. Regional sites should be submitted to their respective regional category.

With the ODP you do not need to keep resubmitting over and over. If for some reason your site can not get listed after 30 days ask at the Resource Zone and inquire about your site every 6 months thereafter. You may want to apply to become an editor if you really enjoy your category. As with your site submissions you should take your time when applying to become an editor.

The Value of a DMOZ Listing:

The Open Directory Project is syndicated by many other sites and inclusion into it often provides your site with dozens of inbound links. Many people are quick to state that the Open Directory is worthless or that it is super important.

The fact is that it is fairly important for some sites and fairly unimportant for others. There are a ton of variables that go into the value of a listing. I usually just submit and forget about it. I do not find that it helps a bunch to be preoccupied with a DMOZ listing. Many high ranking sites are listed in DMOZ, and many high ranking sites are not.

The Yahoo! Directory:

You will still list in Yahoo! powered search results even if you do not submit your site to their directory. Yahoo! charges a $299 recurring annual fee for commercial sites (double that for adult sites), which is a bit expensive for many newer smaller sites. Generally I recommend paying for placement in many second tier directories before paying for placement in the Yahoo! Directory since most of the second tier directories only charge a one time site submission fee.

Unlike most directories, Yahoo! recently shifted their directory to list sites in order of popularity instead of alphabetical.

Non commercial sites can list in the Yahoo! Directory free, and I can attest to the fact that they have listed multiple sites I own for free.

When a site gets submitted to the Yahoo! Directory an editor quality checks the site. Since Yahoo! controls their own directory it would be logical for them to place extra weighting on a Yahoo! Directory listed site. Some top SEOs have told me they have seen significant increases in their Yahoo! Search rankings after submitting a site to the Yahoo! Directory.

Regional Yahoo! Directories:

Some of the regional Yahoo! directories only charge a one time fee. Some of them charge no fee at all. If you site is specific to only one region you may want to check the local prices and see if it is a better deal to submit to the local directory instead of the global one.

Second Tier Directories:

Directories such as Gimpsy, GoGuides, RubberStamped, Uncover the Net, SevenSeek, JoeANT, Web Beacon, MSN Small Business Directory, and Skaffe all cost less than $50 to submit to.

JoeANT is free if you become an editor, and it only takes a couple minutes to sign up. Gimpsy is free if you are willing to wait a few months. Skaffe is free for editors. GoGuides has a bulk submission discount program. Wow Directory is another directory which has been providing free site submission.

If you are going to list your sites in many directories you may be able to save time by using Roboform to save some of your submission details.

Mix Things Up!

When links and citations occur naturally there is no 100% easily definable pattern. If something is easy for a search engine to do and it will improve search quality they probably will do it.

Make sure you mix up your anchor text and your site descriptions so that there is no easily identifiable unnatural pattern.

If you start directories yourself and you use common default directory software you may want to look to remove common footprints the script leaves so that other sites which may be abusing said script do not cause your site to be filtered as well if a search engine decides to penalize sites which are using a commonly abused script.

Industry Specific Directories:

Business.com is a business directory which costs $299 annually to list your site. Many search engines may not follow the Business.com links, but it might still be a good buy for B2B type businesses.
Microsoft also has a small business directory which is a bargain at under $50 per year. There are also many industry specific directories you can find by searching for terms such as “my keywords + add url,” “my keywords + submit,” or “my keywords + directory.”

I usually try to find directories which have one time submission fees or directories which look as though they are going to be longstanding directories.

Tips to Pick Directory Categories:

Often times a site will fit many different categories. When choosing a category to submit to in a directory I look at a few different ideas.
• Is my site likely to be accepted if I submit to this category?
• Are there reasons this organization or other sites outside of this organization are likely to place extra emphasis on (or otherwise link into) this category?
• How many links are listed in this category?
• Where does this category fit in the directory structure?

Reasons I Like Second Tier Directories (Great Value):

Since smaller directories are smaller usually your link is closer to the root page and usually most pages have less outbound links in smaller directories than in large directories. Thus you may gain greater link popularity in a smaller directory if your categories in a larger are full of hundreds of sites or are many levels deep into the directory structure.

Directory Traffic:

Directories rarely provide much direct traffic. The bulk of the value of a directory listing is in how search engines will evaluate the links. Ocassionally you will find a directory that does provide good traffic – that is the exception more than it is the rule.

Reciprocal Link Required:

Some directories require reciprocal links to be listed in them. I do not recommend swapping links with most of these types of directories. Directories are intended to list sites. Sites are not intended to list directories. If you like something then feel free to link to it, if not then don’t.

Link popularity is a currency and if you are lacking money (as I was when I started on the web) you may need to reciprocate a few links off the start. Make sure the directory you want to swap with is not shady before swapping links.

The exceptions to this rule are that I am usually willing to reciprocate links with:

• extremely powerful sites that I do not believe are going to get penalized for aggressive link exchange
• directories which are well focused and are defined as an industry hub in my topic

Directory Warnings:

Some sites that pose as directories do not provide static text links and / or their pages do not get indexed. Many of these databases will never provide any direct traffic or link popularity. Additionally many directories require reciprocal linking and use their sites to promote aggressive high margin products. If you link into sites that primarily promote high margin items then you are sharing the business risk that site owner is taking.

If you are to spend money on directory submission you should ensure that the directory provides direct traffic or link popularity. You can ensure it helps your link popularity by ensuring their directory pages have some PageRank on them, are in Google’s cache (search for cache:www.whateverpage.com in Google), and check the links of listed sites to ensure they are static text links. When you scroll over a link in the directory the status bar at the bottom should indicate the domain that the link is pointing to and not some sort of redirect.

• You can check PageRank by downloading the free Google Toolbar.
• To ensure a page is not showing phantom PageRank you can check to make sure the page is in Google’s cache. Search Google for cache:www.whateversite.com/whateverpage.com.
• To ensure the links are indexed by search engines you want to make sure that when you scroll over a listing in the directory that the status bar at the bottom of the browser shows www.whateverlisting.com. A few good directories happen to show some funky characters (Yahoo! Directory & Business.com). Yahoo and Business.com are about the only two directories that I know of that show funky characters and still provide static text links. Most directories that show some funky tracking characters are not providing static spiderable links. If in doubt ask questions at SEO forums before spending any money.
• Some redirect links do get indexed, but there is no simple litmus test to ensure that they do get indexed. You can do right click on and copy links from within the directory and do a server header check on them. If they show a 301 redirect they will probably add to your link popularity. If they show a 302 redirect they may add to your link popularity. If they show a Java redirect then they do not count. When in doubt if a link counts ask in a couple various SEO forums.

Places to Find Directories:

I created the Directory Archives which primarily should only list directories which parse link popularity to sites listed in them or directories which look like they might drive traffic to listed sites. In addition SearchEngineGuide and ISEDB each have a large directory of directories (though many of the directories listed in those may not parse link value).

Ensure that the pages links are on are in Google’s cache and that the address bar show the location of the site the link is going to before paying for placement in any directory.

The Value of Directory Listings:

Most directory listings do not provide a ton of value by themselves. What makes directories powerful is when you list your site in many independent directories.

If you have a keyword rich domain name it will help you get descriptive inbound links from directories. Most sites on the web only have links from a few dozen sites. By listing your site in a few dozen directories and also getting links from other sites you can quickly build up a great linking campaign at minimal cost.

Google Ignoring Some Directories:

Some directories have recently been removed from Google’s cache. Additionally some of them have not had their cache dates updated in a great deal of time. Google might be trying to slow the growth of directories by not counting the links from many of them. Make sure you check the cache before paying for a listing.

Some of the directories will have a greater effect on relevancy in MSN or Yahoo! than they do on Google so even if a directory is not counted by Google the link price might still be cheap for its effects on other search relevancy algorithms.

Many directory owners are building multiple related directories. Some search algorithms such as Google Hilltop will likely automatically filter out some of the relevancy score from a second directory which is identifiable as being related to the first directory.

The one time listing fees make directories exceptionally appealing, but do expect that some of the directories will eventually get hit by some of the search engines.


Rate Rate this article

Comment Comment on this article
Name: Email:
Message:

Comment Current Comments