When you’ve been in the search marketing game for a few years, you tend to add a few interesting tools to your arsenal. Xenu has always been one of my favorites. No, I’m not talking about that Xenu (although the program icon does have an interesting alien motif). I’m talking about Xenu Link Sleuth, a useful site crawling application.
The Xenu Link Sleuth website describes the program as a link checker, a job which it does quite well. However, with a little know-how, it can easily be used to diagnose a number of crawl errors. If you spend an afternoon fixing the bugs you discover through Xenu, your site will be in great shape as far as search engine visibility is concerned. Here are some of the fields to consider.
Address – This column lists every URL available through links on your site. Sort it and look for duplicates. In particular, ensure that you’re not duplicating www and non-www or http and https versions of the same page. Also make sure that only one URL is in use for your home page (e.g., you don’t want both http://somedomain.com/ and http://somedomain.com/index.html showing up). Wherever you find duplication, right-click the entry to find out where it’s being linked, then change the link to the standard version.
Status – I almost always tell Xenu to “skip external links” while running a crawl. That way, I can sort by status to see how many external links I have. As far as SEO goes, excessive external linking isn’t advisable. If you find that you have a lot of external links, it might be worthwhile to trim them down.
Size – Some pages are so large that search engine spiders will only index up to a certain point and stop. The common wisdom is to keep page sizes under 100 kilobytes. Whether you adhere to this guideline or not, limiting your pages to a reasonable file size is still a good practice.
Title – Just like the Address column, you’re checking the Title column for duplicate entries. Often, this is an effective way to weed out poor title schemes and duplicate URLs. If you find duplicate titles, go through and give each of the pages unique, descriptive titles to differentiate them from one another.
Level – Some websites are so vast and labyrinthine that search engine spiders just give up trying to index all of the information. One stopping point is click depth. By sorting the Level column, you can see how many links a normal person would have to click to reach the URL. If this number is greater than 10, you’ll want to reconsider your internal linking and menu structure.
Out Links – As with click depth, search engine spiders eventually give up when they see too many links. Sort this column and look for entries with 100 or more outbound links. For those pages, consider eliminating unnecessary links or splitting the links out into several pages.
Duration – For search engines as well as users, long page load times are not advisable. Check this column for unusually high durations. If you discover any, determine if there is a programmatic reason or contact your hosting provider for help.
In addition to the main program view, Xenu Link Sleuth outputs detailed reports to your browser. Here are the two main areas to pay attention to.
Broken Links – You can get lists of every broken link ordered by link or by page, whichever is more convenient for you. It’s worthwhile to go through and fix each and every broken link on the site. Doing so will offer a better experience for your users and keep the link juice flowing properly through the site.
Redirected URLs – Redirects aren’t quite as bad as broken links, since users still end up somewhere other than a 404 error page. Still, they’re worth fixing for the spiders, especially if they’re temporary (302) redirects. Go through the list and fix every link to point to the correct URL.