Broken Links Part 1: Checking Tools for Your Website

There are few things more annoying to someone browsing the web than to click a link that doesn't work. While building pages on your site, test each link to make sure it works each time you create a link.

Even after you're done, and especially if your site has links to other sites, check all the links on your whole website at least once a week to ensure that things haven't changed. The web is an ever changing place. Websites are always being created and taken down. Webmasters around the world are constantly re-arranging and optimizing their websites so a link that worked yesterday won't necessarily work today.Using WordPress? Check out Broken Link Checker to automates the process of checking your site for broken links. It can even notify you by email when it finds broken links.

While there is no substitute for doing it right in the first place, there are tools to help you automate the link testing process. Here are a few we suggest for small and medium size websites

• WordPress Broken Link Checker - RECOMMEND
• Drupal Link Checker - RECOMMENDED
• Xenu Link Sleuth for Windows - RECOMMENDED
• W3C Web Link Checker (Web application)

Once your website grows beyond a certain size, scanning your website regularly for broken links becomes a lot less feasible because of the time it takes. While you can still use link checkers link Xenu to scan the top levels of your site, you can make use of server logs to identify URL's requests that resulted in 404 page not found - at least for links to your website. The rational is that, if nobody is clicking on the dead link, there probably isn't a need to fix it. If you know of a more practical option, share it in the comments below.

In a follow to this article, we will tale a look at Tips and Techniques For Troubleshooting Broken Links on Your Website.

WordPress Broken Link Checker

If you are fortunate enough to have built your website on WordPress, Broken Link Checker is a wonderful gift. It's one of those install, configure and forget it plugins. Not only will it scan your whole website regularly for broken and redirected links, it will even notify you by email when it find some - if configured properly it.

That being said, with all of the email going through your inbox these days, it is sometimes easy to overlook a message coming from your website. I therefore recommend you check WordPress' Dashboard periodically to see if there are any broken links or to update redirected ones. I also recommend that you scan your website using Xenu at least once per month instead of once per week just to make sure the WordPress plugin didn't miss anything.

Drupal Link Checker

Although the Broken Link Checker not as feature rich as the similarly named plugin above, the Link Checker module for Drupal extracts links from your content when saved and periodically tries to detect broken hypertext links by checking the remote sites. If a link fails, it will appear in the reports/logs section and on the content edit page. An author specific broken links report is also available in "My Account".

What it won't do is send you email notifications so you will need to periodically log into Drupal in order to check the broken links report.

Xenu Link Sleuth for Windows

Xenu Link Sleuth is a wonderful free website link checker for Windows that is much faster that most tools for small and medium size websites.

Here are just some of the ways you can use Xenu Link Sleuth:

• Find Broken links
• Find Redirected links
• Identify your site internal duplicate content issues (sort by Title)
• Find images and pages that are way too big (example: photos that have not been resized for the web) - sort by Size.
• Find least linked-in pages (sort by In Links) - Tip: Right click and select "Properties" to learn exactly which pages link to the current page.
• Find pages with most outlinks (sort by Out Links)
• Find your deepest-buried pages and learn how easily they can be found (sort by Status and Level)
• Find images that lack alternative text (sort by Type and look for blank Titles)
• Generate a static Site Map in the HTML report page
• Get a page count, image count, etc. in the HTML report page.

Xenu Link Sleuth can check any kind of website including but not limited to WordPress, Drupal, Moodle, Joomla, WebsiteBaker, Git Pages, Google Sites, etc. and even custom hand made sites.

Configuring Xenu

To begin, launch Xenu and click the first icon to the left in Xenu's toolbar and configure the settings as follows:

What address do you want to check?

Enter the main address of your website. Example: http://www.yourdomain.com/

Check external links: Check this box.

Consider URLs beginning with this as internal:

If you have more than one domain that should be considered part of your website, click the Add button after entering each website address. Otherwise Xenu will treat these as a separate website and won't scan them too.

Here are some examples:

• https://www.yourdomain.com/
• http://m.yourdomain.com/
• http://blog.yourdomain.com/
• http://ftp.yourdomain.com/
• http://www.yourdomain.ca/
• http://yourdomain.com/
• http://AlternateLanguageDomain.com/

Do not check any URLs beginning with this:

mailto:

Tips For Using Xenu

• Always Empty Your Website Cache - If you are using a web cache of any kind, be sure to clear it before you check your website for broken links. This may even involve restarting your web server software. I've seen more than one webmaster try to troubleshoot a broken link issue only to discover that he/she had resolved the problem an hour earlier but the web server was still serving the old version of the page.
• Too Many Broken Lins - Sometimes Xenu is so fast that it times out on many valid links, so it has to be adjusted to work properly. Set less parallel threads in "Options"=>"Properties" to have fewer failed connections. Many web servers limit you to a maximum of 4 concurrent connections. Setting this to 3 connections seems to be optimal, providing you with the most performance with the least number of timeouts.
• Retrying Instead of Rescanning - Press CTRL-R to retry all failed URLs. It's faster than scanning your site all over again. Note that this will not help in cases where you fixed the link in a page.
• Multiple Domains - Don't forget to include all valid versions of your sites domain name or it may not scan the mobile version of your site for example.
• Large Files Reported as Broken Links - Download files larger than 2 GB files may appear as an error in the result if you have anything less than IE8 or IE9 installed on your computer. Xenu uses the Internet Explorer engine and this was a limitation of older versions of IE.
• Broken Links That Work - Always manually test a link before deciding to remove it. Occasionally Xenu will report a broken link when it's not actually broken. For example, some websites use various techniques to prevent non-humans from visiting their website.

The (W3C) World Wide Web Consortium’s Web Link Checker

This is a convinient tool when you need something and are away from your computer. It is a bit slow so you may want to let it run why doing something else. It also respects robot.txt rules on your website and as such, parts of your website may not be included in the validation.

To use the W3C web link checker, fill in the fields as follows:

• Enter the address (URL) of a document that you would like to check: Provide the complete URL of the page you want to check.
• Check linked documents recursively: Check this if you want to also check pages that are linked to the above page.
• Recursion depth: This is the number of levels of links to follow from every page. Make sure the value is set high enough to cover the number of levels deep for your site and also to check links going to other sites. Do not enter an excessively high number, or you will also be checking a lot of links on other websites.

Scroll through the results, and note any broken links (they appear in red).