The best free broken link checker tools

Rusty Results Research

TL;DR

The best free broken link checkers tools are:

  1. Online: Broken Links at a Glance
  2. Windows: Xenu's Link Sleuth(TM)
  3. OSX: Integrity
  4. Linux: Linkcheck

Note that the adviced Windows / OSX / Linux broken link checkers checks an unlimited number of links, while "Broken Links at a Glance" checks up to 5000 links. Hence for larger websites your better use a Windows / OSX / Linux broken link checker.

The advice above is based on a collection of longlist criteria, shortlist criteria, selection criteria and weighing factors.

You should read this entire test report to find out if these criteria and weighing factors apply to your specific situation. Especially for small websites many broken link checkers are suitable.

 

Contents

  1. The selection process
  2. The longlist
  3. The shortlist
  4. Selection criteria
  5. Weighting factors
  6. The results
  7. Adviced broken link checkers
  8. Notable features
  9. Lessons learned
 

The selection process

Venn diagram where adviced is part of shortlist, shortlist is part of longlist and longlist is part of all broken link checkers To be able to conclude in a decent way what the best free broken link checkers tools are, the process of getting to the conclusion / advice is structured as follows.

  1. The criteria and weighting factors for the ideal broken link are listed
  2. An overview of all broken link checkers is made
  3. A longlist is made (based on the longlist criteria)
  4. The broken link checkers on the longlist are tested, to find out if the belong to the shortlist.
  5. Based on in depth testing of the shortlist broken link checker tools, it will be concluded what the best free broken link checkers tools are.
 

The longlist

For creating a longlist, an overview of all broken linkcheckers and some criteria to place a broken linkchecker at the longlist are required. Based on sites as Link Checkers, Comparison of Link Checkers, Curlie - Link Checking and a lot of searches this overview of broken linkchecker tools has been created.

The longlist of broken link checkers is based on the following criteria:

Longlist criteria
  1. The broken link checker is free to use (or there is a free light version)
  2. The broken link checker checks multiple pages, only checking the current page is not enough
  3. The broken link checker allows you to check every website (not only your website)

Applying these criteria on the available linkcheckers gives this longlist of broken link checkers.

 

The shortlist

Broken link checkers at the longlist will be placed at the shortlist when they meet the following shortlist criteria:

Shortlist criteria
  1. The broken link checker is at the longlist
  2. The broken link checker is able to find at least 90% of the clickable (visible) broken links

Testing the shortlist criteria

To test the 90% criterium in a controlled way the website A Different End Of The Internet (ADEOTI) has been created. This website contains 24 broken links. 12 of these links are visible. 2 of these 12 visible links points to websites which are blocked by robots.txt.

When a broken link checker doesn't respect robots.txt at the-linked-to-website 11 of the 12 visible broken links should be detected (90% of 12 is 10.8).

When a broken link checker respects robots.txt at the-linked-to-website 9 of the 10 visible broken links should be detected.

The first test resulted in the following (relative long) shortlist ...

Venn diagram of the shortlist of broken link checkers

 
The Shortlist
  1. Online
    1. Atom Seo Error 404 - 11/12
    2. Broken Links at a Glance - 12/12
    3. Dead Link Checker - 11/12
    4. Dr. Link Check - 11/12
    5. Online Domain Tools Website Link Checker - 9/10
    6. W3C LinkChecker - 12/12
  2. Cross platform
    1. A1 Website Analyzer - 12/12
    2. DeepTrawl - 10/10
    3. Linkcheck - 9/10
    4. Linkchecker - 10/10
  3. Windows
    1. Insite - 12/12
    2. Xenu's Link Sleuth(TM) - 12/12
  4. OSX
    1. Integrity - 12/12
  5. Linux
    1. Webcheck - 11/12

Note that their might be broken link checker tools which are not at the shortlist but are still ideal for your situation. If that is the case, other assumptions (shortlist criteria) apply to your situation.

For example, Linktiger isn't at the shortlist because it hasn't really a free variant (you might try a free variant for a month).

 

Selection criteria

The following selection criteria are relevant in selecting a broken link checker. After each criterium you will find the MoSCoW weighing factor (Must, Should, Could, Won't). See the explanation below.

Selection criteria
  1. Examines internal links - Must
  2. Examines relative links in combination with the base tag - Should
  3. Examines inline (anchor) links - Could
  4. Examines external links - Must
  5. Examines standard hyperlinks (the a tag) - Must
  6. Examines links to other resources (the src attribute) - Should
  7. Examines structural links (the link tag) - Should
  8. Ignores structural links with the dns-prefetch attribute - Could
  9. Examines links in stylesheets - Could
  10. Examines links with other schemes (at least FTP://) - Could
  11. Examines Unicode links - Should
  12. Detects link error 404 - Must
  13. Detects all link errors in the 400 and 500 range - Should
  14. Detects redirected links - Should
  15. Detects blacklisted links (at least blacklisted by Google Safe Browsing) - Could
  16. Respects robots.txt - Should
  17. Respects robots.txt of the-linked-to-website - Could
  18. Presents unchecked links due to the robots.txt directive at the-linked-to-website - Must
  19. Allows you to take a subdirectory as base for the test - Should
  20. Allows you to start the test with a single click (after entering the to be tested URL) - Could
  21. Allows you to use your mobile device - Could (only for online link checkers)
  22. Allows you to use a graphical user interface - Could (only for offline link checkers)
  23. Allows you to test an unlimited number of websites - Should
  24. Allows you to retest a website unlimited - Must
  25. Allows you to save the results (internal or by an CSV export) - Could
  26. Presents the results in one overview - Should
  27. Presents 200 Ok and broken links - Could
  28. Presents the webpage where an issue is found - Should
  29. Presents in detail (e.g. linenumber, source viewer) where an issue is found - Could
  30. Presents in a clear way the results when there are no issues found - Could
  31. Test at least 5 links per second - Could
  32. Test at least 10 links per second - Could
  33. Test at least 15 links per second - Could
  34. Test at least 20 links per second - Could
  35. Test at least 25 links per second - Could
  36. Allows you to test websites with up to 500 links - Must
  37. Allows you to test websites with up to 2000 links - Should
  38. Allows you to test websites with up to 5000 links - Should
  39. Allows you to test websites without a restriction on the number of links - Should
 

Weighting factors

The weighting factors are the following:

  1. Must - The broken link checker must met this criterium otherwise it will not be advised
  2. Should - The broken link checker should met this criterium
  3. Could - The broken link checker could met this criterium
  4. Won't - This criterium is not used to judge the broken link checker

Each link checker gets a score based on the above criteria and weighting factors.

  1. Should - 5 points
  2. Could - 1 point
  3. Won't - 0 point

There are 14 Shoulds and 17 Coulds, so the maximum score is 87 points. The final score will be shown as percentage of this maximum score.

 

The results

The answer to the question How to test a broken link checker is now relative straight forward. The results are ...

The test results
  1. Xenu's Link Sleuth(TM) Windows - 90% (61/75)
  2. Integrity OSX - 89% (51/75)
  3. Broken Links at a Glance Online - 86% (74/74)
  4. Linkcheck Windows, OSX, Linux - 86% (55/75)
  5. Webcheck Linux - 76% (57/75)
  6. W3C LinkChecker Online - 70% (72/75)
  7. Dead Link Checker Online - 69% (74/74)
  8. A1 Website Analyzer Windows, OSX - 68% (66/74)
  9. Dr. Link Check Online - 55% (69/75)

The percentage is the relative score, the score between brackets "(nn/75)" or "(nn/74)" is the Dead Link City score (which score is relevant depends on the last time tested, due to an update of Dead Link City's robots.txt file).

For more details see the detailed test results.

 

Adviced broken link checkers

All broken link checkers at the shortlist are fine broken link checkers.

However based on the test results the following broken link checkers are adviced.

Venn diagram for adviced broken link checkers

  1. Online: Broken Links at a Glance read the testreport
  2. Windows: Xenu's Link Sleuth(TM) read the testreport
  3. OSX: Integrity read the testreport
  4. Linux: Linkcheck read the testreport
 

Notable features

Testing the shortlist of broken link checkers unraveled the following notable features.

Amount of links tested

Some link checkers have a maximum of links -for the free edition- that might be tested.

It is remarkable that of the adviced broken link checkers only the online broken link checker "Broken Links at a Glance" has a limit of 5000 links. The offline broken link checkers don't have such a limit.

The (free) limits are:

  1. unlimited - DeepTrawl
  2. unlimited - Integrity
  3. unlimited - Linkcheck
  4. unlimited - Linkchecker
  5. unlimited - Webcheck
  6. unlimited - Xenu Link Sleuth (TM)
  7. 5000 links - Broken Links at a Glance
  8. 2000 links - Dead Link Checker
  9. 1500 links - Dr. Link Check
  10. 1000 links - Atom Seo Error 404
  11. 500 links - A1 Website Analyzer
  12. 150 pages (1200 links?) - W3C Link Checker
  13. 500 links - Online Domain Tools Website Link Checker
  14. 50 pages (400 links?) - Insite

Speed

The speed in links per seconds (lps) varies per link checker. The speeds below are measured during testing Rusty Links. Testing another website will probably give other speeds, so consider these speeds as indications. The following speeds are measured:

  1. 48 lps - Insite (measurement based on less than 100 links)
  2. 30 lps - Integrity
  3. 29 lps - Xenu Link Sleuth (TM)
  4. 16 lps - A1 Website Analyzer
  5. 14 lps - Broken Links at a Glance
  6. 10 lps - Atom Seo Error 404
  7. 10 lps - Linkcheck
  8. 8 lps - DeepTrawl
  9. 8 lps - Dead Link Checker
  10. 7 lps - Dr. Link Check
  11. 5.5 lps - Webcheck
  12. 3 lps - Linkchecker
  13. 1 lps - Online Domain Tools Website Link Checker
  14. 0.1 lps - W3C Link Checker

Note that offline link checkers like A1 Website Analyzer and Xenu Link Sleuth (TM) gives you the option to vary this speed by modifying the number of threads.

Note also that Linkcheck is optimized for checking on localhost, before you deploy. The speed on localhost is about 5000 links per second for a large site. For external sites, Linkcheck throttles itself significantly. As it should, because otherwise it would be a good tool for DoS attacks.

Timeout

When a webpage didn't respond in time, it will get a 408 Request Timeout error. What "in time" is varies per link checker.

The maximum latency time tested is 20 seconds. These timeouts are measured:

  1. 20 seconds - Atom Seo Error 404
  2. 20 seconds - DeepTrawl
  3. 20 seconds - Dr. Link Check
  4. 20 seconds - Insite
  5. 20 seconds - Integrity
  6. 20 seconds - Linkchecker
  7. 20 seconds - Online Domain Tools Website Link Checker
  8. 20 seconds - W3C Link Checker
  9. 20 seconds - Xenu Link Sleuth (TM)
  10. 15 seconds - A1 Website Analyzer
  11. 15 seconds - Broken Links at a Glance
  12. 15 seconds - Dead Link Checker
  13. 10 seconds - Webcheck
  14. 5 seconds - Linkcheck

Robots.txt

The extent to which the instructions of robots.txt are followed varies per link checker.

Roughly half of the link checkers tested respects the robots.txt directives of the tested website. These link checkers are:

  1. A1 Website Analyzer
  2. Broken Links at a Glance
  3. Dead Link Checker
  4. Linkcheck
  5. Linkchecker
  6. Online Domain Tools Website Link Checker
  7. W3C Link Checker
  8. Webcheck

Roughly a third of the link checkers tested respects the robots.txt directives at the-linked-to-website. These link checkers are:

  1. DeepTrawl
  2. Linkcheck
  3. Linkchecker
  4. Online Domain Tools Website Link Checker
  5. W3C Link Checker

DeepTrawl and Linkchecker didn't report which links aren't tested due to the robots.txt directives at the-linked-to-website. This requirement was a must, hence these link checkers get the label Not all Musts met!

Other Missed Musts

Besides the missed Musts mentioned above. The following link checkers missed a must:

  1. Atom Seo Error 404 and Online Domain Tools Website Link Checker because it is not possible to retest a website unlimited with the free version.
  2. Insite because it was not possible to test websites with up to 500 links.

Exceptions

Linkcheck, W3C Link Checker and Xenu Link Sleuth (TM) are the only link checkers that examines inline (anchor) links.

Broken Links at a Glance is the only one that detects blacklisted links.

Integrity, W3C Link Checker and Xenu Link Sleuth (TM) are the only link checkers that presents also the 200 Ok links.

 

Lessons learned

During this process of testing free broken link checkers, I learned that ...

  1. there are different thoughts regarding following the robots.txt directives by broken link checkers
  2. there are no two broken link checkers which returns exact the same results
  3. there are more free broken link checkers than I initial thought
  4. the "Allows you to test websites with up to NNN links" requirements has a huge impact.
    1. A1 Website Analyzer allows you to check an unlimited number of links (the paid version does) it will score 83%.
    2. W3C Link Checker will be the adviced online broken link checker when it allows you to check an unlimited number of links.
  5. I may be proud on my own broken link checker "Broken Links at a Glance"

Thank you for reading this test report. I hope you have find the best broken link checker tool(s) for you.

Best regards, Hans.

BTW Don't forget to check (and repair) broken links at a regular base.