TL;DR
The best free broken link checkers tools are:
- Online: Broken Links at a Glance
- Windows: Xenu's Link Sleuth(TM)
- OSX: Integrity
- Linux: Linkcheck
Note that the adviced Windows / OSX / Linux broken link checkers checks an unlimited number of links, while "Broken Links at a Glance" checks up to 5000 links. Hence for larger websites your better use a Windows / OSX / Linux broken link checker.
The advice above is based on a collection of longlist criteria, shortlist criteria, selection criteria and weighing factors.
You should read this entire test report to find out if these criteria and weighing factors apply to your specific situation. Especially for small websites many broken link checkers are suitable.
Contents
- The selection process
- The longlist
- The shortlist
- Selection criteria
- Weighting factors
- The results
- Adviced broken link checkers
- Notable features
- Lessons learned
The selection process
To be able to conclude in a decent way what the best free broken link checkers tools are, the process of getting to the conclusion / advice is structured as follows.
- The criteria and weighting factors for the ideal broken link are listed
- An overview of all broken link checkers is made
- A longlist is made (based on the longlist criteria)
- The broken link checkers on the longlist are tested, to find out if the belong to the shortlist.
- Based on in depth testing of the shortlist broken link checker tools, it will be concluded what the best free broken link checkers tools are.
The longlist
For creating a longlist, an overview of all broken linkcheckers and some criteria to place a broken linkchecker at the longlist are required. Based on sites as Link Checkers, Comparison of Link Checkers, Curlie - Link Checking and a lot of searches this overview of broken linkchecker tools has been created.
The longlist of broken link checkers is based on the following criteria:
Applying these criteria on the available linkcheckers gives this longlist of broken link checkers.
The shortlist
Broken link checkers at the longlist will be placed at the shortlist when they meet the following shortlist criteria:
Testing the shortlist criteria
To test the 90% criterium in a controlled way the website A Different End Of The Internet (ADEOTI) has been created. This website contains 24 broken links. 12 of these links are visible. 2 of these 12 visible links points to websites which are blocked by robots.txt.
When a broken link checker doesn't respect robots.txt at the-linked-to-website 11 of the 12 visible broken links should be detected (90% of 12 is 10.8).
When a broken link checker respects robots.txt at the-linked-to-website 9 of the 10 visible broken links should be detected.
The first test resulted in the following (relative long) shortlist ...
Note that their might be broken link checker tools which are not at the shortlist but are still ideal for your situation. If that is the case, other assumptions (shortlist criteria) apply to your situation.
For example, Linktiger isn't at the shortlist because it hasn't really a free variant (you might try a free variant for a month).
Selection criteria
The following selection criteria are relevant in selecting a broken link checker. After each criterium you will find the MoSCoW weighing factor (Must, Should, Could, Won't). See the explanation below.
Weighting factors
The weighting factors are the following:
- Must - The broken link checker must met this criterium otherwise it will not be advised
- Should - The broken link checker should met this criterium
- Could - The broken link checker could met this criterium
- Won't - This criterium is not used to judge the broken link checker
Each link checker gets a score based on the above criteria and weighting factors.
- Should - 5 points
- Could - 1 point
- Won't - 0 point
There are 14 Shoulds and 17 Coulds, so the maximum score is 87 points. The final score will be shown as percentage of this maximum score.
The results
The answer to the question How to test a broken link checker is now relative straight forward. The results are ...
For more details see the detailed test results.
Adviced broken link checkers
All broken link checkers at the shortlist are fine broken link checkers.
However based on the test results the following broken link checkers are adviced.
- Online: Broken Links at a Glance read the testreport
- Windows: Xenu's Link Sleuth(TM) read the testreport
- OSX: Integrity read the testreport
- Linux: Linkcheck read the testreport
Notable features
Testing the shortlist of broken link checkers unraveled the following notable features.
Amount of links tested
Some link checkers have a maximum of links -for the free edition- that might be tested.
It is remarkable that of the adviced broken link checkers only the online broken link checker "Broken Links at a Glance" has a limit of 5000 links. The offline broken link checkers don't have such a limit.
The (free) limits are:
- unlimited - DeepTrawl
- unlimited - Integrity
- unlimited - Linkcheck
- unlimited - Linkchecker
- unlimited - Webcheck
- unlimited - Xenu Link Sleuth (TM)
- 5000 links - Broken Links at a Glance
- 2000 links - Dead Link Checker
- 1500 links - Dr. Link Check
- 1000 links - Atom Seo Error 404
- 500 links - A1 Website Analyzer
- 150 pages (1200 links?) - W3C Link Checker
- 500 links - Online Domain Tools Website Link Checker
- 50 pages (400 links?) - Insite
Speed
The speed in links per seconds (lps) varies per link checker. The speeds below are measured during testing Rusty Links. Testing another website will probably give other speeds, so consider these speeds as indications. The following speeds are measured:
- 48 lps - Insite (measurement based on less than 100 links)
- 30 lps - Integrity
- 29 lps - Xenu Link Sleuth (TM)
- 16 lps - A1 Website Analyzer
- 14 lps - Broken Links at a Glance
- 10 lps - Atom Seo Error 404
- 10 lps - Linkcheck
- 8 lps - DeepTrawl
- 8 lps - Dead Link Checker
- 7 lps - Dr. Link Check
- 5.5 lps - Webcheck
- 3 lps - Linkchecker
- 1 lps - Online Domain Tools Website Link Checker
- 0.1 lps - W3C Link Checker
Note that offline link checkers like A1 Website Analyzer and Xenu Link Sleuth (TM) gives you the option to vary this speed by modifying the number of threads.
Note also that Linkcheck is optimized for checking on localhost, before you deploy. The speed on localhost is about 5000 links per second for a large site. For external sites, Linkcheck throttles itself significantly. As it should, because otherwise it would be a good tool for DoS attacks.
Timeout
When a webpage didn't respond in time, it will get a 408 Request Timeout error. What "in time" is varies per link checker.
The maximum latency time tested is 20 seconds. These timeouts are measured:
- 20 seconds - Atom Seo Error 404
- 20 seconds - DeepTrawl
- 20 seconds - Dr. Link Check
- 20 seconds - Insite
- 20 seconds - Integrity
- 20 seconds - Linkchecker
- 20 seconds - Online Domain Tools Website Link Checker
- 20 seconds - W3C Link Checker
- 20 seconds - Xenu Link Sleuth (TM)
- 15 seconds - A1 Website Analyzer
- 15 seconds - Broken Links at a Glance
- 15 seconds - Dead Link Checker
- 10 seconds - Webcheck
- 5 seconds - Linkcheck
Robots.txt
The extent to which the instructions of robots.txt are followed varies per link checker.
Roughly half of the link checkers tested respects the robots.txt directives of the tested website. These link checkers are:
- A1 Website Analyzer
- Broken Links at a Glance
- Dead Link Checker
- Linkcheck
- Linkchecker
- Online Domain Tools Website Link Checker
- W3C Link Checker
- Webcheck
Roughly a third of the link checkers tested respects the robots.txt directives at the-linked-to-website. These link checkers are:
- DeepTrawl
- Linkcheck
- Linkchecker
- Online Domain Tools Website Link Checker
- W3C Link Checker
DeepTrawl and Linkchecker didn't report which links aren't tested due to the robots.txt directives at the-linked-to-website. This requirement was a must, hence these link checkers get the label Not all Musts met!
Other Missed Musts
Besides the missed Musts mentioned above. The following link checkers missed a must:
- Atom Seo Error 404 and Online Domain Tools Website Link Checker because it is not possible to retest a website unlimited with the free version.
- Insite because it was not possible to test websites with up to 500 links.
Exceptions
Linkcheck, W3C Link Checker and Xenu Link Sleuth (TM) are the only link checkers that examines inline (anchor) links.
Broken Links at a Glance is the only one that detects blacklisted links.
Integrity, W3C Link Checker and Xenu Link Sleuth (TM) are the only link checkers that presents also the 200 Ok links.
Lessons learned
During this process of testing free broken link checkers, I learned that ...
- there are different thoughts regarding following the robots.txt directives by broken link checkers
- there are no two broken link checkers which returns exact the same results
- there are more free broken link checkers than I initial thought
- the "Allows you to test websites with up to NNN links" requirements has a huge impact.
- A1 Website Analyzer allows you to check an unlimited number of links (the paid version does) it will score 83%.
- W3C Link Checker will be the adviced online broken link checker when it allows you to check an unlimited number of links.
- I may be proud on my own broken link checker "Broken Links at a Glance"
Thank you for reading this test report. I hope you have find the best broken link checker tool(s) for you.
Best regards, Hans.
BTW Don't forget to check (and repair) broken links at a regular base.