If you are using Postman for your API Testing, then you can also you the same to automatically crawl all the pages on our website and check every link for a healthy HTTP status code.
This can be achieved using 2 simple API requests in Postman.
First lets create a new Collection and an Environment in Postman, where you can specify
- root_url
- start_url
Specify the values for the root_url and start_url.
root_url as https://linkeshkannavelu.com/
start_url as https://linkeshkannavelu.com/category/software-testing/selenium/
Create a simple request with Get method and enter url as {{start_url}} and in the Tests tab enter the following code.
// set environment variables to default values
postman.setEnvironmentVariable('links', '[]');
postman.setEnvironmentVariable('url', postman.getEnvironmentVariable('start_url'));
postman.setEnvironmentVariable('index', -1);
Create a second request – Get method and enter URL as {{url}} and in the Tests tab enter the following code.
// Tests and custom scripts can be written in JavaScript.
// get environment variables
var start_url = postman.getEnvironmentVariable('start_url');
var root_url = postman.getEnvironmentVariable('root_url');
var links = JSON.parse(postman.getEnvironmentVariable('links'));
var url = postman.getEnvironmentVariable('url');
var index = parseInt(postman.getEnvironmentVariable('index'));
// increment index counter to access links in array to check
index = index + 1;
// test if link works
if (responseCode.code > 400) {
console.log("This link is broken: ", url);
tests["Link works"] = false;
} else {
tests["Link works"] = true;
}
// if the current url includes the start_url, then this is an internal link and we should crawl it for more links
if (url.includes(start_url)) {
// load the response body as HTML using cheerio, get the <a> tags
var $ = cheerio.load(responseBody);
$('a').each(function (index) {
var link = $(this).attr('href');
// add links to the links array if not already in there
// if you have additional links you would like to exclude, for example, ads, you can add this criteria as well
if (!links.includes(link)) {
links.push(link);
}
});
}
// if we've gone through all the links, return early
if (links.length - 1 === index) {
console.log('no more links to check');
return;
}
// if link is a relative one, prepend with root_url
url = links[index]
if (! /^https?:\/\//.test(url)) {
url = root_url + url;
}
// update environment variable values
postman.setEnvironmentVariable("links", JSON.stringify(links));
postman.setEnvironmentVariable("url", url);
postman.setEnvironmentVariable("index", index);
// continue calling the same request until all links are checked
postman.setNextRequest('Check URL');
Now Open “Runner” Select the Collection, Select the Environment and Click on Start Run Button.
You can see Postman in action crawling all the links until there are no more links to check.
You can also simply download the Postman Collection and import it into your Postman.
Keep Software Weird 