Simply Testable Blog

Figuring out how to automate away the pain of routine front-end web testing; the story behind SimplyTestable.com.

213 posts covering the initial idea, growth of the service, features, advances, failures and successes.

Multiple Sitemaps, Indexed Sitemaps Supported

The URLs to be tested when running a full-site test are gathered from a website’s sitemap.

A sitemap can list a collection of URLs for your site. A sitemap can also act as an index listing a collection of sitemaps. Your robots.txt file can also list many sitemaps (despite this not being part of the standard).

Up until just a few minute ago, we supported only the first type: a sitemap that listed a collection of URLs for your site.

I rewrote our sitemap finder, sitemap retriever and sitemap model libraries to support multiple sitemaps and sitemap indexes so that all sitemap uses are supported.

If your robots.txt lists a sitemap index URL, we’ll now read the index and read in all the referenced sitemaps and extract all URLs across all sitemaps for testing.

If your robots.txt lists multiple sitemaps and are not using a sitemap index, we’ll just grab all those sitemaps and extract all the URLs across all sitemaps for testing.