Simply Testable Blog

Figuring out how to automate away the pain of routine front-end web testing; the story behind SimplyTestable.com.

213 posts covering the initial idea, growth of the service, features, advances, failures and successes.

JavaScript Static Analysis Analysis

I’m in the process of removing the JavaScript static analysis feature.

I’m really quite sure that the results of such tests are of no value and that almost no-one uses this feature.

Does (relatively) anyone use this feature? Let’s flex our SQL fingers and see whether my assumptions match reality.

In Short

From looking at how the different test types (HTML validation, CSS validation, JavaScript static analysis and link integrity) are used, it looks like JavaScript static analysis is used only because it can be and not because it needs to be.

In all cases where a specific URL is tested more than once, just 2.3% of re-tests are for JavaScript static analysis. This says that people may try the feature once but, generally, never again.

Does anyone use this feature? No.

Tests, Tasks and the Difference Between Them

A full-site test covering X aspects for a website with Y URLs will result in X * Y tests being carried out. See how the word ‘test’ occurs twice in that last sentence?

In both cases, the word ‘test’ has a specific meaning. Time to get a little more specific.

A full-site test covering X aspects for a website with Y URLs will result in X * Y tasks being performed. That’s better.

A test covers the whole operation of examining some aspects of a website. A task is the actual examination of a specific aspect for a specific URL.

You need to be aware of this to correctly understand what comes next.

How Many Tests Have Been Run?

If I want to examine how many tests of a certain type have been run compared to either all tests or to tests of different types, I need to know how many tests have been run.

We’re looking at how many times someone chose to run a test of a website. Tests, not tasks, just to be clear.

There have been 72,751 tests run so far. That includes tests run under our free demo account in addition to tests run by all of you who have created accounts. The free demo account does not offer JavaScript static analysis so let’s exclude those.

There have been 9,051 non-demo tests run so far. Of those, 2,426 have been run by me under my own accounts. Best to exclude those since I’m far from a representative user.

There have been 6,625 non-demo tests run so far by everyone with an account who is not me. This feels about right as not everyone who finds us decides to create an account.

Good, what can we look at next?

How Many Tasks Have Been Performed?

The 6,625 non-demo tests run so far (by everyone with an account who is not me) cover 1,003,216 tasks.

What test types do these cover?

Test type Number of tasks Percentage of tasks of this type
HTML Validation 213,598 21%
CSS Validation 182,147 18%
JS Static Analysis 79,193 8%
Link Integrity 528,278 53%

Whilst JavaScript Static Analysis lags behind at only 8% of all tasks carried out, we’re still looking at nearly 80,000 of such tasks being performed. It’s not clear yet if this seemingly-low number is relevant.

Perhaps many people choose to run JavaScript static analysis on its own against very small sites and other people choose to run the other test types on their own against very large sites?

We need to dig a little deeper.

How Many Tasks Are Run Again?

We have so far seen the number of tasks performed for each test type. JavaScript static analysis looks to be underused, link integrity seems highly used and both HTML and CSS validation are somewhat equal somewhere in the middle.

What interests me is how often different types of test are run again.

You might run a specific type of test against a website once and then decide that that sort of testing is not for you. Or you might like what the test results tell you and, possibly after making changes to the website that was tested, you might run the same types of test again.

Repeat usage might suggest greater interest. Let’s see.

  • 1,003,216 tasks have been carried out
  • 373,284 unique URLs have been tested
  • 629,932 unique URLs have been tested more than once
Test type Number of tasks Unique URL count Re-test count Percentage of re-test tasks of this type
HTML Validation 213,598 121,050 92,548 15%
CSS Validation 182,147 110,304 71,843 11%
JS Static Analysis 79,193 64,869 14,324 2.3%
Link Integrity 528,278 77,061 451,217 72%

How Many Tests Are Run That Test X?

A test might cover multiple test types (and we’ll see a bit later that this is most often the case). For now, let’s see how many tests cover at least one specific test type.

Test type Number of tests Percentage of tests covering this type
HTML Validation 4,755 72%
CSS Validation 3,569 54%
JS Static Analysis 1,525 23%
Link Integrity 2,189 33%

We can already see that JavaScript static analysis is the least popular type of test, however nearly a quarter of all tests choose this feature.

We need to dig deeper and look at the situations in which JavaScript Static Analysis is chosen.

How Many Tests Are Run That Test Only X (And Y and Z)?

Most people carry out more than one type of test at a time. Why run a HTML validation test and then separately a CSS validation test when you can carry out both at once?

There are 14 unique ways to select between the four test types. Here’s how many times the different permutations have been chosen. Those with a percentage usage of less than one have been de-emphasised.

Test types Number of tests Percentage of tests covering this type
HTML Validation only 431 6.5%
CSS Validation only 51 0.77%
JS Static Analysis only 11 0.16%
Link Integrity only 198 3%
HTML Validation and CSS Validation only 1,880 28%
HTML Validation and JS Static Analysis only 4 0.06%
HTML Validation and Link Integrity only 794 12%
CSS Validation and JS Static Analysis only 3 0.05%
CSS Validation and Link Integrity only 4 0.06%
JS Static Analysis and Link Integrity only 0 0%
HTML Validation, CSS Validation and JS Static Analysis only 461 7%
HTML Validation, CSS Validation and Link Integrity only 147 2.2%
CSS Validation, JS Static Analysis and Link Integrity only 0 0%
HTML Validation, CSS Validation, JS Static Analysis and Link Integrity 1,023 15%

That’s a lot to look at. Let’s exclude the de-emphasised rows and order them by percentage usage.

Test types Number of tests Percentage of tests covering this type
HTML Validation and CSS Validation only 1,880 28%
HTML Validation, CSS Validation, JS Static Analysis and Link Integrity 1,023 15%
HTML Validation and Link Integrity only 794 12%
HTML Validation, CSS Validation and JS Static Analysis only 461 7%
HTML Validation only 431 6.5%
Link Integrity only 198 3%
HTML Validation, CSS Validation and Link Integrity only 147 2.2%

We can see that 15% of all tests are run with all test types (HTML validation, CSS validation, JavaScript static analysis and link integrity) and 7% of all tests are the same minus the link integrity checks.

That adds up to there being 22% of tests that feature JavaScript static analysis. Those with keen eyes will see that this doesn’t equal the 23% previously stated. I’m dealing with rounded numbers here, it’s close enough.

Analysis

I’m going to admit that I have a bias. I think that running JavaScript static analysis tests against production code is of no benefit.

But what I’m really interested in is where the usage of the JavaScript static analysis feature fits in. What does all of the above data say?

How Many Tasks Are Run Again?

We can learn quite a bit. More tasks have been run than URLs have been tested. Some URLs are tested more than once.

For all tasks that are a re-test, only 2.3% choose JavaScript static analysis. This suggests a usage pattern of trying the JavaScript static analysis feature once but not again.

How Many Tasks Have Been Performed?

There’s nothing useful to learn from this section.

We can see that 8% of tasks are for JavaScript static analysis. This seems like a low number, but without further context this is somewhat meaningless.

How Many Tests Are Run That Test Only X (And Y and Z)?

The table rows in How Many Tests Are Run That Test Only X (And Y and Z) that are de-emphasised can be excluded from consideration.

The now-excluded rows that feature JavaScript static analysis cover just 18 tests, 0.27% of the total. These rows are de-emphasised for a reason: relatively speaking, no-one runs those test configurations.

Side statistic: one of these now-excluded JavaScript static analysis test configurations is used roughly once every 3½ months.

Looking only at the remaining table data, the JavaScript static analysis option is present in only two cases:

  • at the same time as HTML validation, CSS validation and link integrity
  • at the same time as HTML validation and CSS validation only

What this looks like to me is that JavaScript static analysis is selected at the same time as everything else. This seems to suggest that it is selected because it can be and not because it needs to be.

Does this apply to anything else? Let’s see.

Selected ... with everything? on its own? with only one other test type?
HTML Validation yes yes yes
CSS Validation yes no yes
JS Static Analysis yes no no
Link Integrity yes yes yes

All other test types are either selected on their own or with only one other test type. It is only JavaScript static analysis that is only selected with everything.

Conclusion

To answer the question I posed at the very start: Does anyone use JavaScript static analysis?

The JavaScript static analysis feature is (relatively) never selected on its own and is only otherwise selected in addition to all other test types.

In cases where it is selected, it is very rarely selected again.

By “all other test types” I mean in conjunction with HTML validation and CSS validation or in conjunction with HTML validation, CSS validation and link integrity. I consider link integrity tests to be a slightly special case and I would class them as an addition to everything.

Is JavaScript static analysis as a feature used? It is a feature that is chosen when someone opts to choose everything. It is not an option that is specifically chosen to be used, which leads me to assume that it is selected because it can be and not because it needs to be.

So, no, no-one is using the JavaScript static analysis feature for anything useful.