Permutation testing consists of running many tests which are just like the original test except that the dependent variable is permuted differently for each one, and counting the number of times the test statistic resulting from a permuted dependent variable is more significant than the statistic from the original test.

The permuted p-value is the fraction of permuted tests which are more significant than or as significant as the original test. The original test is counted as one of the “permuted tests” in this calculation.

This is meant to approximate the probability that your test could come out as a “positive” by chance alone, which is the definition of the p-value.

NOTE: The permutations that HelixTree uses are not based on a time of day seed but on a constant seed.

NOTE: The comparisons of significance can be performed without converting the test statistics into approximations of p-values based on the test statistic, because all the tests use the same statistic with the same numbers of degrees of freedom.

26.21.1 Single Value Permutations

With single value permutations, the dependent variable is permuted and the given statistical test is performed using the given model on the given marker. This process is repeated the number of times that you select. The permuted p-value is defined as the fraction of times that a test with a permuted dependent variable on the given marker came out as significant as or more significant than the same test on the same marker did with the non-permuted dependent variable.

NOTE: This single-value permutation technique focuses exclusively on the marker in question, and by itself does not offer any type of correction for multiple testing.

26.21.1.1 Single Value Permutation Example

Suppose your approximated p-value (based on a chi-square distribution) is 0.2, you are doing 10 permutations, and the significances of those permutations (based on the same chi-square distribution) may be expressed as the following approximated p-values (counting the original test as the first “permutation”):

In the above column, we observe 7 outcomes where the approximated p-value is equal to or less then 0.2. The permuted p-value is therefore 7/10 = 0.7.

26.21.2 Full Scan Permutations

NOTE: In the (optional) regression module, this technique is selected as Perform all-marker permutation tests with __ permutations if you have also selected Moving window with fixed size or Moving window with genetic distance of ....

The full-scan permutation technique differs from the single-value technique in that it addresses the multiple-testing problem. It does this by comparing the original test result from an individual marker not merely with the permuted result from that marker, but also with the permuted results from all the rest of the markers. If any of these comparisons comes out to be more significant than the unpermuted result of the marker in question, the permutation is considered to have resulted in “the permuted test being more significant than the unpermuted test”.

To be specific, the following procedure is used for full-scan permutation testing:

1. For the number of times you have specified, the dependent variable is permuted and the test is performed over all markers. From each permutation, HelixTree keeps track of the most significant result from any of the markers.
2. The full-scan permuted p-value of each marker may now be found. It is the fraction of permutations for which the most significant result over all markers for that permutation was as significant as or more significant than the test on the marker in question using the non-permuted dependent variable.

NOTE: This procedure will tend to multiply the result by a value somewhat equal to the number of markers being tested, and thus gives answers comparable to those from Bonferroni-corrected p-values.

26.21.2.1 Full Scan Example One

You run the same association test on four spreadsheets. These spreadsheets are all alike, except that one has the real data you are interested in, and the other three have the same data except that the dependent variable has been permuted randomly in those spreadsheets.

You look at the (approximated) p-values from all of these tests, and note that the best approximated p-value from the first spreadsheet on any marker (which came from marker 4) is .0003. You also note that the best approximated p-values (which resulted from various markers) from the second, third, and fourth spreadsheets are .005, .02, and .013.

Using this technique, you could say that the full-scan permuted p-value of marker 4 would be one (the number of spreadsheets whose significance measured as a p-value was .0003 or better) divided by four (the total number of spreadsheets), or .25.

Normally, to get a better idea of the real p-value, you would perform many more permutations than this! For instance, one thousand permutations or more would be necessary to more accurately gauge a p-value near .001.

26.21.2.2 Full Scan Example Two

Suppose you have 5 markers, and their approximated test p-values (based on a chi-square distribution) are:

Their Bonferroni-corrected p-values would be:

Also suppose we do full-scan permutation testing using 10 permutations on these markers, and the approximated p-value results of those permutations on those markers (based on the same chi-square distribution and counting the original test as a “permutation”) are:

In the above data, we find the following best permuted-test outcomes over all markers for each of the permutations are:

For the first marker, whose approximated p-value from actual data was 0.20, we find that the best approximated p-values from the permutations other than permutation 7 (0.26) were all as good as or better than 0.20. Thus, the full-scan permuted p-value for the first marker is 9/10 or 0.90.

For the second marker, at 0.10, all permutations except permutation 5 (0.20), permutation 7 (0.26), and permutation 10 (0.14) were better than its actual-data value. Thus, its full-scan permuted p-value is 7/10 or 0.70.

The third marker has an approximated p-value of 0.02 on actual data, which is much better than that for the other markers, and is only equalled or bettered by itself (“permutation” 1 at 0.02), permutation 4 (0.01) and permutation 6 (0.01). Thus, its full-scan permuted p-value is 3/10 or 0.30.

Meanwhile, every permutation had a better approximated p-value than either the 0.34 or 0.52 values from the last two markers. Thus, these markers both have a full-scan permuted p-value of 10/10 or 1.00.

Comparing the full-scan permuted p-values for every marker with its Bonferroni-corrected p-value for real data, we get these similar values:

26.21.2.3 Full Scan Example Three

Let’s say your full-scan permuted p-value was .463 from when you did 1000 permutations. This means that of the 1,000 permutations performed, the p-value from the internally saved list of the best statistical results from the permutations was better 463 times out of 1000. This would be a poor showing. However, you must take into consideration that each time the permutation was performed, it did not do anything like averaging the approximated p-values for all the markers, but it took the one marker with the best showing (whose approximated p-value would be the lowest).