‹‹ Back to SVS Home
17.7 Permutation Test Methodology (Optional Module)
17.7 Permutation Test Methodology (Optional Module)
In order to be more sure about whether any results took place by chance, it may be desirable to perform permutation testing. In its linear and logistic regression modules, Optimus RP implements a philosophy for permutation testing based on private communications from Dr. Peter Westfall and Dr. Stanley Young. This philosophy is that to make a true permutation test, the whole process should be repeated with permuted data. An analogy might be of first catching fish with a certain bait, and then seeing if fishing all over again with a different bait would catch more fish. This would be distinct from seeing if the same fish caught before somehow react differently to the different bait.
The permutation testing of Optimus RP’s linear and logistic regression modules permutes the dependent variable, then runs the regressions all over again, checking the p-values from these regressions. The original regression matrices are not used. (This is distinct from checking the “fit” of the permuted dependent to the original regression results from a given set of regressors.) The object is to see whether by chance, a different set of dependents could have had a better relationship or “fit” with the covariates. This is tested through performing a new regression for each permutation.
The permuted p-value is the fraction of permuted tests which get a better p-value than the original test did. The original test is counted as one of the “permuted tests”.