Tools

function assignment(stat::Union{BivStatistic, OneSampStatistic}, ns::Int) 

function assignment(stat::IndSampStatistic, ns::Vector{Int}) 

function assignment(stat::RepMeasStatistic, ns::@NamedTuple{n::Int, k::Int})

Analyse the ns argument for test statistic given as singleton stat and return a singleton of type Assignment, Balanced() if the design is balanced (equal subjects in all groups/measurements), Unbalanced() otherwise.

Only for test-statistics of the IndSampStatistic group the result may be Unbalanced(); for all the others the result is always Balanced().

The complete list of test statistics is here.

Examples

using PermutationTests
assignment(PearsonR(), 12) # -> Balanced()
assignment(AnovaF_IS(), [5, 5, 6]) # -> Unbalanced()
assignment(AnovaF_IS(), [6, 6, 6]) # -> Balanced()
assignment(StudentT_IS(), [5, 6]) # -> Unbalanced()
assignment(StudentT_IS(), [5, 5]) # -> Balanced()
assignment(AnovaF_RM(), (n=10, k=3)) # -> Balanced()
assignment(StudentT_1S(), 8) # -> Balanced()

function eqStat(stat, direction::TestDir, design::Assign) 
    where {TestDir<: TestDirection, Assign <: Assignment}

Return the most efficient statistic equivalent to stat, for the given singleton direction of type TestDirection and singleton design of type Assignment.

stat is a singleton of one of the five typical test statistics, that is, PearsonR(), AnovaF_IS(), StudentT_IS, AnovaF_RM() or StudentT_1S(), see Statistic.

To avoid errors, use the result of assignment as argument design.

Do not use StudentT_RM() as stat; PermutationTests.jl never use this test statistic. instead, t-tests for repeated measures are carried out as one-sample t-tests on the difference of the two measurements.

Examples

using PermutationTests
a=assignment
ns=12
eqStat(PearsonR(), Both(), a(PearsonR(), ns))       # -> Covariance()
eqStat(PearsonR(), Right(), a(PearsonR(), ns))      # -> CrossProd()
ns=[6, 6, 6]
eqStat(AnovaF_IS(), Both(), a(AnovaF_IS(), ns))     # -> SumGroupTotalsSq_IS()
ns=[6, 7, 6]
eqStat(AnovaF_IS(), Both(), a(AnovaF_IS(), ns))     # -> SumGroupTotalsSqN_IS()
ns=[6, 7]
eqStat(StudentT_IS(), Both(), a(StudentT_IS(), ns)) # -> SumGroupTotalsSqN_IS()
eqStat(StudentT_IS(), Left(), a(StudentT_IS(), ns)) # -> Group1Total_IS()
ns=(n=10, k=3)
eqStat(AnovaF_RM(), Both(), a(AnovaF_RM(), ns))     # -> SumTreatTotalsSq_RM()
eqStat(AnovaF_RM(), Right(), a(AnovaF_RM(), ns))    # -> SumTreatTotalsSq_RM()
ns=7
eqStat(StudentT_1S(), Both(), a(StudentT_1S(), ns)) # -> Sum()
eqStat(StudentT_1S(), Left(), a(StudentT_1S(), ns)) # -> Sum()

function allPerms(stat::BivStatistic, ns::Int) 

function allPerms(stat::IndSampStatistic, ns::Vector{Int})

function allPerms(stat::RepMeasStatistic, ns::@NamedTuple{n::Int, k::Int})

function allPerms(stat::OneSampStatistic, ns::Int)

Total number of systemetic permutations (exact test) for a test statistic given as singleton stat of a given group.

For the ns argument see ns.

For an exact tests with $N$ observations and $K$ groups/measuremets, the total number of systematic permutations is

• for stat a BivStatistic :$\quad N!$
• for stat a IndSampStatistic : $\quad \frac{N!}{N_1! \cdot \ldots \cdot N_K!}$
• for stat a RepMeasStatistic : $\quad K!^N$
• for stat a OneSampStatistic : $\quad 2^N$

For the number of non-redundant permutations, which is the actual number of permutations that are listed for exact tests, see nrPerms.

Examples

# Total number of possible permutations for a 1-way ANOVA for indepedent samples 
# test with a total of 18 observations in three balanced groups.
allPerms(AnovaF_IS(), [6, 6, 6]) # -> 17153136

# Total number of possible permutations for a correlation test with 12 observations.
allPerms(PearsonR(), 12) # -> 479001600

function nrPerms(stat::Union{BivStatistic, OneSampStatistic}, ns::Int, total, 
                direction::TestDir, design::Assign=Balanced()) 

function nrPerms(stat::IndSampStatistic, ns::Vector{Int}, total, 
                direction::TestDir, design::Assign)

function nrPerms(stat::RepMeasStatistic, ns::@NamedTuple{n::Int, k::Int}, total, 
                direction::TestDir, design::Assign=Balanced())
                
    where {TestDir<: TestDirection, Assign <: Assignment}

Number of non-redundant systemetic permutations. This is the actual number of permutations that are listed for an exact test.

Depending on the test direction and design, there may exists redundant permutations, i.e., permutations that yields the same test statistic. These permutations are therefore eliminated, yielding a faster test.

stat is the test statistic used by the test given as a singleton. It belongs to one group of test statistics.

For the ns argument see ns.

total is the total number of systematic permutations. In order to avoid errors, it should be given as the resut of the allPerms function.

direction is a singleton of type TestDirection.

design is a singleton of type Assignment. In order to avoid errors, use the result of assignment as argument design.

With stat a IndSampStatistic and a balanced design, the non-redundant permutations are $\frac{total}{K!}$, with $K$ the number of groups.

With stat a RepMeasStatistic and a bi-directional tests, the non-redundant permutations are $\frac{total}{K!}$, with $K$ the number of measures.

For stat a BivStatistic or a OneSampStatistic return total, that is, there is no possible redundancy.

Note that the elimination of redundant permutations is rarely implemented in software packages for permutation tests.

Examples

using PermutationTests
# Number of possible permutations for a 1-way ANOVA for indepedent samples 
# test with a total of 18 observations in three balanced groups.
ns=[6, 6, 6]
total=allPerms(AnovaF_IS(), ns) # -> 17153136
# Number of non-redundant permutations that will be actually listed
# to perform the test.
design=assignment(AnovaF_IS(), ns)
nrPerms(AnovaF_IS(), ns, total, Both(), design) # -> 2858856

function genPerms(stat::BivStatistic, x::UniData, 
            ns::Int, direction::TestDir, design::Assign) 
    
function genPerms(stat::IndSampStatistic, x::UniData, 
            ns::Vector{Int}, direction::TestDir, design::Assign)
                
function genPerms(stat::RepMeasStatistic, x::UniData, 
            ns::@NamedTuple{n::Int, k::Int}, direction::TestDir, design::Assign)
            
function genPerms(stat::OneSampStatistic, x::UniData, 
            ns::Int, direction::TestDir, design::Assign)

    where {TestDir<: TestDirection, Assign <: Assignment}

Generate a lazy iterator over all possible (systematic) permutations according to the permutation scheme to be used for test statistic stat, which is given as a singleton of type Statistic.

Note that data permutation in PermutationsTests.jl are always obtained by lazy iterators, that is, permutations are never listed physically. This is one of the main reasons why the package is fast and allocates little memory.

You do not need these functions for general usage of the package, however you need to know them if you wish to create your own test.

There exists a permutation scheme for each group the test statistics stat belong to.

x is the membership vector.

For the ns argument see ns.

direction is a singleton of type TestDirection.

design is a singleton of type Assignment. In order to avoid errors, use the result of assignment as argument design.

With stat a BivStatistic (e.g., PearsonR()), the iterations unfold the $N!$ possible reorderings of the $N$ elements in vector x and argument ns is ignored. In this case x is either a trend (e.g., [1,...,N] for a linear trend) or a variable to be permuted for correlation/trend tests.

With stat a IndSampStatistic (e.g., AnovaF_IS()), the iterations unfold the $\frac{N!}{N_1 \cdot \ldots \cdot N_K}$ possible arrangements of $N$ elements in $K$ groups. In this case x is ignored and ns is a vector holding the K group numerosities (i.e., [N1,...,Nk]). For this scheme, if the design is balanced, i.e., all elements of ns are equal, some permutations are redundant, see nrPerms.

With stat a RepMeasStatistic (e.g., AnovaF_RM()), the iterations unfold the $(K!)^N$ reordering of $K$ measures (e.g., treatments) in all $N$ observation (e.g., subjects). In this case x is ignored and ns is a named tuple with form (n=N, k=K). For this scheme, if the test is bi-directional some permutations are redundant, see nrPerms.

With stat a OneSampStatistic (e.g., StudentT_1S()), the iteartions unfold the $2^N$ flip-sign patterns. In this case x is ignored and ns=N, where $N$ is the number of observations.

Examples

using PermutationsTests

x=membership(PearsonR(), 3) # -> [1, 2, 3]
# The observations are always listed in the natural order 1, 2,...
Pr = genPerms(PearsonR(), x, 0, Both(), Balanced()) 
collect(Pr) # physically list the iterations
# yields the 6 = 3! elements [1, 2, 3]...[3, 2, 1]
# Here the integers represent the permuted position of the obervations in x

PfIS = genPerms(AnovaF_IS(), Int[], [2, 3], Both(), Unbalanced()) 
collect(PfIS)
# yields the 10 = 5!/2!3! elements [1, 1, 2, 2, 2]...[2, 2, 2, 1, 1]
# Here the integers represent the permuted group to which the corresponding 
# obervations in the data vector `y` belongs. see `_permTest!`.
# If the design is balanced and the test is bi-directional, 
# only 1/k! of the permutations are retained, thus 
PfIS_ = genPerms(AnovaF_IS(), Int[], [2, 2], Both(), Balanced()) 
collect(PfIS_) 
# yields the 3 = (4!/2!2!)/2! elements [1, 1, 2, 2], [1, 2, 2, 1] and [2, 1, 2, 1]

PfRM = genPerms(AnovaF_RM(), Int[], (n=2, k=3), Right(), Balanced()) 
collect(PfRM)
# yields the 36 = (k!)^n elements [1, 2, 3, 4, 5, 6], [1, 3, 2, 4, 5, 6], 
# ..., [3, 2, 1, 6, 5, 4]
# Here the integers represent the treatement for each subject to which the 
# corresponding obervation in the data vector `y` belongs. see `_permTest!`.
# If the test is bi-directional, 
# only 1/k! of the permutations are retained, thus 
PfRM_ = genPerms(AnovaF_RM(), Int[], (n=2, k=3), Both(), Balanced()) 
collect(PfRM_)
# yields instead the 6 = (k!)^n / elements [1, 2, 3, 4, 5, 6], [1, 3, 2, 4, 5, 6], 
# ..., [3, 2, 1, 4, 5, 6]

Pt1S = genPerms(StudentT_1S(), Int[], 3, Both(), Balanced())
collect(Pt1S)
# yields the 8=2^3 elements (1, 1, 1), (-1, 1, 1), (1, -1, 1), (-1, -1, 1), 
#..., (-1, -1, -1) 
# Here the integers represent the sign to be applied to each observation 
# at each data permutation. 
# Note that only for OneSampStatistic, the iterator generates tuples and not arrays.

function table2vec(table::Matrix{I}, stat::IndSampStatistic) 

function table2vec(tables::AbstractVector{Matrix{I}}, stat::IndSampStatistic) 

function table2vec(table::Matrix{I}, stat::RepMeasStatistic) 

function table2vec(tables::AbstractVector{Matrix{I}}, stat::RepMeasStatistic) 

    where I<:Int

Format tables of dicothomous data so as to create data input for test functions.

Return the 2-tuple holding the formatted table as a vector (for univariate test) or a vector of vectors (for multiple comparisons tests) and the appropriate argument ns.

You do not need to use these functions in general, as they are called internally by the test functions. However they may be useful if you create your own test for dicothomous data.

stat is a singleton of the Statistic type, that is, a test-statistic belonging to one of the group of test statistics.

For the usage of these functions, see the documentation of the test functions using them:

Univariate test functions

• chiSquaredTest
• fisherExactTest
• cochranqTest
• mcNemarTest

Multiple comparisons test functions

• chiSquaredMcTest
• fisherExactMcTest
• cochranqMcTest
• mcNemarMcTest