Quantcast

How to obtain P value from Monte Carlo sampling for adonis (permanova)?

classic Classic list List threaded Threaded
5 messages Options
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to obtain P value from Monte Carlo sampling for adonis (permanova)?

ellenpape
Dear all,

I have recently decided to switch from Permanova/Primer to R, because the
latter is freeware (and I don't know for how long I will still have a
license). However, if I cannot seem to resolve my problem (see below), I
might have to go back to using Primer/Permanova.

If I run pairwise permanova/adonis tests on my data, the number of unique
permutations is small (I have two groups, each group has 3 observations)
and the minimum P value I can get is larger than the alpha value I (and
most people) that I use to determine statistical significance (i.e. 0.05).
In the manual of the PERMANOVA+ add-on in Primer by Anderson et al. (2008)
 it is mentioned (page 28 and onwards) that when the number of unique
permutations is small (<100) than one should preferably interpret the
Monte-Carlo p value.

Does anyone know how to do this in R?


In my internet search I stumbled upon this page:
http://r.789695.n4.nabble.com/monte-carlo-simulations-in-permanova-in-vegan-package-td4714034.html.
"JAri Oksanen answered here: 2. If you want to do so, you can generate your
resampling matrices by hand and use that matrix as the argument of
permutations=. See the documentations (?adonis) which tells how to do so.
", but I don't understand how to this..

Thank you very much!

Ellen

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-ecology mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to obtain P value from Monte Carlo sampling for adonis (permanova)?

Eduard Szöcs-3
Dear Ellen,

You can create a Permutation matrix holding ALL possible permutations
for a given permutation design using permute::allPerms() [Checkout the
help therein].

This matrix can be passed to adonis/adonis2 via the permutations= argument.

Hope this helps,

Eduard

On 18.11.2016 09:39, Ellen Pape wrote:

> Dear all,
>
> I have recently decided to switch from Permanova/Primer to R, because the
> latter is freeware (and I don't know for how long I will still have a
> license). However, if I cannot seem to resolve my problem (see below), I
> might have to go back to using Primer/Permanova.
>
> If I run pairwise permanova/adonis tests on my data, the number of unique
> permutations is small (I have two groups, each group has 3 observations)
> and the minimum P value I can get is larger than the alpha value I (and
> most people) that I use to determine statistical significance (i.e. 0.05).
> In the manual of the PERMANOVA+ add-on in Primer by Anderson et al. (2008)
>  it is mentioned (page 28 and onwards) that when the number of unique
> permutations is small (<100) than one should preferably interpret the
> Monte-Carlo p value.
>
> Does anyone know how to do this in R?
>
>
> In my internet search I stumbled upon this page:
> http://r.789695.n4.nabble.com/monte-carlo-simulations-in-permanova-in-vegan-package-td4714034.html.
> "JAri Oksanen answered here: 2. If you want to do so, you can generate your
> resampling matrices by hand and use that matrix as the argument of
> permutations=. See the documentations (?adonis) which tells how to do so.
> ", but I don't understand how to this..
>
> Thank you very much!
>
> Ellen
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-ecology mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
>

--
Eduard Szöcs

Quantitative Landscape Ecology
Institute for Environmental Sciences
University Koblenz-Landau
Tel: +49 6341 280 31552
Office: Building I, Room 2.27
http://www.uni-koblenz-landau.de/en/campus-landau/faculty7/environmental-sciences/landscape-ecology/Staff/eduardszoecs

_______________________________________________
R-sig-ecology mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to obtain P value from Monte Carlo sampling for adonis (permanova)?

Cade, Brian
In reply to this post by ellenpape
Ellen:  If you are running permutation procedures with data that have very
small sample sizes in each group (your two groups of n = 3 each yields only
6!/(3!3!) = M = 20 permutations under Ho), then you just have to live with
the fact that the smallest possible P-value is 1/M (= 0.05 for your two
group example).  There is nothing magical about P < 0.05 anyways.  But as
the Monte Carlo resampling approach to obtaining permutation P-values
really is just a method to attempt to approximate the exact permutation
P-value (and usually used when M is so large that you can not enumerate it
exactly in reasonable computation time), you should not rely on it when the
number or permutations M is so small, and especially not just because you
might obtain a P < 1/M.  If you obtain a P-value <0.05 in your example
using the Monte Carlo resampling procedure, all that indicates is that the
Monte Carlo resampling approach is a poor approximation in this small
sample situation.  I think it is always preferable to obtain and interpret
the exact permutation distribution if it is easily calculable.  Using a
crummy approximation just because you want P < 0.05 seems unreasonable to
me.

Brian

Brian S. Cade, PhD

U. S. Geological Survey
Fort Collins Science Center
2150 Centre Ave., Bldg. C
Fort Collins, CO  80526-8818

email:  [hidden email] <[hidden email]>
tel:  970 226-9326


On Fri, Nov 18, 2016 at 1:39 AM, Ellen Pape <[hidden email]> wrote:

> Dear all,
>
> I have recently decided to switch from Permanova/Primer to R, because the
> latter is freeware (and I don't know for how long I will still have a
> license). However, if I cannot seem to resolve my problem (see below), I
> might have to go back to using Primer/Permanova.
>
> If I run pairwise permanova/adonis tests on my data, the number of unique
> permutations is small (I have two groups, each group has 3 observations)
> and the minimum P value I can get is larger than the alpha value I (and
> most people) that I use to determine statistical significance (i.e. 0.05).
> In the manual of the PERMANOVA+ add-on in Primer by Anderson et al. (2008)
>  it is mentioned (page 28 and onwards) that when the number of unique
> permutations is small (<100) than one should preferably interpret the
> Monte-Carlo p value.
>
> Does anyone know how to do this in R?
>
>
> In my internet search I stumbled upon this page:
> http://r.789695.n4.nabble.com/monte-carlo-simulations-in-
> permanova-in-vegan-package-td4714034.html.
> "JAri Oksanen answered here: 2. If you want to do so, you can generate your
> resampling matrices by hand and use that matrix as the argument of
> permutations=. See the documentations (?adonis) which tells how to do so.
> ", but I don't understand how to this..
>
> Thank you very much!
>
> Ellen
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-ecology mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
>
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-ecology mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to obtain P value from Monte Carlo sampling for adonis (permanova)?

Law, Jason
Brian has some really sound advice here. Using a Monte Carlo approximation rather than the exact result kind of misses the entire point. 0.05 is arbitrary and using an approximation to an exact distribution that you can easily calculate is misguided. Getting a p-value of 0.1 versus 0.05 really shouldn't result in any groundbreaking scientific differences in interpretation, despite what a journal might say about your p-values.

You should read Greenland et al (http://link.springer.com/article/10.1007/s10654-016-0149-3) which has some choice quotes from Neyman and Pearson and a lot of wisdom regarding the insanity surrounding our use of p-values.

Jason Law
Statistician, City of Portland
Water Pollution Control Laboratory
6543 N Burlington Ave,
Portland, OR

503-823-1038
[hidden email]



-----Original Message-----
From: R-sig-ecology [mailto:[hidden email]] On Behalf Of Cade, Brian
Sent: Friday, November 18, 2016 8:51 AM
To: Ellen Pape <[hidden email]>
Cc: <[hidden email]> <[hidden email]>
Subject: Re: [R-sig-eco] How to obtain P value from Monte Carlo sampling for adonis (permanova)?

Ellen:  If you are running permutation procedures with data that have very small sample sizes in each group (your two groups of n = 3 each yields only
6!/(3!3!) = M = 20 permutations under Ho), then you just have to live with the fact that the smallest possible P-value is 1/M (= 0.05 for your two group example).  There is nothing magical about P < 0.05 anyways.  But as the Monte Carlo resampling approach to obtaining permutation P-values really is just a method to attempt to approximate the exact permutation P-value (and usually used when M is so large that you can not enumerate it exactly in reasonable computation time), you should not rely on it when the number or permutations M is so small, and especially not just because you might obtain a P < 1/M.  If you obtain a P-value <0.05 in your example using the Monte Carlo resampling procedure, all that indicates is that the Monte Carlo resampling approach is a poor approximation in this small sample situation.  I think it is always preferable to obtain and interpret the exact permutation distribution if it is easily calculable.  Using a crummy approximation just because you !
 want P < 0.05 seems unreasonable to me.

Brian

Brian S. Cade, PhD

U. S. Geological Survey
Fort Collins Science Center
2150 Centre Ave., Bldg. C
Fort Collins, CO  80526-8818

email:  [hidden email] <[hidden email]>
tel:  970 226-9326


On Fri, Nov 18, 2016 at 1:39 AM, Ellen Pape <[hidden email]> wrote:

> Dear all,
>
> I have recently decided to switch from Permanova/Primer to R, because
> the latter is freeware (and I don't know for how long I will still
> have a license). However, if I cannot seem to resolve my problem (see
> below), I might have to go back to using Primer/Permanova.
>
> If I run pairwise permanova/adonis tests on my data, the number of
> unique permutations is small (I have two groups, each group has 3
> observations) and the minimum P value I can get is larger than the
> alpha value I (and most people) that I use to determine statistical significance (i.e. 0.05).
> In the manual of the PERMANOVA+ add-on in Primer by Anderson et al.
> (2008)  it is mentioned (page 28 and onwards) that when the number of
> unique permutations is small (<100) than one should preferably
> interpret the Monte-Carlo p value.
>
> Does anyone know how to do this in R?
>
>
> In my internet search I stumbled upon this page:
> http://r.789695.n4.nabble.com/monte-carlo-simulations-in-
> permanova-in-vegan-package-td4714034.html.
> "JAri Oksanen answered here: 2. If you want to do so, you can generate
> your resampling matrices by hand and use that matrix as the argument
> of permutations=. See the documentations (?adonis) which tells how to do so.
> ", but I don't understand how to this..
>
> Thank you very much!
>
> Ellen
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-ecology mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
>
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-ecology mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-ecology

_______________________________________________
R-sig-ecology mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: How to obtain P value from Monte Carlo sampling for adonis (permanova)?

ellenpape
Ok! Thank you all for your advice! This really helps! I will not be using
Monte Carlo approx to obtain the P value for the data that I have..

On 18 November 2016 at 20:39, Law, Jason <[hidden email]>
wrote:

> Brian has some really sound advice here. Using a Monte Carlo approximation
> rather than the exact result kind of misses the entire point. 0.05 is
> arbitrary and using an approximation to an exact distribution that you can
> easily calculate is misguided. Getting a p-value of 0.1 versus 0.05 really
> shouldn't result in any groundbreaking scientific differences in
> interpretation, despite what a journal might say about your p-values.
>
> You should read Greenland et al (http://link.springer.com/
> article/10.1007/s10654-016-0149-3) which has some choice quotes from
> Neyman and Pearson and a lot of wisdom regarding the insanity surrounding
> our use of p-values.
>
> Jason Law
> Statistician, City of Portland
> Water Pollution Control Laboratory
> 6543 N Burlington Ave,
> Portland, OR
>
> 503-823-1038
> [hidden email]
>
>
>
> -----Original Message-----
> From: R-sig-ecology [mailto:[hidden email]] On
> Behalf Of Cade, Brian
> Sent: Friday, November 18, 2016 8:51 AM
> To: Ellen Pape <[hidden email]>
> Cc: <[hidden email]> <[hidden email]>
> Subject: Re: [R-sig-eco] How to obtain P value from Monte Carlo sampling
> for adonis (permanova)?
>
> Ellen:  If you are running permutation procedures with data that have very
> small sample sizes in each group (your two groups of n = 3 each yields only
> 6!/(3!3!) = M = 20 permutations under Ho), then you just have to live with
> the fact that the smallest possible P-value is 1/M (= 0.05 for your two
> group example).  There is nothing magical about P < 0.05 anyways.  But as
> the Monte Carlo resampling approach to obtaining permutation P-values
> really is just a method to attempt to approximate the exact permutation
> P-value (and usually used when M is so large that you can not enumerate it
> exactly in reasonable computation time), you should not rely on it when the
> number or permutations M is so small, and especially not just because you
> might obtain a P < 1/M.  If you obtain a P-value <0.05 in your example
> using the Monte Carlo resampling procedure, all that indicates is that the
> Monte Carlo resampling approach is a poor approximation in this small
> sample situation.  I think it is always preferable to obtain and interpret
> the exact permutation distribution if it is easily calculable.  Using a
> crummy approximation just because you want P < 0.05 seems unreasonable to
> me.
>
> Brian
>
> Brian S. Cade, PhD
>
> U. S. Geological Survey
> Fort Collins Science Center
> 2150 Centre Ave., Bldg. C
> Fort Collins, CO  80526-8818
>
> email:  [hidden email] <[hidden email]>
> tel:  970 226-9326
>
>
> On Fri, Nov 18, 2016 at 1:39 AM, Ellen Pape <[hidden email]> wrote:
>
> > Dear all,
> >
> > I have recently decided to switch from Permanova/Primer to R, because
> > the latter is freeware (and I don't know for how long I will still
> > have a license). However, if I cannot seem to resolve my problem (see
> > below), I might have to go back to using Primer/Permanova.
> >
> > If I run pairwise permanova/adonis tests on my data, the number of
> > unique permutations is small (I have two groups, each group has 3
> > observations) and the minimum P value I can get is larger than the
> > alpha value I (and most people) that I use to determine statistical
> significance (i.e. 0.05).
> > In the manual of the PERMANOVA+ add-on in Primer by Anderson et al.
> > (2008)  it is mentioned (page 28 and onwards) that when the number of
> > unique permutations is small (<100) than one should preferably
> > interpret the Monte-Carlo p value.
> >
> > Does anyone know how to do this in R?
> >
> >
> > In my internet search I stumbled upon this page:
> > http://r.789695.n4.nabble.com/monte-carlo-simulations-in-
> > permanova-in-vegan-package-td4714034.html.
> > "JAri Oksanen answered here: 2. If you want to do so, you can generate
> > your resampling matrices by hand and use that matrix as the argument
> > of permutations=. See the documentations (?adonis) which tells how to do
> so.
> > ", but I don't understand how to this..
> >
> > Thank you very much!
> >
> > Ellen
> >
> >         [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-ecology mailing list
> > [hidden email]
> > https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
> >
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-ecology mailing list
> [hidden email]
> https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
>

        [[alternative HTML version deleted]]

_______________________________________________
R-sig-ecology mailing list
[hidden email]
https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
Loading...