Dear all,
I have recently decided to switch from Permanova/Primer to R, because the latter is freeware (and I don't know for how long I will still have a license). However, if I cannot seem to resolve my problem (see below), I might have to go back to using Primer/Permanova. If I run pairwise permanova/adonis tests on my data, the number of unique permutations is small (I have two groups, each group has 3 observations) and the minimum P value I can get is larger than the alpha value I (and most people) that I use to determine statistical significance (i.e. 0.05). In the manual of the PERMANOVA+ add-on in Primer by Anderson et al. (2008) it is mentioned (page 28 and onwards) that when the number of unique permutations is small (<100) than one should preferably interpret the Monte-Carlo p value. Does anyone know how to do this in R? In my internet search I stumbled upon this page: http://r.789695.n4.nabble.com/monte-carlo-simulations-in-permanova-in-vegan-package-td4714034.html. "JAri Oksanen answered here: 2. If you want to do so, you can generate your resampling matrices by hand and use that matrix as the argument of permutations=. See the documentations (?adonis) which tells how to do so. ", but I don't understand how to this.. Thank you very much! Ellen [[alternative HTML version deleted]] _______________________________________________ R-sig-ecology mailing list [hidden email] https://stat.ethz.ch/mailman/listinfo/r-sig-ecology |
Dear Ellen,
You can create a Permutation matrix holding ALL possible permutations for a given permutation design using permute::allPerms() [Checkout the help therein]. This matrix can be passed to adonis/adonis2 via the permutations= argument. Hope this helps, Eduard On 18.11.2016 09:39, Ellen Pape wrote: > Dear all, > > I have recently decided to switch from Permanova/Primer to R, because the > latter is freeware (and I don't know for how long I will still have a > license). However, if I cannot seem to resolve my problem (see below), I > might have to go back to using Primer/Permanova. > > If I run pairwise permanova/adonis tests on my data, the number of unique > permutations is small (I have two groups, each group has 3 observations) > and the minimum P value I can get is larger than the alpha value I (and > most people) that I use to determine statistical significance (i.e. 0.05). > In the manual of the PERMANOVA+ add-on in Primer by Anderson et al. (2008) > it is mentioned (page 28 and onwards) that when the number of unique > permutations is small (<100) than one should preferably interpret the > Monte-Carlo p value. > > Does anyone know how to do this in R? > > > In my internet search I stumbled upon this page: > http://r.789695.n4.nabble.com/monte-carlo-simulations-in-permanova-in-vegan-package-td4714034.html. > "JAri Oksanen answered here: 2. If you want to do so, you can generate your > resampling matrices by hand and use that matrix as the argument of > permutations=. See the documentations (?adonis) which tells how to do so. > ", but I don't understand how to this.. > > Thank you very much! > > Ellen > > [[alternative HTML version deleted]] > > _______________________________________________ > R-sig-ecology mailing list > [hidden email] > https://stat.ethz.ch/mailman/listinfo/r-sig-ecology > -- Eduard Szöcs Quantitative Landscape Ecology Institute for Environmental Sciences University Koblenz-Landau Tel: +49 6341 280 31552 Office: Building I, Room 2.27 http://www.uni-koblenz-landau.de/en/campus-landau/faculty7/environmental-sciences/landscape-ecology/Staff/eduardszoecs _______________________________________________ R-sig-ecology mailing list [hidden email] https://stat.ethz.ch/mailman/listinfo/r-sig-ecology |
In reply to this post by ellenpape
Ellen: If you are running permutation procedures with data that have very
small sample sizes in each group (your two groups of n = 3 each yields only 6!/(3!3!) = M = 20 permutations under Ho), then you just have to live with the fact that the smallest possible P-value is 1/M (= 0.05 for your two group example). There is nothing magical about P < 0.05 anyways. But as the Monte Carlo resampling approach to obtaining permutation P-values really is just a method to attempt to approximate the exact permutation P-value (and usually used when M is so large that you can not enumerate it exactly in reasonable computation time), you should not rely on it when the number or permutations M is so small, and especially not just because you might obtain a P < 1/M. If you obtain a P-value <0.05 in your example using the Monte Carlo resampling procedure, all that indicates is that the Monte Carlo resampling approach is a poor approximation in this small sample situation. I think it is always preferable to obtain and interpret the exact permutation distribution if it is easily calculable. Using a crummy approximation just because you want P < 0.05 seems unreasonable to me. Brian Brian S. Cade, PhD U. S. Geological Survey Fort Collins Science Center 2150 Centre Ave., Bldg. C Fort Collins, CO 80526-8818 email: [hidden email] <[hidden email]> tel: 970 226-9326 On Fri, Nov 18, 2016 at 1:39 AM, Ellen Pape <[hidden email]> wrote: > Dear all, > > I have recently decided to switch from Permanova/Primer to R, because the > latter is freeware (and I don't know for how long I will still have a > license). However, if I cannot seem to resolve my problem (see below), I > might have to go back to using Primer/Permanova. > > If I run pairwise permanova/adonis tests on my data, the number of unique > permutations is small (I have two groups, each group has 3 observations) > and the minimum P value I can get is larger than the alpha value I (and > most people) that I use to determine statistical significance (i.e. 0.05). > In the manual of the PERMANOVA+ add-on in Primer by Anderson et al. (2008) > it is mentioned (page 28 and onwards) that when the number of unique > permutations is small (<100) than one should preferably interpret the > Monte-Carlo p value. > > Does anyone know how to do this in R? > > > In my internet search I stumbled upon this page: > http://r.789695.n4.nabble.com/monte-carlo-simulations-in- > permanova-in-vegan-package-td4714034.html. > "JAri Oksanen answered here: 2. If you want to do so, you can generate your > resampling matrices by hand and use that matrix as the argument of > permutations=. See the documentations (?adonis) which tells how to do so. > ", but I don't understand how to this.. > > Thank you very much! > > Ellen > > [[alternative HTML version deleted]] > > _______________________________________________ > R-sig-ecology mailing list > [hidden email] > https://stat.ethz.ch/mailman/listinfo/r-sig-ecology > > [[alternative HTML version deleted]] _______________________________________________ R-sig-ecology mailing list [hidden email] https://stat.ethz.ch/mailman/listinfo/r-sig-ecology |
Brian has some really sound advice here. Using a Monte Carlo approximation rather than the exact result kind of misses the entire point. 0.05 is arbitrary and using an approximation to an exact distribution that you can easily calculate is misguided. Getting a p-value of 0.1 versus 0.05 really shouldn't result in any groundbreaking scientific differences in interpretation, despite what a journal might say about your p-values.
You should read Greenland et al (http://link.springer.com/article/10.1007/s10654-016-0149-3) which has some choice quotes from Neyman and Pearson and a lot of wisdom regarding the insanity surrounding our use of p-values. Jason Law Statistician, City of Portland Water Pollution Control Laboratory 6543 N Burlington Ave, Portland, OR 503-823-1038 [hidden email] -----Original Message----- From: R-sig-ecology [mailto:[hidden email]] On Behalf Of Cade, Brian Sent: Friday, November 18, 2016 8:51 AM To: Ellen Pape <[hidden email]> Cc: <[hidden email]> <[hidden email]> Subject: Re: [R-sig-eco] How to obtain P value from Monte Carlo sampling for adonis (permanova)? Ellen: If you are running permutation procedures with data that have very small sample sizes in each group (your two groups of n = 3 each yields only 6!/(3!3!) = M = 20 permutations under Ho), then you just have to live with the fact that the smallest possible P-value is 1/M (= 0.05 for your two group example). There is nothing magical about P < 0.05 anyways. But as the Monte Carlo resampling approach to obtaining permutation P-values really is just a method to attempt to approximate the exact permutation P-value (and usually used when M is so large that you can not enumerate it exactly in reasonable computation time), you should not rely on it when the number or permutations M is so small, and especially not just because you might obtain a P < 1/M. If you obtain a P-value <0.05 in your example using the Monte Carlo resampling procedure, all that indicates is that the Monte Carlo resampling approach is a poor approximation in this small sample situation. I think it is always preferable to obtain and interpret the exact permutation distribution if it is easily calculable. Using a crummy approximation just because you ! want P < 0.05 seems unreasonable to me. Brian Brian S. Cade, PhD U. S. Geological Survey Fort Collins Science Center 2150 Centre Ave., Bldg. C Fort Collins, CO 80526-8818 email: [hidden email] <[hidden email]> tel: 970 226-9326 On Fri, Nov 18, 2016 at 1:39 AM, Ellen Pape <[hidden email]> wrote: > Dear all, > > I have recently decided to switch from Permanova/Primer to R, because > the latter is freeware (and I don't know for how long I will still > have a license). However, if I cannot seem to resolve my problem (see > below), I might have to go back to using Primer/Permanova. > > If I run pairwise permanova/adonis tests on my data, the number of > unique permutations is small (I have two groups, each group has 3 > observations) and the minimum P value I can get is larger than the > alpha value I (and most people) that I use to determine statistical significance (i.e. 0.05). > In the manual of the PERMANOVA+ add-on in Primer by Anderson et al. > (2008) it is mentioned (page 28 and onwards) that when the number of > unique permutations is small (<100) than one should preferably > interpret the Monte-Carlo p value. > > Does anyone know how to do this in R? > > > In my internet search I stumbled upon this page: > http://r.789695.n4.nabble.com/monte-carlo-simulations-in- > permanova-in-vegan-package-td4714034.html. > "JAri Oksanen answered here: 2. If you want to do so, you can generate > your resampling matrices by hand and use that matrix as the argument > of permutations=. See the documentations (?adonis) which tells how to do so. > ", but I don't understand how to this.. > > Thank you very much! > > Ellen > > [[alternative HTML version deleted]] > > _______________________________________________ > R-sig-ecology mailing list > [hidden email] > https://stat.ethz.ch/mailman/listinfo/r-sig-ecology > > [[alternative HTML version deleted]] _______________________________________________ R-sig-ecology mailing list [hidden email] https://stat.ethz.ch/mailman/listinfo/r-sig-ecology _______________________________________________ R-sig-ecology mailing list [hidden email] https://stat.ethz.ch/mailman/listinfo/r-sig-ecology |
Ok! Thank you all for your advice! This really helps! I will not be using
Monte Carlo approx to obtain the P value for the data that I have.. On 18 November 2016 at 20:39, Law, Jason <[hidden email]> wrote: > Brian has some really sound advice here. Using a Monte Carlo approximation > rather than the exact result kind of misses the entire point. 0.05 is > arbitrary and using an approximation to an exact distribution that you can > easily calculate is misguided. Getting a p-value of 0.1 versus 0.05 really > shouldn't result in any groundbreaking scientific differences in > interpretation, despite what a journal might say about your p-values. > > You should read Greenland et al (http://link.springer.com/ > article/10.1007/s10654-016-0149-3) which has some choice quotes from > Neyman and Pearson and a lot of wisdom regarding the insanity surrounding > our use of p-values. > > Jason Law > Statistician, City of Portland > Water Pollution Control Laboratory > 6543 N Burlington Ave, > Portland, OR > > 503-823-1038 > [hidden email] > > > > -----Original Message----- > From: R-sig-ecology [mailto:[hidden email]] On > Behalf Of Cade, Brian > Sent: Friday, November 18, 2016 8:51 AM > To: Ellen Pape <[hidden email]> > Cc: <[hidden email]> <[hidden email]> > Subject: Re: [R-sig-eco] How to obtain P value from Monte Carlo sampling > for adonis (permanova)? > > Ellen: If you are running permutation procedures with data that have very > small sample sizes in each group (your two groups of n = 3 each yields only > 6!/(3!3!) = M = 20 permutations under Ho), then you just have to live with > the fact that the smallest possible P-value is 1/M (= 0.05 for your two > group example). There is nothing magical about P < 0.05 anyways. But as > the Monte Carlo resampling approach to obtaining permutation P-values > really is just a method to attempt to approximate the exact permutation > P-value (and usually used when M is so large that you can not enumerate it > exactly in reasonable computation time), you should not rely on it when the > number or permutations M is so small, and especially not just because you > might obtain a P < 1/M. If you obtain a P-value <0.05 in your example > using the Monte Carlo resampling procedure, all that indicates is that the > Monte Carlo resampling approach is a poor approximation in this small > sample situation. I think it is always preferable to obtain and interpret > the exact permutation distribution if it is easily calculable. Using a > crummy approximation just because you want P < 0.05 seems unreasonable to > me. > > Brian > > Brian S. Cade, PhD > > U. S. Geological Survey > Fort Collins Science Center > 2150 Centre Ave., Bldg. C > Fort Collins, CO 80526-8818 > > email: [hidden email] <[hidden email]> > tel: 970 226-9326 > > > On Fri, Nov 18, 2016 at 1:39 AM, Ellen Pape <[hidden email]> wrote: > > > Dear all, > > > > I have recently decided to switch from Permanova/Primer to R, because > > the latter is freeware (and I don't know for how long I will still > > have a license). However, if I cannot seem to resolve my problem (see > > below), I might have to go back to using Primer/Permanova. > > > > If I run pairwise permanova/adonis tests on my data, the number of > > unique permutations is small (I have two groups, each group has 3 > > observations) and the minimum P value I can get is larger than the > > alpha value I (and most people) that I use to determine statistical > significance (i.e. 0.05). > > In the manual of the PERMANOVA+ add-on in Primer by Anderson et al. > > (2008) it is mentioned (page 28 and onwards) that when the number of > > unique permutations is small (<100) than one should preferably > > interpret the Monte-Carlo p value. > > > > Does anyone know how to do this in R? > > > > > > In my internet search I stumbled upon this page: > > http://r.789695.n4.nabble.com/monte-carlo-simulations-in- > > permanova-in-vegan-package-td4714034.html. > > "JAri Oksanen answered here: 2. If you want to do so, you can generate > > your resampling matrices by hand and use that matrix as the argument > > of permutations=. See the documentations (?adonis) which tells how to do > so. > > ", but I don't understand how to this.. > > > > Thank you very much! > > > > Ellen > > > > [[alternative HTML version deleted]] > > > > _______________________________________________ > > R-sig-ecology mailing list > > [hidden email] > > https://stat.ethz.ch/mailman/listinfo/r-sig-ecology > > > > > > [[alternative HTML version deleted]] > > _______________________________________________ > R-sig-ecology mailing list > [hidden email] > https://stat.ethz.ch/mailman/listinfo/r-sig-ecology > [[alternative HTML version deleted]] _______________________________________________ R-sig-ecology mailing list [hidden email] https://stat.ethz.ch/mailman/listinfo/r-sig-ecology |
Free forum by Nabble | Edit this page |