A short introduction into Bayesian evaluation of informative hypotheses as an alternative to exploratory comparisons of multiple group means

This paper presents an introduction into Bayesian evaluation of informative hypotheses, that is, hypotheses representing explicit expectations about multiple group means (Hoijtink, 2011; Hoijtink, Klugkist & Boelen, 2008). The authors begin by discussing some limits of exploratory methods before presenting a non-technical overview of the Bayesian approach. References are provided for the technical details. A particular effort is made to illustrate the method with an example from psychology. References to software, more elaborate textbooks and tutorials enable researchers to apply this novel method to their own data.

S. Béland and G. Raîche are at the Collectif pour le développement et les applications en mesure et évaluation, département d'éducation et de pédagogie (Cdame); I. Klugkist is at the Department of Methodology and Statistics; David Magis is at the département de mathématiques de l'Université de Liège.hypothesis is not very informative because it states that something is happening but without any specification about what is going on.
In a situation where the researcher wants to evaluate H0 against Ha it is common to apply a J group analysis of variance (ANOVA).The model can be written as (1) where yi is a dependent variable of interest, μj denotes the mean of group j, dji is one if person i is a member of group j, and zero otherwise, and .The F-statistic, , where is the between group means variance and is the within groups or residual variance, and associated F-distribution are used to test H0 against Ha.The resulting p-value is sometimes called a measure of surprise, that is, it represents the incompatibility of data with the null hypothesis.
A significant ANOVA is usually followed by multiple pair wise comparisons.Using a traditional approach like, for instance, pair wise comparisons with a Bonferroni correction can lead to some methodological problems that will be discussed in the next section.The aim of this article is to present an alternative method based on a Bayesian approach that can evaluate specific expectations about multiple group means directly.Hypotheses expressing these expected patterns of means are called informative hypotheses and contain combinations of equality (=) and inequality (< or >) constraints; e.g., Hinf1: μ1 > μ2 > μ3 or Hinf2: (μ1 = μ3) > μ2.
The remainder of this paper is organized as follows.Section 1 discusses an exploratory approach for the evaluation of informative hypotheses and some of its disadvantages.Section 2 presents the Bayesian approach.Then, an example from psychology is provided in Section 3 and the paper is concluded with some reference material in Section 4.

Evaluating informative hypotheses through multiple comparisons
Many exploratory strategies are available to produce comparisons between groups to test a hypothesis.These approaches are based on pair wise comparisons between the groups under investigation.For example, a hypothesis concerning four means can be followed by a total of six pair wise tests.For each comparison, a t test can be used with a pre-specified α level; usually 0.05.This strategy does, however, not allow for family-wise error control.Here, the family-wise probability of having one or more false discoveries is ; that is, much larger than the pre-specified level of 0.05.A well-known strategy to control family-wise error is to use the Bonferroni correction where the α level is divided by the number of comparisons.For example, in the case of six tests, the α level per test becomes 0.05/6 = 0.0083.Unfortunately, the control of the family-wise error through a Bonferroni correction also causes a great loss of power.Although other strategies that are less conservative than the Bonferonni correction have been developed, finding a balance between proper control over the type I error and power remains problematic (Maxwell, 2004).
Another limitation of using multiple comparisons for the evaluation of informative hypotheses is that the test results can be inconclusive (i.e., the hypothesis is partially but not completely supported) or logically inconsistent (e.g., H0: μ1 = μ2 and H0: μ1 = μ3 are not rejected (i.e., not statistically significant), while H0: μ2 = μ3 is rejected (p<.05)).
Finally, researchers may have multiple, competing informative hypotheses.While with significance (multiple) testing it is not possible to compare them against each other, with the Bayesian approach presented in this paper this is straightforward, irrespective of the hypotheses to be compared being nested or non-nested.A hypothetical example of two competing hypotheses, Hinf1 and Hinf2, was provided in the introduction and in Section 3 we will present a real data example with three informative hypotheses.But first, in the next section, we will present the Bayesian method that was specifically designed for the evaluation of one or more informative hypotheses.

Bayesian evaluation of informative hypotheses
Hoijtink (2011) and Hoijtink, Klugkist & Boelen (2008) presented a method for the evaluation of informative hypotheses that (i) does not suffer from such multiple testing issues, (ii) has the capacity to test more informative hypotheses than the traditional H0 and Ha, and (iii) can be used as a confirmatory method to mutually compare two or more hypotheses of interest.In the next subsection, the Bayes factor, a Bayesian model selection tool, is presented.This is followed by some remarks about the prior and posterior distributions in Section 2.2, and a presentation of the sampling based estimation of the Bayes factor in Section 2.3.

The Bayes factor
Bayesian model selection is a procedure that can be used to mutually evaluate two competing hypotheses.The Bayes factor comparing the support in the data for a hypothesis Ht relative to Ht' is defined as: , that is, it is the ratio of two marginal likelihoods (Kass & Raftery, 1995).Loosely stated, a marginal likelihood is the likelihood of observed data given a specific model or hypothesis, taking both the fit and the complexity/size (also known as parsimoniousness) of the hypotheses into account.
A resulting Btt' > 1 implies more support for Ht than for Ht' whereas a Btt' < 1 implies the opposite.A resulting Bayes factor of, for instance, 4 can be interpreted as four times stronger support for Ht than for Ht'.Klugkist, Laudy & Hoijtink (2005) derived a simplified formulation of the Bayes factor for the comparison of a constrained hypothesis against its unconstrained counterpart.This simplification can be made because an informative hypothesis as, for instance, Hinf1: μ1 > μ2 > μ3, is nested in the encompassing unconstrained hypothesis Ha: μ1, μ2, μ3.The Bayes factor Binf,a is, in this context, defined as: where finf is the proportion of the posterior distribution of the unconstrained hypothesis that is in agreement with the constraints of Hinf, and cinf is the proportion of the unconstrained prior in agreement with the constraints.The role of priors and posteriors in the Bayesian approach will be further elaborated in the following section.Note finally, that the constant cinf can be interpreted as the complexity or size of the constrained hypothesis relative to the unconstrained hypothesis, and finf as the fit of Hinf.

Prior and posterior distributions
In Bayesian statistics, a prior distribution must be specified for the parameters of each model under investigation.The prior represents the knowledge about the model parameters before observing the data and can be informative (i.e., "knowledge" is based on experts opinions, historical data, or any acceptable assumption from theory) or nonor low-informative (specifying as little prior information about the parameters as possible).For a mean μ a low-informative prior is, for instance, a normal distribution with a very large variance, which is almost equal to stating that any value for μ is, a priori, equally likely.
Updating the prior distribution with observed data provides the posterior distribution.The posterior, therefore, represents the knowledge about the model parameters after seeing the data while taking the prior information into account.With low-informative priors, the posterior is dominated by the data and Bayesian parameter estimates will, in this case, provide results that are very similar to non-Bayesian estimators.Low-informative priors can, however, strongly affect Bayesian model selection results and should therefore be used with great care in those applications (e.g., Berger & Pericchi, 1996).
In the context of evaluating informative hypotheses, Klugkist, Laudy & Hoijtink (2005) formulated general criteria for the specification of the prior for the unconstrained hypothesis.Due to the nesting, priors for the constrained hypotheses follow automatically by truncation according to the constraints (see Klugkist & Hoijtink (2007) for an elaborate explanation).They showed that for hypotheses containing order constraints (i.e., < or >) it is possible to define low-informative priors that do not affect the resulting Bayes factor.However, as soon as equality constraints (i.e., =) are present this is not the case and the choice of priors becomes more complicated.How this is done in the proposed method and accompanying software is extensively described in, for instance, Hoijtink (2011) and Hoijtink, Klugkist & Boelen (2008).

Sampling based estimation of the Bayes factor
In the ANOVA context, a low-informative prior for J means μj and the residual variance σe 2 is specified.To obtain conditional on the current values of all other parameters (and in the case of the posterior also conditional on the data) in a fixed order.step 3: repeat step 2 many times.
With the Gibbs sampler, many values for each of the parameters are obtained and together provide the marginal posterior distribution of each parameter and subsequently of all posterior estimates of interest.Note, however, that some important issues always need to be considered, that is, burnin and convergence (initial values of the sampler need to be discarded because the chain is still 'starting up') and sufficient total number of iterations.Several diagnostics and plots are available to monitor this; see for a general overview, for instance, Cowles & Carlin (1996).
To illustrate how the sampled values provide an estimate for Binf,a we present, for a 3 means ANOVA, (part of) a sample of 1000 iterations from an unconstrained prior and posterior in Table 1 and 2, respectively.On each row in both tables, it is determined if the sampled values for μ1, μ2, μ3 are in agreement with the hypothesis of interest: Hinf1: μ1 > μ2 > μ3.Consider, for instance the first row of Table 1 with means μ1 = 0.57, μ2 = 0.41, μ3 = 0.20.These means are indeed increasing and therefore in the last column "1 hit" is recorded.Likewise, on the next row the last column lists a "0" because the means violate the order stated in Hinf1.Overall, Table 1 shows that the estimate for the complexity cinf1 is 163 / 1000 = 0.163 and Table 2 shows an estimate of 659 / 1000 = 0.659 for the fit finf.
In this hypothetical illustration, the resulting Bayes factor is Binf1,a = 0.659 / 0.163 = 4.04, that is, the constrained hypothesis is supported by the data.In the next section an analysis based on a real data set from a psychological clinical trial is discussed.

Analysis of a real data set
A recent study by Van den Hout et al. (2012) aimed to investigate the efficacy of the use of tones as the source of bilateral stimulation in the treatment of posttraumatic stress disorder (PTSD).Interventions using Eye Movement and Desensitization and Reprocessing (EMDR), and not tones, have repeatedly proven to be effective.In the clinical practice, however, eye movements have been replaced by tones; apparently it is assumed that tones are equally or more effective.
In a clinical study, 12 patients were asked to recall their most upsetting memory while making eye movements (EM condition), or while hearing beep tones (tones condition), or without bilateral stimulation (recall only condition).The researchers formulated three competing hypotheses: For the theoretical motivation of the informative hypothesis and all details of the study we refer to the original publication and references therein.Here, we will summarize how the Bayesian approach provides direct answers to the question about the effectiveness of using tones: if tones are equally effective as the (evidence-based) EM intervention, Hinf1 should receive the strongest support.Hinf2 represents the possibility that tones are not effective at all, and support for the third hypothesis would suggest some effect of tones but inferiority to EM.
In Table 3, the results for two outcome measures are provided; reduction in emotionality and reduction in vividness of trauma memories.For both measures the findings clearly show that there is no support for Hinf1 (Bs < 1), that is, there is reason to believe that tones are not as effective as EM.Furthermore, the Bayes factors for the second and third informative hypothesis indicate stronger evidence for no effect at all (2.07 and 2.50) than for a smaller effect of tones compared to EM (1.12 and 1.24).Note that,

Conclusion
In this paper we shortly introduced the Bayesian model selection approach to the evaluation of informative hypotheses.With this approach a powerful tool is provided that can directly evaluate specific expectations about the outcomes of a study.As was shown in the psychological illustration, it is also possible to mutually compare different competing hypotheses on the same data set.
In Section 2, the methodology was introduced for a simple (between subjects) ANOVA.In this context, a FORTRAN program called confirmatoryANOVA (Kuiper, Klugkist and Hoijtink, 2010) can be used.The illustration in Section 3 concerned a within subjects design and therefore requires other software.Mulder et al. (2009Mulder et al. ( , 2010) )

Table 2 : Posterior estimation for means μ1, μ2 and μ3 using the Gibbs sampler
Casella & George, 1992;Jackman, 2009)lo (MCMC) sampling to obtain a sample of parameter values from the unconstrained prior.In a similar vein, finf is estimated by taking an MCMC sample from the unconstrained posterior.For both prior and posterior, a Gibbs sampler (see also, for instance,Casella & George, 1992;Jackman, 2009)consisting of the following steps is applied: step 1: specification of initial values for the parameters μ1, …, μJ, σe 2 .step 2: sample a new value for each parameter μ1, …, μJ, σe 2

Table 3 : Bayes factor comparing the informative hypothesis against the unconstrained hypothesis for two outcome measures.
Van den Hout et al. (2012)monstrate the usefulness of the Bayesian approach for researchers with explicit expectations about multiple group means.Readers that have an interest in the particular study are referred toVan den Hout et al. (2012)for an elaborate discussion of the interpretation (much more careful than presented here) of the results.