A Study of Clustered Data and Approaches to Its Analysis

Part 1.1: A specific example of clustered data

Our focus is on testing hypotheses by comparing observations from two groups, such as a treated group versus a control group. As a case in point, consider a hypothetical experiment directed to assess whether a drug leads to altered rates of exocytosis from synapses in vitro. To address this question on a given day, neurons on 10 coverslips are each to be treated with 10 μg/ml drug, while neurons on another 9 coverslips are to be treated with vehicle. The following day the rate of exocytosis at between 10 and 40 synapses per coverslip is to be determined. Let us say that such an experiment yielded exocytosis rate data on a total of 123 synapses from 10 drug-treated coverslips and data from 157 synapses from 9 untreated coverslips. The aim of the study is to test the null hypothesis that there is no effect of the drug on the exocytosis rate.

In fact, in this hypothetical experiment, the synapses are nested within coverslip and, hence, the data obtained from individual coverslips constitute clusters. For each synapse, we observe unique exocytosis kinetics, and the exocytosis rate can be estimated. The exocytosis rates for treated and untreated synapses could be compared using, for example, a two-sample t test (if the data are approximately normally distributed). However, it is possible that other conditions within a coverslip, apart from treatment, are more similar than conditions on different coverslips. Hence, the synapses from the same coverslip share similar conditions and may be more or less likely to release neurotransmitter through exocytosis than synapses from different coverslips, quite apart from the treatment effect.

In the above example, the individual observations belong to a number of different clusters. While we are not usually primarily interested in the cluster effect, it must be taken into account to ensure validity of the treatment comparison. If the clustering is not taken into account for this type of data, then the variability is likely to be underestimated and the results of the analysis invalidated.

Part 1.2: Two classes of clustered data

Throughout this document, we will refer to two classes of clustered data, which for convenience we will call case 1 data and case 2 data.

In the hypothesis testing context, it is useful to distinguish between two different kinds of clustered datasets: case 1, where only one of the groups that is being compared is represented in each cluster; and case 2, where (at least some of) the clusters contain observations from both groups. The hypothetical experiment in Part 1.1 above is an example of case 1. Another example of case 1 is a cluster randomized trial where, for practical reasons, it is necessary to randomize all of the patients from a particular clinic to the same intervention. For example, all patients from one clinic might receive the usual standard of care, while all patients from another clinic participate in an additional health care education program. Here, the unit of randomization is the clinic. The data from each clinic is clustered and each cluster will contain either patients receiving standard care or patients receiving the education program in addition to standard care. An example of case 2 is a multicenter clinical trial where, to widen the patient base, patients are recruited from a number of different centers, but within each center some patients will be randomized to one treatment and some to the other treatment. Here, the unit of randomization is the patient. Data from each clinic are clustered, but this time the clusters will contain both control and treated patients. The best methods for analyzing case 1 and case 2 data may differ, and this will be discussed in later sections.

It is a fact that clustered data are very common in neuroscience. Experiments yield multiple observations, animals yield observations on multiple litter members, or, at a series of times, research centers contribute data on multiple patients. In each case we are faced with clustered data that raise the question: is conventional statistical analysis sufficient, or should a method be used that specifically accounts for clustering? We shall address this question in detail.

Part 1.3: A statistical model for intracluster correlation

Suppose we collect data from several independent experiments that represent the clusters. The similarity between clusters might be accounted for by assuming that there is a common mean response within a cluster that varies randomly between clusters. The response within clusters also varies due to individual-level heterogeneity. This leads to the model for the data, y_ik = μ + b_k + ε_ik, where y_ik is the value of the response variable for unit i in cluster k, and μ is the overall mean. The remaining two terms represent the two levels of variation in the data, with ε_ik representing the “within-cluster” variation between observations from the same cluster, and b_k the “between cluster” variation. In statistical terms, b_k is a random cluster-specific effect with mean zero and variance σ_b², and ε_ik is a residual error term with mean zero and variance σ_w². The residual errors from different individuals (ε_ik) and the random effects from different clusters (b_k) are assumed to be independent. In this model, the correlation between two different observations from the same cluster arises because of the random effect b_k, and this correlation is equal to σ_b²/(σ_b² + σ_w²). While observations within each cluster are correlated, there is no correlation between observations from different clusters because the random effects from different clusters and the residual errors from different individuals are independent.

Part 1.4: Analysis of clustered data

Having defined clustered data, we will now address the various ways in which clustering can be treated. In reviewing the literature, it would appear that four approaches have generally been used in the analysis of clustered data: (A) ignoring clustering; (B) reducing clusters to independent observations; (C) fixed effects regression/ANOVA approaches; and (D) explicitly accounting for clustering.

Each of these approaches will be discussed in detail here. By way of example, we will make specific reference to optical analysis of presynaptic function. Optical methods of studying presynaptic function use fluorescent tracers, such as styryl dyes or synaptopHluorin, to mark synaptic vesicles that are undergoing release. Optical methods yield clustered data because each independent experiment provides data from multiple synapses that can be visualized individually under the microscope. Observations from a population of synapses within the same experiment are clustered and hence are more similar than observations of synapses acquired in separate experiments. Studies using these optical methods have adopted various approaches to analyze their data, and thus these studies provide an insight into ways in which clustered data may be approached.

Part 2.1: Generation of simulated data

In preparing our simulated data, the number of clusters and the total number of observations were fixed for each simulation. Cluster sizes were generated from a multinomial distribution. We considered separately the two types of data distinguished in Part 1.2: case 1, where only a single group is represented in each cluster, and case 2, where clusters contain observations from both groups. For case 1, half of the clusters were assigned to each of the two groups of observations being compared. For case 2, approximately half the observations in a cluster were assigned to each of the two groups (exactly half, if the cluster size was even, or an imbalance of 1, if the cluster size was odd). Cluster sizes and type allocations were then held fixed for each simulation.

Normally distributed data were generated according to the model y_ik = μ + βz_ik + b_k + ε_ik, and skewed data were generated according to the model y_ik = exp(μ + b_k + ε_ik) + βz_ik, where y_ik is the value of the response variable for unit i in cluster k, z_ik represents the group to which observation i in cluster k belongs (0 for group 1, 1 for group 2), b_k are independent and normally distributed with mean zero and variance σ_b² = 0.7, and ε_ik are independent and normally distributed with mean zero and variance σ_w² = 0.3. This corresponds to an intracluster correlation of 0.7. In these models, the parameter β corresponds to the group effect, such that β = 0 corresponds to no difference between the two groups of observations. Table 1 shows the remaining parameter choices for each simulation. By altering these parameters, we generated datasets from eight different models, which were examined using various statistical methods (see below).

Part 2.2: Methods of statistical analysis

Having generated clustered data under eight different models (Table 1), we will next demonstrate how effectively seven different established statistical methods performed in the analysis of these different simulated datasets. The statistical tests we examined are as follows:

• (1) A 2-sample t test, applied to the two groups of individual observations. In this test, clustering is not accounted for, since each observation is treated as independent.

• (2) A Wilcoxon rank-sum test, applied to the two groups of individual observations. Clustering is not accounted for, as above.

• (3) A t test, applied to the cluster means. For case 1 data, each cluster was reduced to a single mean and a two-sample t test was performed. For case 2 data, each cluster was reduced to a pair of means (one for each group) and a paired t test was performed. In these tests, clustering is effectively removed by reducing the data to the cluster means.

• (4) A Wilcoxon rank-sum test, applied to the cluster means (for case 1 data) or a Wilcoxon signed rank test, applied to the paired cluster means (for case 2 data). Clustering is effectively removed by reducing the data to the cluster means, as above.

• (5) LMM (Laird and Ware, 1982). This modeling approach incorporates the intracluster correlation effect, and thus accounts for data clustering without reducing the data to the cluster means.

• (6) GEE (Liang and Zeger, 1986), using an “exchangeable” working correlation structure, i.e., correctly assuming equal correlation between observations within a cluster. This modeling approach also accounts for data clustering.

• (7) Rank-sum test for clustered data (Datta and Satten, 2005). This is a modification of the Wilcoxon rank-sum test that accounts for the intracluster correlation.

LMM and t tests assume that the data are approximately normally distributed, whereas the other approaches do not require this assumption.

For case 2 data, we also considered a normal linear regression model (i.e., ANOVA, as described in Part 1.4, approach C) with a factor for cluster as well as the group effect. This is similar to LMM, except that cluster is treated as a fixed effect rather than a random effect. Since the results were very similar to LMM for the scenarios we considered, where clusters were approximately equally split between the two groups, we have not reported them separately. Note that this approach cannot be used for case 1 data, since there is no within-cluster comparison of the groups available and, hence, insufficient information to estimate both the group effect and a fixed effect for each cluster.

Part 2.3: Outcome of comparing statistical tests

A single simulated dataset is shown in Figure 1. This dataset consists of 200 observations in 20 clusters. The observations fall into two categories, group 1 and group 2, with only a single group represented in each cluster (i.e., case 1 data). In this simulated dataset, we have specified that there is no difference between group 1 and group 2 observations.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

A sample dataset generated from model 1. For illustrative purposes, a single dataset generated under model 1 is shown. On the left are shown the individual observations using different colors for the different clusters within each group. In this simulation, no difference exists between group 1 and group 2 data. However, the similarity of observations within a cluster, which induces intracluster correlation, is apparent here, as observations within each cluster lie close to each other. On the right are shown the data reduced to cluster means. The color of each mean represents the cluster to which it belongs.

We compared group 1 against group 2 observations using the methods described above (Table 2). We observed that ignoring the clustering (applying the t test or Wilcoxon test to the individual observations) would lead to rejection of the null hypothesis, since the p values are considerably <0.05. Thus, when clustering was ignored, both the t and Wilcoxon tests reported a significant difference between group 1 and group 2 data, where in fact no such difference actually exists. None of the other methods indicated that the null hypothesis should be rejected. Thus, although the estimated difference between groups was similar for all of the methods, the t and Wilcoxon tests in which clustering was ignored underestimated the variability in the data due to the positive correlation in the data, as shown by the narrower confidence intervals. These findings demonstrate the hazardous nature of ignoring data clustering, since the tests in which clustering was ignored would report a significant difference between groups 1 and 2 when no such difference actually exists.

We next generated 10,000 datasets from each model detailed in Table 1. Analyzing such a large number of datasets provides a more comprehensive view of how the tests perform. Results for case 1 data, where only a single group is represented in each cluster, are shown in Table 3, while Table 4 shows results for case 2, in which clusters contain approximately equal numbers of observations from each group. For data generated under models 1, 3, 5, and 7 there is no real difference between the groups being compared, so that a valid statistical test should reject the null hypothesis for ∼5% of the datasets.

Table 3 shows that for case 1 data, the t and Wilcoxon tests on individual observations (i.e., ignoring clustering) were far too liberal, rejecting the null hypothesis for >40% of the datasets under each null model. These two tests exhibited high power when there was a significant difference between the groups (models 2, 4, 6, and 8, in which β ≠ 0). However, this is no consolation, because for a real dataset we would not know whether the null hypothesis is true or not. Thus, the probability of recording a “false positive” result would be much higher in tests where clustering is simply ignored. By contrast, Table 4 shows that for case 2 data, the t and Wilcoxon tests on individual observations were too conservative, rejecting the null hypothesis <1% of the time. These results are consistent with the mathematical findings presented in the supplemental Fig. S1, available at www.jneurosci.org as supplemental material. As the strength of the intracluster correlation increases, the effects of ignoring clustering become more pronounced. This is clear when the type I error is plotted against the intracluster correlation for case 1 and case 2 data (supplemental Fig. S1, available at www.jneurosci.org as supplemental material).

We then considered the performance of statistical methods that do not ignore clustering: t and Wilcoxon tests on cluster means, LMM, GEE, and the Datta and Satten (2005) method (described below).

Part 2.4: Large datasets, normally distributed data

Datasets from models 1–4 consist of 20 clusters and a total of 200 observations. Models 1 and 2 produce normally distributed data, and we expect that methods designed for normal data should perform well here. Table 3 shows that for case 1 data the LMM and t test on means, which are designed for normally distributed data, perform well. The Datta and Satten method, although not specifically designed for normal data, also performed well. These three methods controlled the type I error at close to 5% and achieved power of ∼0.7. The Wilcoxon test on means had an acceptable type I error but was slightly less powerful.

Table 4 shows that for normally distributed case 2 data, in which clusters contain observations from both groups, LMM performed best, with type I error of 5% and power >0.7. The paired t and Wilcoxon tests performed similarly in terms of type I error but had slightly lower power. The Datta and Satten method had lower power in this case.

For both case 1 and case 2 data, GEE was slightly liberal when analyzing normally distributed data, with the type I error rate around 8% (Tables 3 and 4). The finding that hypothesis tests based on GEE tend to be too liberal when the number of clusters is relatively small has been noted previously (Feng et al., 1996; Mancl and DeRouen, 2001).

Part 2.5: Large datasets, non-normal data

For non-normal (skewed) data generated by models 3 and 4, the Datta and Satten method performed best, achieving the highest power while maintaining the correct rate of type I error (Tables 3 and 4).

Part 2.6: Smaller datasets

Datasets from models 5–8 are smaller, with 8 clusters and a total of 48 observations. For case 1 data, LMM performed the best for all model datasets, even for the non-normal datasets considered (Table 3). While the method of Datta and Satten also performed well in terms of power, it was somewhat more liberal with these smaller datasets than it was with the larger datasets of models 1–4.

For case 2 data, LMM clearly outperformed the other methods when analyzing normally distributed data (Table 4). It also performed well for skewed data, with a slightly conservative type I error but good power. The Datta and Satten method was also slightly conservative in this case but had relatively high power. The paired analyses on cluster means maintained type I error <0.05 but had slightly lower power. GEE was even more liberal for the smaller datasets than for the larger datasets analyzed under Models 1–4.