Version 5/20/05 Answer in italics.
1. Read the Patients and Methods of the article. Is this study experimental or nonexperimental? Explain your reasoning.
The study is observational (non-experimental) because exposure was not assigned as part of the study protocol. Patients received drugs based on availability, cost, or perhaps clinical considerations. The investigators did not intervene.
2. Suppose you could redesign the study . . . determine how many people to study . . . [handout distributed S05]
|
|
Detectable risk ratio (risk ratio worth detecting) |
|||
|
Power |
1.5 |
2.0 |
4.0 |
6.0 |
|
.80 |
1372 | 398 | 64 | 28 |
|
.90 |
1836 | 532 | 84 | 34 |
|
.95 |
2270 | 658 | 104 | 42 |
a) How does lowering the expected elevation in risk effect the required sample size?
This will increase the required sample size.
b) How will increasing the desired power . . .
This, too, will increase the required sample size
Next, randomize the treatment . . .
Assignments will be unique due to chance.
c) Randomization of the treatment works by tending balancing extraneous factors (potential confounders) in the groups. With a small study, it is still possible to have unbalanced groups. With large studies, extraneous factors will tend to balance (law of large numbers).
3. What is the primary benefit of randomization?
Randomization balances extraneous factors ("confounders") in the groups. When the confounders are equally distributed in the exposure groups, they can no longer confound results.
4. The study is a record-based cohort study (Jolson, 1992, p. 501). Suppose we had conducted this study as an ecological study by comparing cerebellar toxicity rates in hospitals using the generic drug to cerebellar toxicity rates in hospitals using the innovator drug (i.e., the unit of observation would have been the hospital as opposed to the individual.) What difficulties might be encountered in such an ecological comparison? What difficulties might be encountered in such an ecological comparison?
The ecological design would be more likely to be confounded by factors associated with each hospital. For example, patients at hospitals using the generic drug could conceivably be older or sicker or less careful dosed than patients at the other hospital. This could result in an ecological fallacy (EKS, pp. 195 - 196).
5. The investigators found a higher risk of cerebellar toxicity with the generic drug than with the innovator drug (44% vs. 9%). Results were based on 11 cases in the generic group and 3 cases in the innovator-drug group. Randomly, if several fewer cases had occurred in the generic group and handful of additional cases had occurred in the innovator-drug group, the measured effects would have been altered. How did the investigators deal with the potential of random error in their study?
The investigators used standard inferential techniques to address random error (i.e., confidence intervals and p values). Specific techniques included one-way ANOVA , chi-square statistics (and Fisher's tests when needed), and 95% confidence intervals for risk ratios.
6. The investigators were concerned with confounding as an explanation for the observed association. Confounding derives from inherent differences in risk that would exist even if the exposure had been absent. It can also be viewed as a problem of non-comparability of groups at baseline. To help address the potential for confounding, the investigators compared group characteristics (e.g., see Table 1 on p. 502). What potential confounding variables were address? Did the investigators find group differences that might confound results? Discuss.
The following potential confounders were addressed: age, sex, type of leukemia, stage of disease, creatinine clearance (as a measure of kidney function), liver function, total dose. Table 1 (p. 502) shows the groups were pretty comparable with respect to these factors. .
7. The authors discuss possible misclassification due to heightened physician awareness (p. 504). They describe this as a type of diagnostic suspicion bias. (Diagnostic suspicion bias is a type of information bias that occurs when the exposed group undergoes greater diagnostic scrutiny for the outcome than the non-exposed group.) Could this type of bias explain the higher rate in the generic-drug exposed group? Explain your reasoning.
Yes, the greater suspicion in the exposed cohort would uncover a greater percentage of the cases in the exposed group and perhaps some "false positives". In contrast, some of the cases in the non-exposed cohort might go undetected. This would make it seem as if the exposure caused a greater increase in risk than it did in fact. (Epidemiologists would refer to this as a bias away from the null.)
8. In addition to the problem of confounding, the authors discuss problems of selection (p. 504, second paragraph in the second column). The article states "it is unknown at this time whether this cluster of cases represented national experience with the product or a chance event." This type of selection has been compared to shooting the broad-side of a barn and drawing the bull�s-eye around the bullet hole after the fact. How does this alter your interpretation of the p values from the study?
The p values for an elevated risk were all significant, no matter how you cut the data. However, we must consider what this means in the context of the study. Suppose we are dealing with a p value of .01. This means that the observed difference would occur about 1% of the time even if the null hypothesis were true. Suppose there are a 100 hospitals using the generic product. How do we know that we haven't just selected that 1 in 100 hospitals in which the rate of adverse events is much higher with the generic product? Answer: we don't because we "shot the arrow and then drew the bulls-eye" after the fact. (This problem is common to the investigation of any cluster -- clusters happen randomly and we can't determine whether a given cluster is random or not when studied after the fact.)
9. The authors state "The biological basis of our finding of increased toxicity of a drug associated with a specific manufacturer is unknown "(p. 505). In other words, the authors were uncertain whether the generic drug caused increases in toxicity. Do you think a basis for causality can be met without a firm biological basis?
This is a rhetorical question, but personally, I believe the causal argument cannot be made in this instance without biological support. There are two reasons I believe this: 1) The logical fallacy of post hoc ergo propter hoc (also known as The Problem of Induction) and 2) the post hoc nature of studying this cluster, as addressed in #8, above.
10. Based on this study, would you remove the generic product from the market? Explain your reasoning:
The decision to remove the generic drug from the market was a difficult one based on many factors (scientific, public health, financial, legal). The decision was to pull the drug from the market. The generic drug maker went out of business.