Impact of Information Lost to Follow-Up in RCTs
Impact of Information Lost to Follow-Up in RCTs
Up to a third of trials published in five top general medical journals and reporting significant results for binary primary outcomes that are patient important lose significance if one makes plausible assumptions about their loss to follow-up. Thirteen percent of trials did not report whether loss to follow-up occurred. In those that did report loss to follow-up, the median percentage of loss to follow-up was 6%; a fifth of the trials did not report on how missing data from participants were handled.
The plausibility of our assumptions about the outcomes of patients lost to follow-up varied. The first two assumptions (none of the participants lost to follow-up had the event and all participants lost to follow-up had the event) are commonly used but are implausible. The third assumption (worst case scenario) can be used to verify the robustness of a trial results but is extreme and generally unrealistic. In our sample, results of only 42% of randomised controlled trials would retain significance under a worst case scenario.
We designed the remaining assumptions (that is, the combinations of RILTFU/FU) to be more plausible on the basis of limited evidence that patients who are lost to follow-up tend to have worse outcomes. We would have tested other imputation methods (such as multiple imputations and regression models) if individual patient data had been available, which, as is typical in most study reports, it never was.
The most plausible assumption can depend on the question being examined by the trial. For example, an assumption that all patients lost to follow-up experienced an adverse event could be reasonable when patients are expected to comply with the trial protocol and follow-up. This might be the case in a trial used to evaluate a drug to prevent rejection after cardiac transplantation. Indeed, the investigators of such a trial implicitly made this assumption by including loss to follow-up as a component of a composite primary end point along with morbidity and mortality outcomes. Smoking cessation trials generally make the assumption that those lost to follow-up have failed to quit.
The most plausible assumption also depends on the reason for loss to follow-up. For example, participants who were lost to follow-up because they “moved” are likely to have better outcomes than those who were excluded because of “failure to improve.” Higgins et al proposed choosing the assumption based on the reason for loss to follow-up and, if necessary, using different assumptions for different groups of loss to follow-up in the same trial.
The most appropriate assumption could also be determined by using empirical data from studies assessing the outcomes of patients lost to follow-up from related trials. Investigators could also use data from population based studies. For example, a systematic review of studies tracing the outcomes of patients lost to follow-up from antiretroviral treatment programmes found that mortality was inversely associated with the rate of loss to follow up. Caution and judgment are needed when these observations are applied to randomised controlled trials. Baseline characteristics of participants lost to follow-up is another factor to consider. When these characteristics suggest poorer prognosis relative to participants followed up (such as older people and higher percentages of comorbidities), participants lost to follow-up would probably have poorer outcomes.
Loss to follow-up with no bias reduces power because the effective sample size is reduced. By imputing some events in those lost to follow-up, the total number of events rises and some power is regained, particularly if one assumes the same risk in those lost to follow-up and their respective treatment groups (such as when a value of 1 for the RILTFU/FU is used in both groups). Given, however, that the investigators are making up the data, one could argue that the apparent increased precision is misleading. Indeed, Higgins et al have highlighted the need to take into account the uncertainty about the imputed data. Taking into account the uncertainty would result in wider confidence intervals; if we had done this, results of more trials would have lost significance. Our results are therefore conservative estimates of the percentage of trials losing significance.
The association we found between the length of follow-up and the extent of loss to follow-up is not surprising as longer follow-up will inevitably result in difficulties retaining all randomised patients. The association between inadequate concealment of allocation and the extent of loss to follow-up could represent less methodological rigor in both aspects of design and conduct. One might have expected that the extent of loss to follow-up would be associated with the type of outcome (that is, more loss to follow-up with non-fatal compared with fatal outcomes); we did not find this to be the case.
Our study has several strengths. Our a priori publication of the study protocol made our objectives and methods transparent and subjected them to peer review. We used transparent and systematic methods to search for and select eligible studies, select the primary outcomes, and abstract data. We also ensured rigorous data abstraction by using detailed written instructions, conducting formal calibration exercises, conducting duplicate abstraction, measuring agreement, and implementing a consensus approach to resolve disagreement. We contacted authors to verify our abstracted data and achieved a 46% response rate and most data were verified as accurate. While other authors have proposed many of these assumptions, we think that this is the first study to propose and test approaches based on RILTFU/FU.
One limitation of our study is its generalisability because of its focus on reports published in five top general medical journals rather than a wider range of journals. Randomised controlled trials published in lower profile journals, however, might report smaller effects than those published in top medical journals and might be of lower methodological quality and thus have higher rates of loss to follow-up. Lower effect estimates and higher rates of loss to follow-up would inflate the potential impact of loss to follow-up. Thus, our findings are more likely an underestimate of the impact of loss to follow-up in a wider range of randomised controlled trials. Our results do not apply to continuous data, which present specific challenges that need to be dealt with separately.
We focused on reports of trials with significant effect estimates because these studies are more likely to influence clinical practice. Also, unless event rates in those lost to follow-up are greater relative to those followed up in control groups rather than intervention groups, ignoring missing data will not result in misleading inferences. The reasonably narrow time range of the included studies (2005-07) was determined to a large extent by our sampling method; we first sampled all eligible trials published in 2007 and included trials from preceding years until we reached our target sample size. If trials in these years are idiosyncratic, or if strategies for avoiding loss to follow-up have improved in the years since 2007, our results could be unrepresentative. Neither of these possibilities, however, is likely.
Finally, we used a frequentist statistical approach to explore the impact of loss to follow-up on effect estimates. An alternative would have been a Bayesian approach.
This study has important implications for trialists, editors of medical journals, systematic reviewers, and users of medical literature. Investigators should of course aim to reduce the extent of loss to follow-up in the design and implementation of their trials. They should also be transparent and detailed in reporting loss to follow-up (such as, extent, timing, reasons, and baseline characteristics of those lost to follow-up, all by study arm) and describe the potential implications for their primary analysis. Specifically, conducting sensitivity analyses with reasonable assumptions about loss to follow-up is necessary to test the robustness of their results. The assumptions we have made could be a reasonable standard from which trialists could deviate if they have compelling reasons to do so. Our study was limited to relatively simple assumptions that do not require individual participant data. If this level of data is available then investigators should consider more sophisticated statistical methods such as multiple imputation. Editors of medical journals have the opportunity to improve the quality of the medical literature by enforcing the use of the CONSORT statement, particularly as it relates to reporting the patient flow diagram and the number of patients lost to follow-up, the reasons for loss to follow-up, and the number of patients analysed.
Systematic reviewers should consider all available information about the extent of loss to follow-up and the assumptions used in the primary analysis of original reports. They should also routinely conduct sensitivity analyses with reasonable assumptions about the outcomes of those lost to follow-up to test the robustness of the results of their meta-analyses.
Users of published medical literature should be aware of the potential vulnerability of apparently positive results to loss to follow-up. Important factors that might be associated with higher vulnerability include a small magnitude of effect, a high number of participants lost to follow-up (particularly when compared with the number of events), differential loss to follow-up in study arms (in terms of numbers and reasons), poorer baseline prognosis of participants lost to follow-up, reasons for loss to follow-up likely to be associated with poorer prognosis, and loss of statistical or clinical significance, or both, when reasonable assumptions about participants lost to follow-up are applied.
Future research should include collection of empirical evidence to define the most reasonable assumptions about the outcomes of patients lost to follow-up. Assumptions will probably vary with the population involved, the nature of the intervention, and the outcome under consideration. Similar work is also needed to inform the impact of loss to follow-up for continuous outcomes. For now, authors of individual randomised controlled trials and of systematic reviews should test their results against various reasonable assumptions. Only when the results are robust to all reasonable assumptions can inferences from those results be viewed as secure.
Discussion
Summary of Findings
Up to a third of trials published in five top general medical journals and reporting significant results for binary primary outcomes that are patient important lose significance if one makes plausible assumptions about their loss to follow-up. Thirteen percent of trials did not report whether loss to follow-up occurred. In those that did report loss to follow-up, the median percentage of loss to follow-up was 6%; a fifth of the trials did not report on how missing data from participants were handled.
Interpretation of Findings
The plausibility of our assumptions about the outcomes of patients lost to follow-up varied. The first two assumptions (none of the participants lost to follow-up had the event and all participants lost to follow-up had the event) are commonly used but are implausible. The third assumption (worst case scenario) can be used to verify the robustness of a trial results but is extreme and generally unrealistic. In our sample, results of only 42% of randomised controlled trials would retain significance under a worst case scenario.
We designed the remaining assumptions (that is, the combinations of RILTFU/FU) to be more plausible on the basis of limited evidence that patients who are lost to follow-up tend to have worse outcomes. We would have tested other imputation methods (such as multiple imputations and regression models) if individual patient data had been available, which, as is typical in most study reports, it never was.
The most plausible assumption can depend on the question being examined by the trial. For example, an assumption that all patients lost to follow-up experienced an adverse event could be reasonable when patients are expected to comply with the trial protocol and follow-up. This might be the case in a trial used to evaluate a drug to prevent rejection after cardiac transplantation. Indeed, the investigators of such a trial implicitly made this assumption by including loss to follow-up as a component of a composite primary end point along with morbidity and mortality outcomes. Smoking cessation trials generally make the assumption that those lost to follow-up have failed to quit.
The most plausible assumption also depends on the reason for loss to follow-up. For example, participants who were lost to follow-up because they “moved” are likely to have better outcomes than those who were excluded because of “failure to improve.” Higgins et al proposed choosing the assumption based on the reason for loss to follow-up and, if necessary, using different assumptions for different groups of loss to follow-up in the same trial.
The most appropriate assumption could also be determined by using empirical data from studies assessing the outcomes of patients lost to follow-up from related trials. Investigators could also use data from population based studies. For example, a systematic review of studies tracing the outcomes of patients lost to follow-up from antiretroviral treatment programmes found that mortality was inversely associated with the rate of loss to follow up. Caution and judgment are needed when these observations are applied to randomised controlled trials. Baseline characteristics of participants lost to follow-up is another factor to consider. When these characteristics suggest poorer prognosis relative to participants followed up (such as older people and higher percentages of comorbidities), participants lost to follow-up would probably have poorer outcomes.
Loss to follow-up with no bias reduces power because the effective sample size is reduced. By imputing some events in those lost to follow-up, the total number of events rises and some power is regained, particularly if one assumes the same risk in those lost to follow-up and their respective treatment groups (such as when a value of 1 for the RILTFU/FU is used in both groups). Given, however, that the investigators are making up the data, one could argue that the apparent increased precision is misleading. Indeed, Higgins et al have highlighted the need to take into account the uncertainty about the imputed data. Taking into account the uncertainty would result in wider confidence intervals; if we had done this, results of more trials would have lost significance. Our results are therefore conservative estimates of the percentage of trials losing significance.
The association we found between the length of follow-up and the extent of loss to follow-up is not surprising as longer follow-up will inevitably result in difficulties retaining all randomised patients. The association between inadequate concealment of allocation and the extent of loss to follow-up could represent less methodological rigor in both aspects of design and conduct. One might have expected that the extent of loss to follow-up would be associated with the type of outcome (that is, more loss to follow-up with non-fatal compared with fatal outcomes); we did not find this to be the case.
Strengths and Limitations of Study
Our study has several strengths. Our a priori publication of the study protocol made our objectives and methods transparent and subjected them to peer review. We used transparent and systematic methods to search for and select eligible studies, select the primary outcomes, and abstract data. We also ensured rigorous data abstraction by using detailed written instructions, conducting formal calibration exercises, conducting duplicate abstraction, measuring agreement, and implementing a consensus approach to resolve disagreement. We contacted authors to verify our abstracted data and achieved a 46% response rate and most data were verified as accurate. While other authors have proposed many of these assumptions, we think that this is the first study to propose and test approaches based on RILTFU/FU.
One limitation of our study is its generalisability because of its focus on reports published in five top general medical journals rather than a wider range of journals. Randomised controlled trials published in lower profile journals, however, might report smaller effects than those published in top medical journals and might be of lower methodological quality and thus have higher rates of loss to follow-up. Lower effect estimates and higher rates of loss to follow-up would inflate the potential impact of loss to follow-up. Thus, our findings are more likely an underestimate of the impact of loss to follow-up in a wider range of randomised controlled trials. Our results do not apply to continuous data, which present specific challenges that need to be dealt with separately.
We focused on reports of trials with significant effect estimates because these studies are more likely to influence clinical practice. Also, unless event rates in those lost to follow-up are greater relative to those followed up in control groups rather than intervention groups, ignoring missing data will not result in misleading inferences. The reasonably narrow time range of the included studies (2005-07) was determined to a large extent by our sampling method; we first sampled all eligible trials published in 2007 and included trials from preceding years until we reached our target sample size. If trials in these years are idiosyncratic, or if strategies for avoiding loss to follow-up have improved in the years since 2007, our results could be unrepresentative. Neither of these possibilities, however, is likely.
Finally, we used a frequentist statistical approach to explore the impact of loss to follow-up on effect estimates. An alternative would have been a Bayesian approach.
Implications of Findings
This study has important implications for trialists, editors of medical journals, systematic reviewers, and users of medical literature. Investigators should of course aim to reduce the extent of loss to follow-up in the design and implementation of their trials. They should also be transparent and detailed in reporting loss to follow-up (such as, extent, timing, reasons, and baseline characteristics of those lost to follow-up, all by study arm) and describe the potential implications for their primary analysis. Specifically, conducting sensitivity analyses with reasonable assumptions about loss to follow-up is necessary to test the robustness of their results. The assumptions we have made could be a reasonable standard from which trialists could deviate if they have compelling reasons to do so. Our study was limited to relatively simple assumptions that do not require individual participant data. If this level of data is available then investigators should consider more sophisticated statistical methods such as multiple imputation. Editors of medical journals have the opportunity to improve the quality of the medical literature by enforcing the use of the CONSORT statement, particularly as it relates to reporting the patient flow diagram and the number of patients lost to follow-up, the reasons for loss to follow-up, and the number of patients analysed.
Systematic reviewers should consider all available information about the extent of loss to follow-up and the assumptions used in the primary analysis of original reports. They should also routinely conduct sensitivity analyses with reasonable assumptions about the outcomes of those lost to follow-up to test the robustness of the results of their meta-analyses.
Users of published medical literature should be aware of the potential vulnerability of apparently positive results to loss to follow-up. Important factors that might be associated with higher vulnerability include a small magnitude of effect, a high number of participants lost to follow-up (particularly when compared with the number of events), differential loss to follow-up in study arms (in terms of numbers and reasons), poorer baseline prognosis of participants lost to follow-up, reasons for loss to follow-up likely to be associated with poorer prognosis, and loss of statistical or clinical significance, or both, when reasonable assumptions about participants lost to follow-up are applied.
Future research should include collection of empirical evidence to define the most reasonable assumptions about the outcomes of patients lost to follow-up. Assumptions will probably vary with the population involved, the nature of the intervention, and the outcome under consideration. Similar work is also needed to inform the impact of loss to follow-up for continuous outcomes. For now, authors of individual randomised controlled trials and of systematic reviews should test their results against various reasonable assumptions. Only when the results are robust to all reasonable assumptions can inferences from those results be viewed as secure.
Source...