Jump to
Abstract
Background Surrogate endpoint trials test strategies more efficiently but are accompanied by uncertainty about the relationship between changes in surrogate markers and clinical outcomes.
Methods and Results We identified cardiovascular trials with primary surrogate endpoints published in the New England Journal of Medicine, Lancet, and JAMA: Journal of the American Medical Association from 1990 to 2011 and determined the trends in publication of surrogate endpoint trials and the success of the trials in meeting their primary endpoints. We tracked for publication of clinical outcome trials on the interventions tested in surrogate trials. We screened 3016 articles and identified 220 surrogate endpoint trials. From the total of 220 surrogate trials, 157 (71.4%) were positive for their primary endpoint. Only 59 (26.8%) surrogate trials had a subsequent clinical outcomes trial. Among these 59 trials, 24 outcomes trial results validated the positive surrogates, whereas 20 subsequent outcome trials were negative following positive results on a surrogate. We identified only 3 examples in which the surrogate trial was negative but a subsequent outcomes trial was conducted and showed benefit. Findings were consistent in a sample cohort of 383 screened articles inclusive of 37 surrogate endpoint trials from 6 other high‐impact journals.
Conclusions Although cardiovascular surrogate outcomes trials frequently show superiority of the tested intervention, they are infrequently followed by a prominent outcomes trial. When there was a high‐profile clinical outcomes study, nearly half of the positive surrogate trials were not validated. Cardiovascular surrogate outcome trials may be more appropriate for excluding benefit from the patient perspective than for identifying it.
Introduction
According to an Institute of Medicine report, a surrogate endpoint is intended to substitute for a clinical endpoint, and expected to predict clinical benefit based on epidemiologic, therapeutic, pathophysiologic, or other evidence.1 Health interventions are expected to impact clinical endpoints through pathways that include intermediary (surrogate) endpoints, and testing of these interventions can be done more quickly using smaller, shorter, and less expensive trials, when focused on surrogates.2, 3 This testing strategy has been particularly adopted in cardiovascular medicine, where many interventions required years to manifest their effects on clinical outcomes.4, 5
For example, dyslipidemia is known to be associated with increased risk of cardiovascular events. Investigators have tested the impacts of health interventions on dyslipidemia, hoping that an intervention that targets dyslipidemia in the short term would likewise lead to better cardiovascular outcomes in the long term. However, torcetrapib, a cholesteryl ester transfer protein inhibitor, improved lipid profile dramatically while clinical outcomes were not improved (and there was suggestion of clinical harm). Similarly, there have been several studies where interventions showed benefits on other surrogate endpoints, while the clinical outcomes were unchanged or worsened.6, 7, 8 Therefore, despite the potential advantages and efficiency, use of surrogate outcomes for testing strategies is accompanied by concern for lack of efficacy on clinical outcomes because of loose (or no) causal relationship between the surrogates and clinically important outcomes, or because of coexisting but unexpected consequences that health interventions may have on pathways other than those of the surrogate outcome.9, 10 However, the extent of this phenomenon has not been characterized.
Our objective was to perform a systematic review of cardiovascular trials published in the highest‐impact journals, characterizing the success of these trials in meeting their primary endpoints. We also examined subsequent publication of clinical outcome trials on interventions identified in our review, determining concordance between the surrogate outcome trials and subsequent clinical outcomes trials.
Methods
Data Source, Inclusion, and Exclusion Criteria
We searched MEDLINE with PubMed interface to screen all publications in the New England Journal of Medicine (NEJM), Lancet, and JAMA: Journal of the American Medical Association (JAMA) (January 1, 1990, to December 31, 2011) based on a search for cardiovascular trials using keywords and Medical Subject Heading terms (Table 1). Studies have shown that trials published in the highest‐impact journals have a higher methodological quality, larger sample size, and lower risk of bias.11 Such journals are more likely to publish important and potentially practice‐changing clinical trials.12, 13
We excluded noncardiovascular trials, safety trials, and manuscripts that reported secondary or post hoc analyses. We chose the cutoff date of December 31, 2011, for inclusion of surrogate outcome clinical trials. This decision was made to provide time for publication, in the 3 journals, of pertinent subsequent clinical outcomes trials that followed the included surrogate outcomes clinical trials.
Characterization of Surrogate and Clinical Endpoints and Positive or Negative Results
We first characterized all trials as clinical endpoint trials, or surrogate endpoint trials. Detailed definitions of surrogate endpoints have been provided elsewhere.1 In brief, study endpoints that could not be perceived and directly related to patients but were derived from tests with plausibly important medical information were considered as surrogate endpoints. Common examples included blood tests and various imaging test results. Clinical endpoint trials were those whose primary endpoint was a patient‐important and patient‐perceived outcome, such as mortality, myocardial infarction, stroke, or a composite of such variables. We determined the proportion of surrogate endpoint trials that had positive results (ie, were positive for and met their primary endpoint). Following identification of a surrogate endpoint trial, we searched for clinical endpoint trials on the same tested intervention in the 3 journals published until April 1, 2015. Among surrogate trials that had a subsequent clinical outcomes trial, we determined the concordance rate of the results in surrogate trials and clinical outcomes trials (ie, whether the positive or negative nature of results in a surrogate trial were replicated in the clinical outcome trial). We explored the findings according to the enrolled population (primary prevention, secondary prevention, or hybrid cohorts). We also divided the surrogate outcome trials into 3 subgroups of clinical biomarkers, imaging markers, and others4 (such as blood pressure).
Sample Replication in Another Cohort
To ascertain the robustness of the findings across surrogate endpoint cardiovascular trials published in other journals, we subsequently searched PubMed with the same search strategy, but this time for publications in 6 additional high‐impact journals (Circulation, Journal of the American College of Cardiology, European Heart Journal, JAMA Internal Medicine [formerly Archives of Internal Medicine], British Medical Journal, and Annals of Internal Medicine). Since such a search retrieved an extremely large sample (14 279 hits), we selected a 6‐month period in the middle point of our study (ie, from January 1, 2002, and July 1, 2002) to investigate the broader cohort of journals for publication of surrogate endpoint trials, as well as subsequent publication of clinical outcome trials on those scenarios in any of the aforementioned 6 journals or NEJM, Lancet, or JAMA.
Statistical Analyses
The study was designed by B.B., J.S.R., and H.M.K.. Three additional authors (N.P., Y.A., and I.L.) performed the primary screening and data extraction process. B.B. reviewed all the results for consistency. Disagreements and questions were resolved with J.S.R. and H.M.K. We reported qualitative variables with frequencies and percentages. We used the chi‐square test to report the differences between categorical variables. We used the Mantel–Haenszel test to report the trends. We used a linear regression model to report the R2 between two variables with a presumably linear relationship. Microsoft Excel (Redmond, WA) and Stata version 12.0 (StataCorp, College Station, TX) were used for data extraction and conducting the statistical analyses. This is a systematic review of the literature, and, hence, no request for institutional review board approval was required.
Results
Between January 1990 and December 2011, the number of articles published in the 3 journals declined, while the number of citations that included “trial*” in the PubMed search remained relatively stable (Table 2). In the study period, we manually screened 3016 articles through the systematic search and identified 220 surrogate endpoint trials. There was an increase in the annual number of surrogate endpoint trials from 1990 to 2007 (P<0.01 for trend) and a decline thereafter (Figure 1).
The number and proportion of cardiovascular trials with a primary surrogate endpoint from 1990 to 2011. Circles and blue lines represent the number of trials. Rectangles and green lines represent the proportion of trials with primary surrogate endpoints per 100 cardiovascular trials per year. The y axis represents the number of surrogate endpoint clinical trials or the proportion per 100 cardiovascular trials. Note that there was an increase in publication of surrogate endpoint trials from the 1990s to the mid‐2000s (P<0.01 for trend), and a subsequent decline started.
From the total of 220 surrogate trials, 157 (71.4%) were positive for their primary outcome. Fifty‐nine (26.8%) surrogate endpoint trials were followed by at least 1 clinical endpoint trial. Year of publication had a modest association with presence of subsequent outcome trials (R2=0.15), with older trials being slightly more frequently followed by a clinical outcomes trial. Among these 59 surrogate endpoint trials that had a subsequent clinical endpoint trial, in 24 cases the clinical endpoint trial results validated the positive surrogate trials, while in 20 the subsequent clinical endpoint trial was negative (Table 3).14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50 A negative surrogate endpoint trial was less likely to be followed by a positive outcome trial and we identified only 3 such examples (P=0.02, Figure 2).
Cardiovascular surrogate endpoint clinical trials published from 1990 to 2011 and subsequent results in clinical outcomes trials. Overall, there were 220 surrogate endpoint trials, of which 157 (71.3%) were positive for their primary endpoint. Overall, 59 of the surrogate endpoint clinical trials were followed by clinical outcome trials. See text for further details.
Among the 220 surrogate endpoint cardiovascular trials, 56 enrolled primary prevention populations, 138 had a secondary prevention population, and 26 had a hybrid cohort (ie, a mix of primary and secondary prevention patients). There was no difference based on the enrolled population in the proportion of studies that had subsequent clinical outcomes trials (P=0.51, Table 4).
From the total of 220 surrogate endpoint cardiovascular trials, 101 had an imaging endpoint, 42 had clinical biomarkers, and 77 had other surrogate endpoints. Trials with an imaging‐related primary endpoint were more frequently followed by a clinical outcome trial (37 of 101 versus 22 of 119; P=0.02).
In our robustness search of 6 additional journals (Circulation, Journal of the American College of Cardiology, European Heart Journal, JAMA Internal Medicine, British Medical Journal, and Annals of Internal Medicine) from January 1, 2002, to July 1, 2002, we screened 383 articles and identified 37 eligible surrogate endpoint trials. Most of these trials were small (median sample size: 71 patients) and were positive for their primary surrogate outcome (N=25, 67.5%). Of these 37 trials, the overwhelming majority (N=35, 94.5%) did not have a subsequent clinical outcomes trial.
Discussion
We found that while surrogate endpoint trials published in the highest‐impact journals frequently show superiority of the tested intervention, less than one third of them had a clinical outcomes trial of the intervention for the same purpose published in those same highest‐impact journals. Moreover, when there was a subsequent clinical outcomes trial, nearly half of them failed to validate the positive impact of the intervention on the surrogate marker. The results were fundamentally robust irrespective of enrolled population or type of the surrogate outcome. The findings were also similarly replicated (if not more pronounced) in a sample cohort from other high‐impact journals. Although surrogate markers are intended to ultimately predict benefits for patient‐important outcomes, our findings question this premise. The issue may be not that there are flaws in a surrogate endpoint, but that interventions have a multitude of effects beyond the surrogate, and it is difficult to judge the net result on outcomes based on the surrogate endpoint, even one thought to be central to the mechanism of disease or highly predictive of outcomes.
The suboptimal rate of outcomes trials that accompany a surrogate endpoint trial is concerning and should draw the attention of investigators and policymakers. The increase in publication of surrogate endpoint cardiovascular trails from 1990 until a few years ago is likely reflective of surging enthusiasm and dominance of surrogate endpoints among investigators and academicians.2 The decline in recent years, however, possibly reflective of lessons learned from unexpected results of surrogates on clinical endpoints, is encouraging.
Of the surrogate endpoint trials that were accompanied by a clinical outcomes trial, we noticed several positive surrogate endpoint trials that had a related negative clinical outcomes trials. We hypothesize that the disconnect between the surrogate endpoints and clinical outcomes is multifactorial. Some surrogate endpoints might merely be risk markers but not within the causal pathway, and therefore, intervening on them might have had little impact to improve clinical outcomes.9 Others may have been in the causal pathway but not targeted by the right intervention. Yet, some other surrogate endpoints might have been in the causal pathway but targeted by interventions that had coexisting off‐target effects.6, 51 Trials with positive imaging surrogate endpoints were more frequently followed by clinical outcomes trials. It could be hypothesized that structural changes usually need more time to reflect a change based on an intervention than blood biomarkers and could therefore better predict the ultimate impact of a health intervention. These findings warrant further investigation.
The choice regarding our study cohort is worthy of further discussion. We investigated surrogate endpoint cardiovascular trials published in the NEJM, Lancet, and JAMA in order to focus on those most likely to be the highest‐impact and highest‐quality studies. Although inclusion of all other surrogate endpoint trials for the study could have been ideal, achieving such a task is improbable for our group and many others. A search of merely 6 additional impactful journals, discussed above, retrieved more than 14 000 articles, and expansion to other journals would have made the cohort much larger. Our choice for searching subsequent outcome trials in the top 3 journals should also be discussed. Most often, high‐impact cardiovascular clinical endpoint randomized controlled trials are published in these 3 journals and it would be less frequent, if not rare, that an adequately powered well‐conducted cardiovascular clinical outcome trial gets published outside of these 3 journals. There could be a potential theoretical concern that the negative clinical outcome trials are less likely to get published in those journals. However, negative clinical outcome trials are commonly published in NEJM, Lancet, and JAMA.52 The results in a sample cohort of surrogate endpoint cardiovascular trials published in 6 other high‐impact journals further support the generalizability of our key findings. In our search of a sample of surrogate endpoint trials in other journals, the few associated clinical endpoint trials, all were identified from the 3 highest‐impact journals, with no clinical outcomes trials being found from the other 6 prestigious journals.
Study Strengths
Our study provides real‐world evidence about promises and limitations of surrogate endpoint trials in cardiovascular medicine. Infrequent follow‐up with a clinical outcome trial and poor concordance between positive surrogate endpoint trials and subsequent clinical outcome trials are concerning. We also observed that when a surrogate endpoint trial showed negative results, it was rare that a subsequent clinical endpoint trial proved the benefits of the intervention. We believe that our key findings would be helpful not only for investigators and funders but also for clinicians to recognize the benefits and concerns about clinical decisions based on surrogate endpoint trials. For investigators and funders, if the surrogate endpoint trial shows promise for the tested intervention, subsequent investigation with a clinical outcome trial would be the best next step. However, if the surrogate endpoint trial is well conducted and negative, the available finite resources could be shifted towards more promising health interventions. For practitioners and policymakers, including the Food and Drug Administration, it might be best to focus on clinically important endpoints, unless in scenarios where there is no interim way to obtain clinical outcomes from well‐conducted randomized trials (eg, young patients with familial hypercholesterolemia or those with rare conditions)—and even then, the label should express the limitations of the evidence. These issues are particularly important as the US Congress debates new legislation that directs the Food and Drug Administration to consider being more permissive in its approval process and to depend more heavily on surrogate endpoints.53
Study Limitations
Our study, however, had limitations other than the choice of study cohort discussed above. First, although in many cases the associated clinical outcome trials were negative, a smaller benefit on clinical endpoints could not be excluded. Second, although the surrogate endpoint trials and the identified associated outcome trials were very similar with regards to patient population and interventions, inevitably the subsequent trials may not have been a full replica of the initial surrogate endpoint trials (eg, using the same class of drug, but not necessarily the same agent or the same dose). However, using extremely strict criteria for identicalness of the surrogate endpoint trials and subsequent clinical endpoint trials would mean that an even lower proportion of the 220 surrogate endpoint trials were followed by clinical outcome trials than the 59 that we identified by reasonable clinical similarity. Third, although we believe that our study elucidates some fundamental advantages and challenges of surrogate endpoint trials, the focus was on cardiovascular trials. Therefore, extrapolation to other study fields requires further investigation. Preliminary results from other fields such as oncology concur with our findings.54, 55
Conclusions
Our findings raise concern about the certainty of assuming efficacy based on surrogate endpoints. Even if used for approval of therapies in urgent situations, postmarketing outcome trials are necessary. The good sensitivity of surrogate endpoint trials for detection of possible benefits, however, is encouraging. Based on our findings, cardiovascular surrogate endpoint trials may be more appropriate for excluding benefit from the patient perspective than for identifying it—and all surrogate endpoint trials should be interpreted in light of the possibility that they might not be validated in a clinical outcomes trial.
Disclosures
Drs Ross and Krumholz receive support through Yale University from Johnson and Johnson to develop methods of clinical trial data sharing and from Medtronic, Inc. and the Food and Drug Administration to develop methods for postmarket surveillance of medical devices; and from the Centers of Medicare & Medicaid Services to develop and maintain performance measures that are used for public reporting. Dr Ross receives support through Yale University from the Blue Cross Blue Shield Association to better understand medical technology evaluation, the Laura and John Arnold Foundation to support the Collaboration on Research Integrity and Transparency at Yale, and the Food and Drug Administration as part of the Centers for Excellence in Regulatory Science and Innovation program. Dr Krumholz reported that he chairs a cardiac scientific advisory board for UnitedHealth, is on the advisory board for Element Science, and is a participant/participant representative of the IBM Watson Health Life Sciences Board. Dr Krumholz is also the founder of Hugo, a personal health information platform.
References
This Issue
Jump to
Article Tools
Share this Article
- Two Decades of Cardiovascular Trials With Primary Surrogate Endpoints: 1990–2011
