Conducting and reporting indirect treatment comparisons: a critical appraisal of oncology technology appraisals

Langham J, Langham S. Conducting and reporting indirect treatment comparisons: a critical appraisal of oncology technology appraisals. Value Health 2018;21(Suppl 3):S68

Abstract

Objectives: The National Institute for Health and Care Excellence (NICE) Guide to the methods of technology appraisal, published in 2013, outlines the principles for conducting and reporting indirect treatment comparisons (ITCs). We conducted a review of Single Technology Appraisal (STA) submissions in oncology published in 2018 to assess adherence to these guidelines and report the most common concerns and mistakes identified by Evidence Research Groups (ERGs).

Methods: We carried out a qualitative content analysis of ERG reports for oncology STA submissions published in the first five months of 2018 that included an ITC. Data relating to the conduct and reporting of the systematic review and data synthesis were extracted based on the NICE guide to technology appraisal and criteria outlined in the critical appraisal checklist reported in the NICE Technical Support Document on Evidence Synthesis (NICE Decision Support Unit).

Results: Seven STAs reporting ITCs were identified. ERG critique of only three submissions described methods and reporting as adequate. The most commonly reported concerns included: lack of transparency, particularly around the search strategy, which prohibited replication; lack of rationale or poor application of inclusion/exclusion criteria; lack of, or poor assessment of risk of bias; lack of assessment of ITC assumptions, including heterogeneity; and uncertainty regarding the application of best practice, particularly in relation to statistical methods.

Conclusions: Of the oncology STAs reporting ITCs published in 2018, over half were criticised by the ERG for lack of transparency and non-adherence to methods of best practice. Whilst uncertainty is expected if there are insufficient relevant data available, unnecessary errors in conducting and reporting ITCs will lead to a lack of confidence in the validity of results. Better adherence to NICE guidelines is needed to ensure decision makers are able to formulate treatment recommendations from ITC results.