Has NICE Got It Right? A Qualitative Case Study of Technology Appraisal No. 98
The report – published as a monograph with Springer Nature in 2007 – provides a comprehensive analysis of the robustness of the technology appraisal process and the methods for health technology assessments (HTAs) adopted by the National Institute for Health and Clinical Excellence (NICE) in England (now known as the "National Institute for Health and Care Excellence").
"While the focus of this monograph is the NICE ADHD analysis and appraisal process, the careful step-by-step critique might be used as a guide for future appraisal processes, not just for NICE, but for all health care analysts as well."Peter Jensen, Ruane Professor of Child Psychiatry, Columbia University. New York, NY (United States)
"The study at hand offers significant lessons for policy-makers, not only in England and Wales, but, given NICE’s international standing, in other settings as well."Panos Kanavos, Professor and Senior Lecturer in International Health Policy, London School of Economics, London (United Kingdom)
"[The study] is an excellent piece of work and represents the highest standing of scholarship. It reflects impressive and detailed insights into the assessment of benefits in the process of economic evaluation."Jeffrey Richardson, Professor and Founding Director, Centre for Health Economics, Monash University. Melbourne, Vic (Australia)
Background
NICE was established as a Special Health Authority for England and Wales on April 1, 1999 (as “National Institute for Clinical Excellence”). Since its inception, one of its key roles has been the production of Technology Appraisals (based on relevant evidence on clinical and cost effectiveness) and of Clinical Guidelines.
NICE has been internationally recognized for its “inclusive, consultative, and transparent” processes and its “objective, unbiased, and methodologically sound” appraisals (cf. presentation by C. Longson, 2001). From its inception, NICE claimed that its “advice ends the uncertainty [over the value of particular drugs and devices] and helps to standardize access to health care across the country” (NICE, 2006). The editor of the British Medical Journal, Richard Smith, even proclaimed “the triumph of NICE”, asserting “NICE may prove to be one of Britain’s greatest cultural exports, along with Shakespeare, Newtonian physics, the Beatles, Harry Potter, and the Teletubbies” (Smith, 2004). Indeed, clinicians developing European treatment guidelines for the prescription of long-acting medications for attention-deficit/hyperactivity disorder (ADHD) relied heavily on a systematic review by NICE as a starting point (cf. Banaschewski et al., 2006), thus providing an example for international spill-over effects of NICE Technology Appraisals.
The NICE Approach
The approach to Technology Appraisals chosen by NICE and reviewed in the study can be described as highly standardized and extensively manualized.
(a) The NICE Technology Appraisal process consisted of three (to four) phases with precisely defined timelines:
- Scoping,
- Assessment,
- Appraisal,
- Appeal (only if lodged by a “consultee”).
(b) Since April 2004, NICE assessment methods were laid down in a Reference Case, including
- systematic reviews as the source of evidence on outcomes,
- cost-effectiveness analysis as tool for economic evaluation,
- an NHS perspective on costs,
- probabilistic sensitivity analysis to address uncertainty,
- QALYs as a universal and comprehensive measure of health benefits,
- choice-based methods for health state valuation from a representative sample of the public,
- a fundamental equity position that each additional QALY should have equal value.
Some of the major upgrades introduced with Reference Case analysis in April 2004 were a change in the discounting of future costs and health benefits (no more differential discounting), the mandatory use of probabilistic sensitivity analyses, and explicit consideration of subgroup analysis.
The Case Analysis
A qualitative case study of NICE Technology Appraisal No. 98 (“Methylphenidate, atomoxetine, and dexamfetamine for ADHD”, March 2006) was done to explore the robustness of the NICE approach when addressing complex clinical decision problems.
(a) Process-related observations included:
- Predictability (or “adherence” to defined approach) was high, and specified timelines were either met or adapted sensibly to changes in the environment and communicated to stakeholders; opportunities for stakeholder participation were offered.
- Publicity (or “transparency”) was generally high through-out the process, with notable exceptions regarding (i) “commercial-in-confidence” data and (ii) the economic model itself.
- Relevance was less clear, given (i) that the scope of the assessment was substantially narrower than that of the clinical guideline devlopment process (which it should inform) and (ii) that the ethical proposition of QALY maximization as the primary objective of the NHS is empirically flawed and subject to normative controversy.
- NICE provisions for appeal appear more restrictive than those proposed by Daniels and Sabin (“Accountability for Reasonableness” framework).
(b) Assessment-related observations included:
- A remarkable dissociation between the clinical effectiveness and cost effectiveness reviews, owing to an exclusive focus on QALYs for economic modeling – confounded by the notable absence of any review of the extensive literature on clinical ADHD effect measures, and some apparent confusion in relation to their relevance.
- No distinction was made between clinical efficacy and effectiveness data, including a notable absence of any review of the extensive literature on treatment compliance.
- Highly selective (and occasionally enigmatic) use of clinical evidence, broadly neglecting clinical long-term studies.
- Inability of assessment to differentiate between treatments on grounds of effectiveness, due to “paucity of evidence” considered.
- Various anomalies of a technical nature (e.g., discount rates used violating NICE Reference Case guidance, idiosyncratic interpretation of study inclusion criteria for review).
“Clear” conclusions proposed by NICE assessment group differed from results produced by international HTA agencies (e.g., Scotland, Australia) and were not upheld by the NICE appraisal committee.
Suggested Underlying Problems
An analysis of the anomalies and inconsistencies found in the present case studied indicates several underlying problems as potential sources of the symptoms observed:
- Insufficient integration of clinical and economic perspectives.
- High level of standardization, requiring to make the clinical decision problem fit a predetermined solution strategy (e.g., exclusive focus on QALYs as a measure of health benefits), instead of adapting the problem-solving strategy (making the approach “rigid” rather than “rigorous”, as intended); this insufficient flexibility is proposed to extend to the resources (e.g., time, clinical input) made available for economic assessment.
- Absence of a system for effective quality assurance of technology assessments.
Collectively, these observations suggested that the current NICE approach to health technology assessments may not (yet) be robust enough to consistently solve complex clinical decision problems.
Suggested Implications for International Policy-Makers
- International cooperation of HTA agencies including transfer of assessments may also lead to an unintended transfer of inconsistencies of the original assessment.
- There is a need to consider the objectives of health care provision as well as the specific local institutional context.
- There appears to be a need for (more) effective integration of clinical and economic perspectives.
- There appears to be a need for an effective mechanism for quality assurance of technology assessments.
- There appears to be a need for (more) flexibility to adjust evaluation approaches (e.g., regarding type of economic evaluation and resources made available) so as to fit the problem at hand.