Reviews in Health Care 2014; 5(3): 109-115

http://dx.doi.org/10.7175/rhc.v5i3.893

Public health

Systematic review

Guidelines and evidence based medicine. The importance of stakeholder involvement

Francesco Massoni 1, Pasquale Ricci 1, Serafino Ricci 1

1 Department of Anatomical Sciences, histological, legal medicine and locomotor apparatus - University "Sapienza" of Rome

Abstract

With the depenalization of professional conduct of the physician in case of adherence to the guidelines proposed by L. 189/2012 the discussion on the quality of the available guidelines in the literature has led the authors to a review with particular regard to stakeholder involvement in the elaboration process. Evident critical points arise in the accessible studies that have used as an assessment tool a scientifically validated questionnaire (Appraisal of Guidelines for Research and Evaluation – AGREE). As a result the solution of the legislature, although useful to contain the phenomenon of defensive medicine, may be useless and ineffective in criminal cases where fundamental and absolute rights, as the right to life and health of patients, are involved.

Keywords

 Guidelines, Stakeholders; Professional responsibility

Corresponding author

Prof. Serafino Ricci

Viale Regina Elena, 336

00161 Roma

Email: serafino.ricci@uniroma1.it

Disclosure

No potential conflict of interest relevant to this article was reported.

There were no external funding sources for this study.

Introduction

After the L. 189/2012 in Italy the debate on the topic of professional responsibility has increased and the attention of health policy has once again raised the issue of the definition of serious fault.

In order to put a stop to the phenomenon of defensive medicine and considerable increases in health care spending, the legislature depenalized the offense in the case of adherence to guidelines, without any reference to the quality of the recommendations proposed as best scientific evidence.

In particular, the stakeholder involvement is one of the most important aspects in the process of formulation of recommendations that could go also to influence their operational applicability.

And, therefore, serious doubts may arise in the evaluation of professional conduct also in case of adherence.

The authors propose a review of the literature on previous experience of assessing the quality of guidelines through the use of a standardized and validated scientifically tool.

Material and method

The used instrument in the selected articles is Appraisal of Guidelines for Research and Evaluation (AGREE), in both versions I and II, developed by the AGREE Next Steps Consortium and available on www.agreetrust.org.

This instrument consists of 23 items grouped into six areas: objectives and scope, stakeholder involvement, methodological rigor, clarity, applicability and editorial independence.

The items related to the domain “stakeholder involvement” are shown in Table I with clear removal of a single question in the new version of the AGREE II.

Each item is rated on a seven-point Likert scale from “strongly disagree” to “strongly agree” (1-7 respectively) [1].

The study predicts that the guidelines on a specific subject are submitted to the evaluation of more experts chosen on the basis of the interests in the management of type of pathology and patient. For each of the six dimensions of the instrument a score is calculated by adding together all the scores of each evaluator for the items that compose the area. The obtained score is standardized as a percentage of the maximum possible score for that area applying the following form:

OS = Obtained Score; MinPS = Minimum Possible score; MaxPS = Maximum Possible score

In research of articles on Medline were used title’s keywords “guidelines” and “AGREE”. Forty-three results have been obtained, but removing the studies that did not provide for the use of the AGREE instrument or comments / letters to the editor or those that lacked one of the parameters considered (or just average, or median, or just only range) or was not possible to derive it, there are only eleven.

Results

The articles selected are shown in Table II.

In the study of Lytras et al. [2] six expert evaluators examined seven guidelines on occupational asthma choices on the Medline database: two guidelines remained much below the 50%, EAACI [3] 39% and EAACI PEF [4] 36%.

In the systematic review [5] of published guidelines on the treatment of acne vulgaris from July 2002 to July 2012 in English, French, German and Spanish, three expert evaluators have worked on six selected articles and in this case 3/6 have reported values <50%: Strauss et al. [6] (27,7%), AFSSAPS [7] (31,4%) and Orozco et al. [8] (14,8%).

Sabharwal et al. [9] have studied the quality of cardiac clinical practice guidelines published in seven cardiac cardiovascular journals between January 2001 and May 2011 (one hundred and one!), and, examined by two researchers, the minimum value achieved 39.9%.

In the analysis of thirteen guidelines in English on genetic syndromes that confer susceptibility to breast cancer [10], selected until October 2010 and reviewed by three reviewers, 6/13 <50% with the recommendations of the University of Michigan even 8.3% [11].

In the study by Polus et al. [12] on the quality of four WHO guidelines on maternal and child health published between 2007 and 2011 can be appreciated a chronological evolution from 32% of the P-PPH (2007) [13] to 86% of PE/E (2011) [14].

In the work of Don-Wauchope et al. [15] the AGREE II was applied by two evaluators on eleven guidelines for laboratory medicine available until December 2011. Nearly half (5/11) less than 50%, with the lowest result 11% [16].

In the study by Van Den Berg et al. [17] on the methodological quality of twelve international clinical guidelines for the management of malnutrition in adult cancer patients and reviewed by three reviewers, 8/12 ≤ 50% with Bozzetti et al. [18] and Braga et al. [19] 17%.

Burda et al. [20] with five independent evaluators have reviewed eleven clinical practice guidelines on the frequency of screening mammography in asymptomatic women (40-49 years) and published from 2005 to 2010, and 9/11 <50% with Michigan Quality Improvement Consortium 0% [21] and Society of Breast Imaging 8% [22].

In the study by Lo Vecchio et al. [23] on nine guidelines in English about the management of acute gastroenteritis in children examined from six auditors 7/9 <50%, with 14% of Sandhu et al. [24].

The evaluation of the chapters of ACOEM guidelines [25] on the management of low back pain [26] and chronic pain [27] score was not different from 6.25%.

And finally, only in the study by Hulshof and Hoenen [28] on six guidelines of Netherlands Society of Occupational Medicine examined by twenty experts 6/6> 50%.

Discussion

Stakeholders are all subjects who have an interest in the purpose / objective of the recommendations, such as health professionals, patients, public and private research funding, managers, entrepreneurs and producers.

The involvement of these subjects is justified by many reasons, in particular for transparency, but in some cases a stakeholder engagement inclusive, fair and appropriate can be difficult [29].

There have been experiences that have analyzed the stakeholder engagement only in the development and evaluation of the recommendations, neglecting the next and operational phase, that is sharing in the decision making process [30,31], maybe because they are not understood [32,33] or because the heterogeneity of “users” and their degree of involvement in decision making [34].

However it has been shown that the adoption of shared decision making and the involvement in the implementation process of the disclosure of scientific evidence [35] has had important and significant impact in efficiency [36], as was the case for infection control [37].

The involvement of a part / interest can lead, for example, to a simplification of complex innovations [32]. And also, like the involvement of the management can help to mobilize resources, so that patients, defined as “temporary members” of health care organizations, contributes to faster adoption of the recommendation [38,39].

Health policies, both U.S.A. than UK, explicitly promote stakeholder engagement considering a means to achieve the aim of reducing costs [40] and improving quality, increasing satisfaction [41]. These policies define user as a exacting subject and buyer of a service, with the responsibility to contribute to the common good through active participation [42].

The reasons for failure in stakeholder engagement, as can be appreciated in most of the articles examined in this paper must be sought in the scarce organization and competence, financial difficulties and time [43].

Resulting defective in this aspect, it is also conditioned the implementation of the recommendation and, therefore, also the subsequent use in the medico legal evaluation of professional responsibility where the respect for the prevalent right to the life or health of single patient is more important than health care cost containment.

Conclusion

The legislature, decriminalizing the professional conduct only on the basis of the adherence to guidelines, neglecting the consideration of the quality of the guidelines currently available in the literature, has strongly influenced the effectiveness of this instrument of health policy.

References

1. Brouwers MC, Kho ME, Browman GP, et al.; AGREE Next Steps Consortium. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ 2010; 182: E839-42; http://dx.doi.org/10.1503/cmaj.090449

2. Lytras T, Bonovas S, Chronis C, et al. Occupational Asthma guidelines: a systematic quality appraisal using the AGREE II instrument. Occup Environ Med 2014; 71: 81-6; http://dx.doi.org/10.1136/oemed-2013-101656

3. Guidelines for the diagnosis of occupational asthma. Subcommittee on ‘Occupational Allergy’ of the European Academy of Allergology and Clinical Immunology. Clin Exp Allergy 1992; 22: 103-8; http://dx.doi.org/10.1111/j.1365-2222.1992.tb00121.x

4. Moscato G, Godnic-Cvar J, Maestrelli P, et al. Statement on self-monitoring of peak expiratory flow in the investigation of occupational asthma. Subcommittee on Occupational Allergy of the European Academy of Allergology and Clinical Immunology. Allergy 1995; 50: 711-7; http://dx.doi.org/10.1111/j.1398-9995.1995.tb01211.x

5. Sanclemente G, Acosta JL, Tamayo ME, et al. Clinical practice guidelines for treatment of acne vulgaris: a critical appraisal using the AGREE II instrument. Arch Dermatol Res 2014; 306: 269-77; http://dx.doi.org/10.1007/s00403-013-1394-x

6. Garcia-Gutierrez JF, Bravo-toledo R. Guias de practica clinica en internet. Aten Primaria 2001; 28: 74-9; http://dx.doi.org/10.1016/S0212-6567(01)78899-3

7. GUIPCAR Group. Clinical Practice Guideline for the Management of Rheumatoid Arthritis in Spain. Spanish Society of Rheumatology. Madrid, 2001: 301

8. Guyatt GH, Oxman AD, Kunz R, et al. What is “quality of evidence” and why is it important to clinicians? BMJ 2008; 336: 995-8; http://dx.doi.org/10.1136/bmj.39490.551019.BE

9. Sabharwal S, Patel V, Nijjer SS, et al. Guidelines in cardiac clinical practice: evaluation of their methodological quality using the AGREE II instrument. J R Soc Med 2013; 106: 315-22; http://dx.doi.org/10.1177/0141076813486261

10. Simone B, De Feo E, Nicolotti N, et al. Methodological quality of English-language genetic guidelines on hereditary breast-cancer screening and management: an evaluation using the AGREE instrument. BMC Med 2012; 10: 143; http://dx.doi.org/10.1186/1741-7015-10-143

11. University of Michigan Health System: Guidelines for clinical care. Adult preventive health care: cancer screening. Available at http://www.med.umich.edu/1info/fhp/practiceguides/adult.cancer/cancergline.pdf

12. Polus S, Lerberg P, Vogel J, et al. Appraisal of WHO guidelines in maternal health using the AGREE II assessment tool. PLoS One 2012; 7: e38891; http://dx.doi.org/10.1371/journal.pone.0038891

13. World Health Organization. WHO recommendations for the prevention of postpartum haemorrhage. Geneva: World Health Organization, 2007

14. World Health Organization WHO recommendations for prevention and treatment of pre-eclampsia and eclampsia. Geneva: World Health Organization, 2011

15. Don-Wauchope AC, Sievenpiper JL, Hill SA, et al. Applicability of the AGREE II instrument in evaluating the development process and quality of current National Academy of Clinical Biochemistry guidelines. Clin Chem 2012; 58: 1426-37; http://dx.doi.org/10.1373/clinchem.2012.185850

16. Bennett MJ, ed. National Academy of Clinical Biochemistry Laboratory Medicine Practice Guidelines: Follow-up testing for metabolic diseases identified by expanded newborn screening using tandem mass spectrometry. Washington (DC): AACC Press, 2009

17. van den Berg T, Engelhardt EG, Haanstra TM, et al. Methodology of clinical nutrition guidelines for adult cancer patients: how good are they according to AGREE criteria? JPEN J Parenter Enteral Nutr 2012; 36: 316-22; http://dx.doi.org/10.1177/0148607111414027

18. Bozzetti F, Arends J, Lundholm K, et al. ESPEN guidelines on parenteral nutrition: non-surgical oncology. Clin Nutr 2009; 28: 445-54; http://dx.doi.org/10.1016/j.clnu.2009.04.011

19. Braga M, Ljungqvist O, Soeters P, et al. ESPEN guidelines on parenteral nutrition: surgery. Clin Nutr 2009; 28: 378-86; http://dx.doi.org/10.1016/j.clnu.2009.04.002

20. Burda BU, Norris SL, Holmer HK, et al. Quality varies across clinical practice guidelines for mammography screening in women aged 40-49 years as assessed by AGREE and AMSTAR instruments. J Clin Epidemiol 2011; 64: 968-76; http://dx.doi.org/10.1016/j.jclinepi.2010.12.005

21. Michigan Quality Improvement Consortium. Adult preventive services (ages 18-49). Southfield, MI: Michigan Quality Improvement Consortium, 2008

22. Lee CH, Dershaw DD, Kopans D, et al. Breast cancer screening with imaging: recommendations from the Society of Breast Imaging and the ACR on the use of mammography, breast MRI, breast ultrasound, and other technologies for the detection of clinically occult breast cancer. J Am Coll Radiol 2010; 7: 18-27

23. Lo Vecchio A, Giannattasio A, Duggan C, et al. Evaluation of the quality of guidelines for acute gastroenteritis in children with the AGREE instrument. J Pediatr Gastroenterol Nutr 2011; 52: 183-9; http://dx.doi.org/10.1097/MPG.0b013e3181e233ac

24. Sandhu BK. European Society of Pediatric Gastroenterology, Hepatology and NutritionWorking Group on Acute Diarrhoea. Practical guidelines for the management of gastroenteritis in children. J Pediatr Gastroenterol Nutr 2001; 33: S36-9; http://dx.doi.org/10.1097/00005176-200110002-00007

25. Manchikanti L, Singh V, Helm S 2nd, et al. A critical appraisal of 2007 American College of Occupational and Environmental Medicine (ACOEM) Practice Guidelines for Interventional Pain Management: an independent review utilizing AGREE, AMA, IOM, and other criteria. Pain Physician 2008; 11: 291-310

26. American College of Occupational and Environmental Medicine (ACOEM) Low Back Disorders. In Occupational Medicine Practice Guidelines: Evaluation and Management of Common Health Problems and Functional Recovery of Workers, Second Edition. Beverly Farms: OEM Press, 2007

27. American College of Occupational and Environmental Medicine (ACOEM) Chronic Pain. In Occupational Medicine Practice Guidelines: Evaluation and Management of Common Health Problems and Functional Recovery of Workers, Second Edition. Beverly Farms: OEM Press, 2008

28. Hulshof C, Hoenen J. Evidence-based practice guidelines in OHS: are they agree-able? Ind Health 2007; 45: 26-31; http://dx.doi.org/10.2486/indhealth.45.26

29. Cluzeau F, Wedzicha JA, Kelson M, et al.; ATS/ERS Ad Hoc Committee on Integrating and Coordinating Efforts in COPD Guideline Development. Stakeholder involvement: how to do it right: article 9 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc 2012; 9: 269-73; http://dx.doi.org/10.1513/pats.201208-062ST

30. Shah SGS, Robinson I. User involvement in healthcare technology development and assessment – structured literature review. Int J Health Care Qual Assurance 2006;19: 500-15; http://dx.doi.org/10.1108/09526860610687619

31. Achterkamp M, Vos JFJ. A framework for making sense of sustainable innovation through stakeholder involvement. Int J Environ Technol Mngmnt 2006; 6: 525-38

32. King N. Involvement in innovation: the role of identity. In: Shavinina LV (Ed.). The international handbook of innovation part VIII. Amsterdam: Elsevier, 2003

33. Rogers EM. Diffusion of innovations (5th ed.). New York: Free Press, 2003

34. Van De Ven AH, Scott Poole M. Alternative approaches for studying organizational change. Organisn Stud 2010; 26: 1377-404

35. Dettenkofera M, Ammonb A, Astagneauc P, et al. Infection control – a European research perspective for the next decade. J Hosp Infect 2011; 77: 7-10; http://dx.doi.org/10.1016/j.jhin.2010.07.025

36. Kyratsis Y, Ahmad R, Holmes A. Technology adoption and implementation in organisations: comparative case studies of 12 English NHS Trusts. BMJ Open 2012; 2: e000872; http://dx.doi.org/10.1136/bmjopen-2012-000872

37. Ahmad R, Kyratsis Y, Holmes A. When the user is not the chooser: learning from stakeholder involvement in technology adoption decisions in infection control. J Hosp Infect 2012; 81: 163-8; http://dx.doi.org/10.1016/j.jhin.2012.04.014

38. National Institute for Health and Clinical Excellence and Health Protection Agency Prevention and control of healthcare-associated infections quality improvement guide. London: NICE/HPA, 2011

39. Pittet D, Panesar SS, Wilson K, et al. Involving the patient to ask about hospital hand hygiene: a National Patient Safety Agency feasibility study. J Hosp Infect 2011; 77: 299-303; http://dx.doi.org/10.1016/j.jhin.2010.10.013

40. Dougherty RH. Consumer-directed healthcare: the next trend? Behavioral Healthcare Tomorrow 2003; 12: 21-7

41. Department of Health & Farrell C. Patient and Public Involvement in Health: The Evidence for Policy Implementation. A Summary of the Results of the Health in Partnership Research Programme. London: Department of Health, 2004

42. Patterson S, Weaver T, Agath K, et al. “They can’t solve the problem without us”: a qualitative study of stakeholder perspectives on user involvement in drug treatment services in England. Health Soc Care Community 2009; 17: 54-62; http://dx.doi.org/10.1111/j.1365-2524.2008.00797.x

43. Crawford MJ, Aldridge T, Bhui K, et al. User involvement in the planning and delivery of mental health services: a cross-sectional survey of service users and providers. Acta Psychiatrica Scandinavica 2003; 107: 410-4; http://dx.doi.org/10.1034/j.1600-0447.2003.00049.x