Background: Evaluation has an integral role in effective health promotion, yet few large scale health promotion interventions prospectively incorporate rigorous evaluation into their planning processes. To promote greater use of evaluation we reviewed the health promotion literature incorporating evaluations and have produced a succinct summary.

Methods: MEDLINE and SCOPUS databases were accessed to identify health promotion publications, which utilised formal evaluation. Publication were then summarized and reviewed informing on evaluation methods relevant to health promotion.

Results: Numerous health promotion evaluation levels and approaches exist some of which include objective-based, needs-based, collaborative and utilization focused. Multiple evaluation frameworks are available including the RE-AIM, Centre for Disease Control and Prevention (CDC) and Medical Research Council framework (MRC). Data collection methods often include both qualitative and quantitative methods and validated tools are limited but important. Evaluation approaches and frameworks have been applied effectively to diverse health programs with useful learnings.

Evaluation enablers include:

1. Prospective planning
2. Tailoring the evaluation to the study context
3. Engaging with evaluation target audience early
4. Ample time to undertake the evaluation

Conclusion: Evaluation of health promotion interventions is imperative to deliver benefits of research into improved health outcomes. Understanding of evaluation concepts including levels, approaches and framework and methods is needed to facilitate consistent use of evaluation in research. Ultimately, health promotion programs require context specific adaptable evaluations. Greater opportunities exist for shared learnings to build evaluation capacity and to deliver greater health impacts.

Keywords: evaluation, health promotion, evaluating health programs, evaluation frameworks, translational research

CDC, centre for disease control and prevention; MRC, medical research council framework; RE-AIM, reach, efficacy, adoption, implementation and maintenance; CIP, context, input, process and product; NHMRC, national health and medical research council

As the global prevalence of obesity and chronic diseases continues to rise, the need for effective health promotion programs is imperative. Whilst research into effectiveness of health promotion programs is needed to improve population health outcomes, translation of these research findings into policy and practice is crucial. Translation requires not only efficacy data around what to implement, but also information on how to implement it. Evaluation seeks to optimise translation by answering questions related to how to implement evidence-based interventions under real world conditions.¹ Evaluation studies the implementation strategies and adds value to efficacy outcomes,² potentially informing on program reach, fidelity and sustainability. Hence evaluation is now recognised as an integral component of all health promotion programs.² Ultimately, comprehensive evaluation provides essential knowledge about program implementation practices and processes, which is vital to the translation of programs into varied setting and population contexts.^3,4

Presently, evaluation has been applied inconsistently to health promotion programs,⁵ limiting the translation of knowledge. Health promotion programs have further been criticised for insufficient evaluation planning⁶ and scrutinised as “fail (ing) to contribute to their own quality enhancement”.⁵ Barriers to evaluation include funding constraints, lack of knowledge, skills and familiarity with evaluation methodologies and poor availability of measurement tools. Information related to evaluation is often difficult to access through conventional academic literature and is more often located in non-academic grey literature. Additionally, terminology in this field is inconsistent and there is considerable overlap between areas such as implementation research and evaluation.

In this context, we aimed to review and summarize the literature and discuss planning and development of comprehensive health promotion evaluations. This is relevant to health professionals, researchers and end users, who seek insight into evaluation designs and are engaged in driving evidence into policy and practice. An enhanced appreciation of evaluation methodologies and terminology aims to provide a foundation for those new to evaluation.

MEDLINE and SCOPUS databases were used to identify publications related to the evaluations of health promotion interventions with no publications date restrictions. Techniques such as “snowballing” otherwise referred to as reference tracking were also utilised. This method involves scanning reference lists of all full text papers and using judgment to decide whether to pursue texts further.7 A systematic review was not conducted, as it was essential to locate literature outside databases containing published journals (text books, government and non for profit websites), as this is a regular repository for evaluation literature. Publications were included and reviewed if it clearly described an evaluation planning process or methodology and had applied these methods to a health promotion intervention. Publications were excluded if it did not include a formal evaluation methodology inclusive of evaluation levels, approaches and frameworks within the context of health promotion programs.

Evaluation planning and purpose

Literature suggests that in depth planning is critical to comprehensive evaluation. The first step in evaluation planning is to identify the purpose for undertaking the evaluation and formulating clear evaluation objectives and questions. Broadly, health promotion evaluation purposes and key questions will commonly focus on program effectiveness and/or program implementation. Evaluation is not an absolute science⁸ and a program evaluation wills unlikely represent all perspectives and components of an intervention, resulting in a “balancing act” or “trade-off” between priorities.⁹ Gathering information on a large number of issues may result in the evaluation lacking depth to draw a confident conclusion.¹⁰ Deciding which aspects to evaluate will need to be informed by relevant health literature, resources allocated, stakeholder needs and determined based on the primary targets audience of the evaluation purpose of the health program,¹¹ mandating the importance of early stakeholder engagement. Stakeholders may include funders, end users, service providers, government employees or the general public. Importantly, providing ample time to plan and conduct an evaluation is imperative. For an overview of the common phases of evaluation planning and implementation please refer to Figure 1 based on the work of Øvretveit⁹ and Van Marris.¹²

Figure 1 Evaluation planning and implementation.

Evaluation concepts

Areas for consideration when developing an evaluation plan can include evaluation levels (also referred to as types), approaches, framework (also referred to as models) and data collection tools. We recommend having an understanding of these commonly used evaluations ‘concepts’ and terminology.

Evaluation levels or types: Evaluation ‘levels’ include process, formative, summative, impact and outcome evaluations. The levels are determined by the purpose of the evaluation and are influenced by the state of the program (under development, settled) and the timing of data collection (before program roll out, during implementation or post implementation).¹³ In Table 1 & 2 we describe evaluation levels (or types) and note that elements can overlap. Common elements assessed in a process evaluation include program reach, fidelity in relation to program protocol, program context, quality and dose delivered and received by participants.^14,15 A community based obesity prevention study in adolescents living in Tonga completed a process evaluation by research staff recording all intervention-related activities, the frequency of the activity, the reach of that activity (how many people were involved in the program) and the resources required.¹⁶ The purpose of a formative evaluation is to improve the quality of performance or the delivery of the program prior to roll-out and is conducted either before or during the program.¹⁷ On the contrary, summative, impact and outcome evaluations typically occur at the completion of an intervention, yet these three evaluation methods differ significantly. In the large scale “Be Active Eat Well” program which promoted healthy lifestyles in disadvantaged communities, impact and outcome evaluation results were assessed as environmental changes, policy implementation and anthropometric data two years post program commencement.¹⁸ When used alone an impact, outcome and summative evaluation may miss the depth of the data available during intervention roll-out. However, process evaluation requires preplanning with stakeholders to enable data collection throughout program delivery. We suggest that combining a range of evaluation levels [E.g. process, summative and outcome evaluation] can strengthen capacity to translate successful health promotion interventions to an array of settings and target groups.

Evaluation approaches: An evaluation approach informs on ‘how’ we will conduct the evaluation i.e. will stakeholders be engaged to participate in the evaluation or will the evaluation be conducted by the researcher only? Each evaluation approach has a number of associated steps, guiding the processes and activities of the evaluation (Table 3). There are a number of evaluation approaches available some of which include objective-based, needs-based, theory based, collaborative, utilization focused and realistic evaluation. Typically there is a large degree of overlap between varied evaluation approaches; however the emphasis and the tasks related to each step of the evaluation varies in accordance with the nature and purpose of the evaluation.¹⁹ Please note this is not an extensive list of approaches, rather we have included approaches that are frequently applied to health promotion evaluations.

Evaluation frameworks or models: Similarly, evaluation frameworks provide detailed guidance for evaluators, as they ensure the evaluation design considers the origins and contexts of the program being examined. An evaluation framework can encourage the prioritisation of evaluation purpose and the selection of data collection tools.²⁰ Moreover, evaluations are program specific as there is no “one size fits all” evaluation design in evaluation.¹⁹ Evaluators will often utilise more than one evaluation level, approach and framework in order to meet the needs and purpose of their evaluation.

Evaluation frameworks

From the literature, the commonly applied evaluation frameworks used in health promotion research include the RE-AIM, Context, Input, Process and Product (CIPP), Predisposing, Reinforcing, Enabling, Constructs in Educational/Ecological Diagnosis and Evaluation (PROCEED, PRECEDE), Centre for Disease and Control and Prevention (CDC), Medical Research Council (MRC) and programme logic model. Here we describe briefly these evaluation frameworks, prior application and practical insights from our experiences of utilising these frameworks.

RE-AIM framework: The RE-AIM (Reach, efficacy, adoption, implementation and maintenance) framework is an adaptable and simple framework for evaluating large scale projects, which considers both the individual and population impacts of the program.²¹ RE-AMI focuses on informing and optimising translation and has been applied numerous times to health promotion programs for example to a diabetes self-management programs²² and school based physical activity programs.²³ In our experience this framework is useful because it is adaptable, easy to use and supports the development of a focused evaluation. The associated resources provide clear examples of evaluation questions, potential measurement indicators and data collection tools which are helpful for novice evaluators.²⁴ However, it is worth noting that within this framework ‘maintenance’ is measured at two years post program delivery; therefore consideration of evaluation timelines is essential. In comparison to other frameworks the RE-AIM is less flexible and the relationship between the five domains of the RE-AIM framework remains unclear.

CIPP framework: The Context, Input, Process and Product (CIPP) framework for evaluation is comprehensive and useful for guiding formative and summative evaluations. The context of the program refers to the evaluation environment in which the program will be implemented. The input refers to the resources and activities required to meet the objectives of the program. The final stage of the CIPP model is the product, which refers to the judgment of the effectiveness and success of the program.^6,25 This framework has been applied to health promotion programs previously and examples include a diabetes management program²⁶ and a behavioural intervention targeting reduction in metabolic syndrome.²⁷ The CIPP framework is highly flexible and adaptable and specifically useful for evaluations seeking insight into a range of factors that influence program effectiveness such as environmental and contextual factors. However, this framework provides less guidance for evaluation planning, including developing evaluation objectives and key questions, compared to the RE-AIM framework.

PROCEED/PRECEDE framework: The PROCEED/PRECEDE is a theoretically grounded and comprehensive framework, combining program planning, implementation and evaluation. It is based on the premise that a thorough social, educational and ecological assessment (Precede) should take place prior to program development and that evaluation is required to assess effectiveness (Proceed). Phase include:

1. Social assessment
2. Epidemiological, environmental and behavioural intervention
3. Educational and ecological assessment
4. A. Intervention alignment
   B.Administrative and policy assessment
5. Implementation
6. Process evaluation
7. Impact and outcome evaluation²⁸

The framework has been applied to multiple health promotion programs including mental health programs.²⁹ The PROCEED/PRECEDE models is a structured and valuable framework for multi-strategy program evaluations and considers multiple determinants of health. However, this framework is complex and has been criticised for being a very medically based and requires significant time to prospectively plan and undertake the evaluation.¹⁹

Centre for disease and control and prevention (organisational framework): The Centre for Disease and Control and Prevention (CDC) framework suggests that effective program evaluation is a systematic way to improve public health actions by conducting evaluations that ensure useful, feasible, ethical and accurate results. The framework outlines 6 inter-connected steps to planning and delivering an evaluation

1. Engage stakeholder
2. Describe the program
3. Focus the evaluation design
4. Gather credible evidence
5. Justify conclusions
6. Share lessons learnt¹⁹

The framework is highly reputable and has been applied to numerous health promotion program including physical activity programs³⁰ and tobacco control initiatives.³¹ In our experience the CDC framework is very practical and easy to follow and is useful for the evaluation of simple through to complex programs and partnerships. It is particularly valuable for capacity building evaluations as it incorporates multiple stakeholder perspectives into the evaluation. Another advantage of the CDC model is that it is underpinned by four central standards for ensuring ethically sound evaluations (utility, feasibility, propriety and accuracy). However, the framework is broad and may be less useful to novice evaluators, seeking a supportive framework to assist in the development of evaluation objectives and questions. Also this framework may require significant funding and strong evaluation facilitation.¹⁹

Medical research council (organisational framework): The Medical Research Council (MRC) framework is theory based and describes the development, evaluation and implementation. The stages involved include:

1. Development of an intervention,
2. Pilot and feasibility assessment,
3. Evaluation of the intervention,
4. Implementation, dissemination, monitoring and long term follow up.³²

Examples of previous application include a physical activity program targeting a population at high risk of diabetes³³ and evaluation of a cardiovascular disease management program.³⁴ The advantages of the MRC framework are that it is highly flexible and promotes a non-linear approach to evaluation with many supportive documents to assists evaluation planning and implementation. However, the framework is broad and does not provide depth and detail around developing evaluation objectives and key questions. Similarly to the CDC framework, the MRC framework may be less useful to novice evaluators, who may prefer a more structured and supportive framework.

Evaluation methodological approaches

Logic model/programme theory methodological approach

A logic model is a methodological approach to program planning and evaluation, describing the assumptions and resources needed to support the activities of the program of interest.³⁵ A logic model typically identifies the background of the program, the resources (inputs), the activities (outputs), immediate impacts and short and long term outcomes.³⁶ Logic models have been applied to diverse health programs including the evaluation of community nutrition education programs³⁷ and a school based programs aimed at reduce the frequencies of teenage pregnancies.³⁸ The value of logic models is well established in health promotion and resources available to develop logic models.³⁹ Logic models provide a practical means to assess the relationship between program inputs and anticipated outcomes. However, a caution is that logic models may not capture all important aspects of programs and associated policies. Logic models are also time consuming requiring prospective planning to consider the underlying program theory and relationships between processes and outcomes.¹⁹

Overall

Ultimately, evaluation methods selected will vary according to the purpose, timing, resources available, stakeholders involved in the evaluation and their prior experience conducting evaluations. Combining a range of evaluation approaches and frameworks is commonly accepted in the literature, enabling evaluators to account for potential methodological limitations. Structured evaluation frameworks may not be appropriate for all evaluations contexts, emphasising the importance of rigorous evaluation planning and the need for context specific adaptation of evaluation frameworks. However, the importance of stakeholder engagement as an integral component of evaluation conduct is recognised across the diverse evaluation frameworks, emphasising the importance of communication and engagement of stakeholders to yield useful evaluation results.⁴⁰ In planning a health promotion program, resources ideally need to be allocated for the evaluation. However, funding bodies commonly fund research but less commonly fund evaluation and advocacy is needed to address this. Providing ample funding for major evaluation strategies would likely shift evaluations from being perceived as a “minor after thought” to a major integral component of a health promotion program.⁵

Data collection methods and tools

A summary of the advantages and disadvantages of quantitative and qualitative data collection methods are outlined in Table 4. The emphasis of qualitative research is depth, detail and individual perceptions.¹⁰ Quantitative research methods use standardised data collection tools to fit diverse opinions and experiences into often predetermined categories.⁸ Most evaluators recognise the synergistic benefits of mixed methods, which incorporate quantitative and qualitative research.⁴¹ Data “triangulation” provides an opportunity to integrate data in mixed-methods studies. Data triangulation protocols commonly involve initially analysing qualitative and quantitative data separately and then comparing the findings from each method, to determine whether they support or contradict each other.⁴² The purpose of data triangulation is to enhance the reliability and confidence of research findings. A practical example of data triangulation in the health promotion setting could involve comparing anthropometric measurements with results from a physical activity questionnaire and focus groups (qualitative), following a specific health promotion intervention.^14,43

Existing validated data tools are preferable for large pre- and post-intervention surveys, as they provide validity and confidence that any change identified is likely to reflect a meaningful result. Yet, it is important to only employ validated tools if they are useful for the evaluation.⁴⁴ Currently there are limited reliable and validated evaluation tools available,¹⁴ with a clear need to develop program specific tools. However, limited large scale studies are beginning to share evaluation tools on websites for public use and descriptions of process evaluation objectives and associated data collections tools employed are increasingly being published in peer reviewed journals. This has occurred in interventions in mental health,⁴⁵ weight management¹⁶ and physical activity.⁴⁶ There are additionally some evaluation guides available online (grey literature) that incorporate evaluation templates as well as sample questions for semi-structured interviews and focus groups.^47,48 There are also guides available to describe the optimal methods of reporting evaluation outcomes, specific to the health promotion context.⁴⁹ Ultimately, data collection tools frequently require adaptation to accommodate and capture within health program processes, which are unique to an intervention, increasing sensitivity and usefulness of results.⁸ Combining a range of data collection methods can partly address the inherent weaknesses of each method. Determining evaluation objectives upfront is integral to ensure that data gathering methods yield useful and appropriate information.⁹ In future, increased availability of validated evaluation tools will improve evaluation standards and enable better comparability between studies.¹⁴

Developing a robust evaluation is challenging due to the lack of guidance in conventional published literature and the wide range of evaluation concepts across evaluation level, approach and framework. There are five available evaluation levels which include formative, process, impact, outcome and summative evaluation. Multiple evaluation approaches are available some of which include objective-based, needs based, collaborative, realistic and utilization focused evaluations. In addition to evaluation levels and approaches, numerous health promotion evaluation frameworks are available such as RE-AIM, CDC, MRC, logic model, CIPP and PREECE/PROCEDE. These evaluation components have been applied effectively to diverse health promotion programs and target populations. Commonly, evaluators will combine and use various aspects of evaluation levels, approaches and frameworks to meet their evaluation purposes. We highlight, that each evaluation will require planning and context specific adaptation to produce useful and credible results.⁵⁰

Moving forward, funders as key stakeholders are increasingly seeking knowledge about how to translate evidence into practice. Increasingly, researchers are likely to be held accountable for optimising opportunities for translating research into practice or at least for ensuring that the evidence generated during their research also informs and enables translation.⁵¹ Subsequently, the importance of up skilling our workforce to build greater evaluation capacity has been established.⁵² Evaluation is a vital step to inform translation; consequently researchers face the challenge of deciding which evaluation level, approach and framework to apply to drive translation. A structured evaluation approach has some practical limitations such as inherently being both resource and labour intensive.¹⁴ Presently, most evaluations involve a combination of program specific questionnaires (developed and/or validated tools) and qualitative research methods, in order to ensure a robust evaluation design.^14,43 An evaluation consultant can provide guidance for making major decisions such as selecting the evaluation level, appropriate framework and data collection methods.

Broad scale evaluations of health promotion interventions are lacking and use of evaluation terminology is ambiguous and inconsistent in the literature. Many health promotion interventions may show efficacy but commonly do not have comprehensive integrated prospective evaluations, limiting the ability to translate outcomes and deliver return on research investments.⁵ This is further confounded by difficulties in accessing published evaluation literature through accessible conventional academic avenues. However, utilising existing levels, approaches and framework creates a greater opportunity to critique and compare evaluations across different health interventions. Ultimately, standardisation of evaluation reporting would enhance the field across a diverse array of health settings. Opportunities for shared learnings, avoidance of repeated mistakes and efficient allocation of resources also exist in this field. Future directions require clear definition of terms in, evaluation, as well as further development of well tested evaluation tools. Improved sharing of evaluation processes, planning and tools will assist novice evaluators to more easily incorporate comprehensive evaluation methods into their research plans and will improve translation of evidence into policy and practice.

Comprehensive within program evaluations is integral to enable translation and scale-up of effective health promotion programs. As health promotion interventions vary significantly, the authors do not suggest a “one size fits all” in evaluation, as each evaluation needs to be tailored to the context and circumstance to ensure results provide credible and useful information. Evaluation concepts including level, approach, framework along with data collections methods and tools all require consideration. Enablers include prospective planning, providing adequate time to develop the evaluation, defining key objectives and target audiences for the evaluation and ensuring the evaluation is practical and within resources allocated. Sound evaluation focuses on the importance of real world impacts and attempts to bridge the gap between research, clinical practice and policy. Arguable evaluation should not be optional, but rather an obligatory in future health promotion research and the summary provided here will assist health promotion researchers to integrate evaluation into future health promotion programs.

This work was supported by the National Health and Medical Research Council (NHMRC). Samantha Kozica is an NHMRC postgraduate scholar, Cheryce Harrison is a National Heart Foundation Postdoctoral Research Fellow (100168) and Helena Teede is a NHMRC Practitioner Fellow.

The author declares no conflict of interest.

1. Rabin BA, Brownson RC, Haire–Joshu D, et al. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14(2):117–123.
2. Kumanyika S, Jeffery R, Morabia A, et al. Obesity prevention: the case for action. Int J Obes Relat Metab Disord. 2002;26(3):425–436.
3. Oldenburg B, Absetz P. Lost in translation: overcoming the barriers to global implementation and exchange of behavioral medicine evidence. Transl Behav Med. 2011;1(2):252–255.
4. Fixsen DL, Naoom SF, Blase KA, et al. Implementation research: A synthesis of the literature: National Implementation Research Network. 2005.
5. Swinburn B, Bell C, King L, et al. Obesity prevention programs demand high–quality evaluations. Aust N Z J Public Health. 2007;31(4):305–307.
6. Fleming ML, Parker E. Health promotion: Principles and practice in the Australian context. Australia: Allen & Urwin; 2007.
7. Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ. 2005;331(7524):1064–1065.
8. Hawe P, Degeling D, Hall J. Evaluating health promotion. Sydney: MacLennan & Petty; 1990.
9. Ovretveit J. Evaluating health interventions: An introduction to evaluation of health treatments, services, policies, and organizational interventions. Open University Press; 1997.
10. Patton MQ. How to use qualitative methods in evaluation. USA: Sage Publications; 1987.
11. Rossi, Lipsey, Freeman. Evaluation: A systematic approach. 7th ed. USA: Sage Publications; 2004.
12. Marris BV, King B. Evaluating Health Promotion Programs Version 3.6. The Health Communication Unit, Centre for Health Promotion, University of Toronto; 2007.
13. Owen JM. Program evaluation: Forms and approaches. Australia: Allen & Unwin; 2006.
14. Nutbeam D. Evaluating health promotion–progress, problems and solutions. Health Promot Int. 1998;13(1):27–44.
15. Saunders RP, Evans MH, Joshi P. Developing a process–evaluation plan for assessing health promotion program implementation: a how–to guide. Health Promot Pract. 2005;6(2):134–47.
16. Fotu KF, Moodie MM, Mavoa HM, et al. Process evaluation of a community–based adolescent obesity prevention project in Tonga. BMC public health. 2011;11:284.
17. Green LW, Lewis FM. Measurement and evaluation in health education and health promotion. California: Mayfield Publishing Company; 1986.
18. de Silva–Sanigorski AM, Bolton K, Haby M, et al. Scaling up community–based obesity prevention in Australia: background and evaluation design of the Health Promoting Communities: Being Active Eating Well initiative. BMC public health. 2010;10:65.
19. Kahan B. Review of Evaluation Frameworks. Saskatchewn Minstry of Education; 2008.
20. Fink A. Evaluation fundamentals: Insights into the outcomes, effectiveness, and quality of health programs. 2nd ed. USA: Sage Publications; 2004.
21. Glasgow RE, McKay HG, Piette JD, et al The RE–AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns. 2001;44(2):119–27.
22. Glasgow RE, Nelson CC, Strycker LA, et al. Using RE–AIM Metrics to Evaluate Diabetes Self–Management Support Interventions. Am J Prev Med. 2006;30(1):67–73.
23. De Meij JS, Chinapaw MJ, Kremers SP, et al. Promoting physical activity in children: The stepwise development of the primary school–based JUMP–in intervention applying the RE–AIM evaluation framework. Br J Sports Med. 2010;44(12):879–87.
24. McKenzie R, Naccarella L, Stewart A, et al. Targeting what matters in health promotion evaluation. Evaluation Journal of Australasia. 2007;7(1):19–26.
25. Stufflebeam LD. Cipp Model (Context, Input, Process, Product). Encyclopedia of Evaluation. USA: Sage Publications; 2005.
26. Clinton J, McNeill R, Irvine R, et al. An evaluation of the Gardening for Health and Sustainability initiative: an initiative supported by Let's Beat Diabetes. Counties Manukau District Health Board, 2009:1–44.
27. Intarakamhang U, Patcharee D. Effects of health behavioral modification program on metabolic diseases in risk Thai clients. Asian Biomedicine. 2012;6:2.
28. Green LW, Kreuter KW. Health promotion planning: An educational and Ecological Approach. 2nd ed. USA: Mayfield Publishing Company; 1998.
29. Wright A, McGorry PD, Harris MG, et al. Development and evaluation of a youth mental health community awareness campaign–The Compass Strategy. BMC public health. 2006;6:215.
30. Gamez R, Parra D, Pratt M, et al. Muevete Bogota: promoting physical activity with a network of partner companies. Promot Educ. 2006;13(2):138–143.
31. MacDonald G, Starr G, Schooley M, et al. Introduction to program evaluation for comprehensive tobacco control programs. USA: Centers for Disease Control and Prevention; 2001.
32. Medical Research Council. A framework for development and evaluation of RCTs for complex interventions to improve health. UK: Medical Research Council; 2000.
33. Hardeman W, Sutton S, Griffin S, et al. A causal modelling approach to the development of theory–based behaviour change programmes for trial evaluation. Health Educ Res. 2005;20(6):676–687.
34. Robinson L, Francis J, James P, et al. Caring for carers of people with stroke: developing a complex intervention following the Medical Research Council framework. Clin Rehabil. 2005;19(5):560–571.
35. Cooksy LJ, Gill P, Kelly PA. The program logic model as an integrative framework for a multimethod evaluation. Evaluation and Program Planning. 2001;24(2):119–128.
36. Helitzer D, Hollis C, de Hernandez BU, et al. Evaluation for community–based programs: the integration of logic models and factor analysis. Eval Program Plann. 2010;33(3):223–233.
37. Medeiros LC, Butkus SN, Chipman H, et al. A logic model framework for community nutrition education. J Nutr Educ Behav. 2005;37(4):197–202.
38. Hulton LJ. An evaluation of a school–based teenage pregnancy prevention program using a logic model framework. J Sch Nurs. 2007;23(2):104–110.
39. Renger R, Titcomb A. A three–step approach to teaching logic models. American Journal of Evaluation. 2002;23(4):493–503.
40. Johnson K, Greenseid LO, Toal SA, et al. Research on Evaluation Use A Review of the Empirical Literature From 1986 to 2005. American Journal of Evaluation. 2009;30(3):377–410.
41. Creswell JW, Clark VLP. Designing and conducting mixed methods research. Australian and New Zealand Journal of Public Health. 2007;31(4):388.
42. Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. BMJ. 2010:341.
43. Davidson EJ. Evaluation methodology basics: The nuts and bolts of sound evaluation. USA: Sage Publications; 2005.
44. Hider K. Frequently Asked Questions about Evaluation. In: SatCfC, editor. Southern Health C–CCH, Effectiveness. Melbourne.
45. Nakkash RT, Alaouie H, Haddad P, et al. Process evaluation of a community–based mental health promotion intervention for refugee children. Health Educ Res. 2012;27(4):595–607.
46. Jenkinson KA, Naughton G, Benson AC. The GLAMA (Girls! Lead! Achieve! Mentor! Activate!) physical activity and peer leadership intervention pilot project:a process evaluation using the RE–AIM framework. BMC public health. 2012;12:55.
47. Zarinposh F. Project Evaluation Guide For Nonprofit Organizations: Fundamental Methods and Steps For Conducting Project Evaluation. Canada: Canada Volunteer Initiative; 2006.
48. de Groot F, Robertson N, Swinburn B, et al. Increasing community capacity to prevent childhood obesity: challenges, lessons learned and results from the Romp & Chomp intervention. BMC public health. 2010;10(1):522.
49. Moskowitz JM. Preliminary guidelines for reporting outcome evaluation studies of health promotion and disease prevention programs. New directions for program evaluation. 1989;1989(43):101–112.
50. Greene JC, Lipsey MW, Schwandt TA, et al. Method choice: Five discussant commentaries. New Directions for Evaluation. 2007;2007(113):111–127.
51. Adams J, Dickinson P. Evaluation training to build capability in the community and public health workforce. American journal of evaluation. 2010;31(3):421–433.
52. Suarez–Balcazar Y, Taylor–Ritzler T Moving From Science to Practice in Evaluation Capacity Building. American Journal of Evaluation. 2013.
53. Rubio DM, Schoenbaum EE, Lee LS, t al. Defining translational research: implications for training. Acad Med. 2010;85(3):470–475.
54. Green LW, Kreuter MW. Health Promotion Planning: an Educational and Ecological Approach. 3rd ed. USA: Mayfield Publishing Company; 1999.
55. Scriven M. Key evaluation checklist. Evaluation Checklists Project. 2007.
56. Rossi PH, Freeman HE, Lipsey MW. Evaluation: A systematic approach. 6th ed. USA: SAGE Publications; 1993.
57. Peters DH, Adam T, Alonge O, et al. Implementation research: what it is and how to do it. BMJ. 2013;347:f6753.
58. Graham ID, Logan J, Harrison MB, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13–24.
59. Straus SE, Tetroe J, Graham I. Defining knowledge translation. CMAJ. 2009;181(3–4):165–168.
60. Mangham LJ, Hanson K. Scaling up in international health: what are the key issues? Health Policy Plan. 2010;25(2):85–96.
61. Slatter T. Measuring healt promotion impacts: A guide to impact evaluation in integrated health promotion. Victoria: Victorian Government Department of Human Services; 2003.
62. McKenzie JF, Neiger B, Smeltzer J. Planning, implementing, and evaluating health promotion programs: a primer. San Francisco: Pearson/Benjamin Cummings; 2005.
63. Naidoo J, Wills J. Health promotion: foundations for practice. UK: Bailliere Tindall; 1994.
64. Miller ER, Erlinger TP, Young DR, et al. Results of the diet, exercise, and weight loss intervention trial (DEW–IT). Hypertension. 2002;40(5):612–618.
65. Wong RK, Franssen E, Szumacher E, et al. What do patients living with advanced cancer and their carers want to know?–a needs assessment. Support Care Cancer. 2002;10(5):408–415.
66. Donaldson SI, Gooler LE. Theory–driven evaluation in action: Lessons from a $20 million statewide work and health initiative. Evaluation and Program Planning. 2003;26(4):355–366.
67. Sanigorski A, Bell A, Kremer P, et al. Reducing unhealthy weight gain in children through community capacity–building: results of a quasi–experimental intervention program, Be Active Eat Well. Int J Obes. 2008;32(7):1060–1097.
68. Sharma M, Deepak S. A participatory evaluation of community–based rehabilitation programme in North Central Vietnam. Disabil Rehabil. 2001;23(8):352–358.
69. Patton MQ. Utilization–Focused Evaluation: The New Century Text. 3rd ed. United Kingdom: SAGE publications; 1996.
70. Stetler CB, Effken J, Frigon L, et al. Utilization–focused evaluation of acute care nurse practitioner role. Outcomes Manag Nurs Pract. 1998;2(4):152–160.
71. Pommier J, Guevel MR, Jourdan D. Evaluation of health promotion in schools: a realistic evaluation approach using mixed methods. BMC public health. 2010;10(1):43.
72. Russ–Eft D, Preskill H. Evaluation in organizations: A systematic approach to enhancing learning, performance, and change. Arizona: Basic Books; 2001.
73. Pope C, Ziebland S, Mays N. Analysing qualitative data. BMJ. 2000;320(7227):114–116.