PCMH Evaluation Guide | pcmh.ahrq.gov
AHRQ Releases Guide to Real-World Evaluations of Primary Care Interventions
Effective evaluation can help determine the best ways to improve primary care health and cost outcomes as well as patient, clinician and staff experiences. AHRQ has released a new evaluation guide for designing real-world evaluations of interventions such as the patient-centered medical home and other models to improve primary care delivery. The guide presents practical steps for designing a primary care evaluation. It can be used by health care delivery systems, employers, practice-based research networks, local or regional insurers and others who want to test an intervention in a relatively small number of primary care practices with limited resources. In addition to this guide, AHRQ has developed a number of resources for evaluators of primary care interventions like the patient-centered medical home.
Highlights
Papers, Briefs, and Other Resources provides access to all of AHRQ's resources on the PCMH
For Policymakers
For Researchers
New Case Studies of Primary Care Practice Facilitation Programs
A How-to Guide on Developing and Running a Primary Care Practice Facilitation Program
New PCMH Research Methods Series
A Guide to Real-World Evaluations of Primary Care Interventions: Some Practical Advice
Prepared for:
Agency for Healthcare Research and Quality
U.S. Department of Health and Human Services
540 Gaither Road Rockville, MD 20850 www.ahrq.gov
Prepared by:
Mathematica Policy Research, Princeton, NJ
Project Director: Deborah Peikes
Principal Investigators: Deborah Peikes and Erin Fries Taylor
Authors:
Deborah Peikes, Ph.D., M.P.A., Mathematica Policy Research
Erin Fries Taylor, Ph.D, M.P.P, Mathematica Policy Research
Janice Genevro, Ph.D., Agency for Healthcare Research and Quality
David Meyers, M.D., Agency for Healthcare Research and Quality
October 2014
Disclaimer
None of the authors has any affiliations or financial involvement that conflicts with the material presented in this guide.
Acknowledgments
The authors gratefully acknowledge the helpful comments on earlier drafts provided by Drs. Eric Gertner, Lehigh Valley Health Network; Michael Harrison, AHRQ; Malaika Stoll, Sutter Health; and Randall Brown and Jesse Crosson, Mathematica Policy Research. We also thank Cindy George and Jennifer Baskwell of Mathematica for editing and producing the document.
This project was funded under contract HHSA290200900019I from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services. The opinions expressed in this document are those of the authors and do not reflect the official position of AHRQ or the U.S. Department of Health and Human Services.
AHRQ Publication No. 14-0069-EF
Quick Start to This Evaluation Guide
Goals. Effective primary care can improve health and cost outcomes, and patient, clinician and staff experience, and evaluations can help determine how best to improve primary care to achieve these goals. This Evaluation Guide provides practical advice for designing real-world evaluations of interventions such as the patient-centered medical home (PCMH) and other models to improve primary care delivery.
Target audience. This Guide is designed for evaluators affiliated with delivery systems, employers, practice-based research networks, local or regional insurers, and others who want to test a new intervention in a relatively small number of primary care practices, and who have limited resources to evaluate the intervention.
Summary. This Guide presents some practical steps for designing an evaluation of a primary care intervention in a small number of practices to assess the implementation of a new model of care and to provide information that can be used to guide possible refinements to improve implementation and outcomes. The Guide offers options to address some of the challenges that evaluators of small-scale projects face, as well as insights for evaluators of larger projects. Sections I through V of this Guide answer the questions posed below. A resource collection in Section VI includes many AHRQ-sponsored resources as well as other tools and resources to help with designing and conducting an evaluation. Several appendices include additional technical details related to estimating quantitative effects.
Do I need an evaluation? Not every intervention needs to be evaluated. Interventions that are minor or inexpensive, have a solid evidence base, or are part of quality improvement efforts may not warrant an evaluation. But many interventions would benefit from study. To decide whether to conduct an evaluation, it’s important to identify the specific decisions the evaluation is expected to inform and to consider the cost of carrying out the evaluation. An evaluation is useful for interventions that are substantial and expensive and lack a solid evidence base. It can answer key questions about whether and how an intervention affected the ways practices deliver care and how changes in care delivery in turn affected outcomes. Feedback on implementation of the model and early indicators of success can help refine the intervention. Evaluation findings can also help guide rollout to other practices. One key question to consider: Can the evaluation that you have the resources to conduct generate reliable and valid findings? Biased estimates of program impacts would mislead stakeholders and, we contend, could be worse than having no results at all. This Guide has information to help you determine whether an evaluation is needed and whether it is the right choice given your resources and circumstances.
What do I need for an evaluation? Understanding the resources needed to launch an intervention and conduct an evaluation is essential. Some resources needed for evaluations include (1) leadership buy-in and support, (2) data, (3) evaluation skills, and (4) time for the evaluators and the practice clinicians and staff who will provide data to perform their roles. It’s important to be clear-sighted about the cost of conducting a well-designed evaluation and to consider these costs in relation to the nature, scope, and cost of the intervention.
How do I plan an evaluation? It’s best to design the evaluation before the intervention begins, to ensure the evaluation provides the highest quality information possible. Start by determining your purpose and audience so you can identify the right research questions and design your evaluation accordingly. Next, take an inventory of resources available for the evaluation and align your expectations about what questions the evaluation can answer with these resources. Then describe the underlying logic, or theory of change, for the intervention. You should describe why you expect the intervention to improve the outcomes of interest and the steps that need to occur before outcomes would be expected to improve. This logic model will guide what you need to measure and when, though you should remain open to unexpected information as well as consequences that were unintended by program designers. The logic model will also help you tailor the scope and design of your evaluation to the available resources.
How do I conduct an evaluation, and what questions will it answer? The next step is to design a study of the intervention’s implementation and—if you can include enough practices to potentially detect statistically significant changes in outcomes—a study of its impacts. Evaluations of interventions tested in a small number of practices typically can’t produce reliable estimates of effects on cost and quality, despite stakeholders’ interest in these outcomes. In such cases, you can use qualitative analysis methods to understand barriers and facilitators to implementing the model and use quantitative data to measure interim outcomes, such as changes in care processes and patient experience, that can help identify areas for refinement and the potential to improve outcomes.
How can I use the findings? Findings from implementation evaluations can indicate whether it is feasible for practices to implement the intervention and ways to improve the intervention. Integrating the implementation and impact findings (if you can conduct an impact evaluation) can (1) provide a more sophisticated understanding about the effects of the model being tested; (2) identify types of patients, practices, and settings that may benefit the most; and (3) guide decisions about refinement and spread.
What resources are available to help me? The resource collection in this Guide contains resources and tools that you can use to develop a logic model, select implementation and outcome measures, design and conduct analyses, and synthesize implementation and impact findings.
I. Do I Need an Evaluation?
Your organization has decided to try to change the way primary care practices deliver care, in the hope of improving important outcomes. The first question to ask is whether you should evaluate the intervention.
Not all primary care interventions require an evaluation. When it is clear that a change needs to be made, the practice may simply move to adoption. For example, if patients are giving feedback about lack of evening hours, and business is being lost to urgent care centers, then a primary care practice might decide to add evening hours without evaluating the change. You may still want to track utilization and patient feedback about access, but a full evaluation of the intervention may not be warranted. In addition, some operational and quality improvement changes can be assessed through locally managed Plan-Do-Study-Act cycles. Examples of such changes include changing appointment lengths and enhancing educational materials for patients. Finally, when previously published studies have provided conclusive evidence in similar settings with similar populations, you do not need to re-test those interventions.
A more rigorous evaluation may be beneficial if it is costly to adopt the primary care intervention and if your organization is considering whether to spread the intervention extensively. An evaluation will help you learn as much as possible about how best to implement the intervention and how it might affect outcomes. You can examine whether it is possible for practices to make the changes you want, how to roll out this (or a refined intervention) more smoothly, and whether the changes made through the intervention sufficiently improve outcomes to justify the effort. You also may be able to ascertain how outcomes varied by practice, market, and patient characteristics. Outcomes of interest typically include health care cost and quality, and patient, clinician, and staff experience. Results from the implementation and impact analyses can help make a case for refining the intervention, continuing to fund it, and/or spreading it to more practices, if the effects of the intervention compare favorably to its costs.
Figure 1 summarizes the steps involved in planning and implementing an evaluation of a primary care intervention; the two boxes on the right-hand side show the evaluation’s benefits.
Figure 1. Steps in Planning and Implementing an Evaluation of a Primary Care Intervention
DO I NEED AN EVALUATION AND, IF SO, WHAT RESOURCES DO I NEED?
(see Section I andSection II)
Is the intervention worth evaluating?
Resources for a strong intervention:
Leadership buy-in, financial, technical assistance, tools, time
Resources for a strong evaluation:
Leadership buy-in, financial resources, research skills and expertise, data, time
HOW DO I PLAN AN EVALUATION?
(see Section III)
Consider the evaluation’s purpose and audience, and plan it at the same time the intervention is planned. What questions do you need to answer?
Understand key evaluation challenges
Keep your expectations realistic
Match the approach to your resources and data
Determine the logic underlying all components of intervention
How is the intervention being implemented?
How does A lead to B lead to C?
What are the intervention’s expected effects on cost; quality; and patient, clinician, and staff experience? When do you expect these to occur?
Which contextual factors, process indicators, and outcomes should you track and when?
Can you foresee any unintended consequences?
HOW DO I CONDUCT AN EVALUATION, AND WHAT QUESTIONS WILL IT ANSWER?
(see Section IV)
Design and conduct a study of implementation, considering burden and cost of each data source:
How and how well is intervention being implemented?
Identify implementation barriers and possible ways to remove them
Identify variations from plans used in implementation and why adaptations were made
Identify any unintended consequences
Refine intervention over time as needed
Design and conduct a study of impacts if there is sufficient statistical power:
Consider comparison group design, estimation methods, and samples for different data sources
Synthesize findings:
Do intervention activities appear to be linked to short-term or interim outcomes?
While results may not be definitive, do these measures point in the right direction?
Does intervention appear to result in changes in cost; quality; and patient, clinician, and staff experience(depending on evaluation’s length and comprehensiveness)?
HOW CAN I USE THE FINDINGS?
(see Section V)
Obtain evidence on what intervention did or did not achieve:
Who did the intervention serve?
How did the intervention change care delivery?
Best practices
Best staffing and roles for team members
How did implementation and impacts vary by setting and patient subgroups (if an impact analysis is possible)?
Findings may enable you to compare relative costs and benefits of this intervention to those of other interventions, if outcomes are similar.
Findings may help make a case for:
Continuing to fund intervention, with refinements
Spreading intervention to other settings
COMMON CHALLENGES IN EVALUATING PRIMARY CARE INTERVENTIONS
Timeframes are too short or intervention too minor to observe changes in care delivery and outcomes. Small numbers of practices make it hard to detect effects statistically due to clustering.
Data are limited, of poor quality, or have a significant time lag.
Results are not generalizable because practices participating in intervention are different from other practices (e.g., participants may be early adopters).
Outcomes may improve or decline for reasons other than participation in the intervention and the comparison group or evaluation design may not adequately account for this.
Differences exist between intervention practices and comparison practices even before the intervention begins. Comparison practices get some form or level of intervention.
II. What Do I Need for an Evaluation?
A critical step is understanding and obtaining the resources needed for successfully planning and carrying out your evaluation. The resources for conducting an intervention and evaluation are shown in Table 1 and Figure 1. We suggest you take stock of these items during the early planning phase for your evaluation. Senior management and others in your organization may need to help identify and commit needed resources.
The resources available for the intervention are linked to your evaluation because they affect (1) the extent to which practices can transform care and (2) the size of expected effects. How many practices can be transformed? How much time do staff have available to implement the changes? What payments, technical assistance to guide transformation, and tools (such as shared decision making aids or assistance in developing patient registries) will practices receive? Are additional resources available through new or existing partnerships? Is this intervention package substantial enough to expect changes in outcomes? Finally, how long is it likely to take practices to change their care delivery, and for these changes to improve outcomes?
Inventory the financial, research, and data resources you can devote to the evaluation, and adjust your evaluation accordingly.
Similarly, the resources available for your evaluation of the intervention help shape the potential rigor and depth of the evaluation. You will need data, research skills and expertise, and financial resources to conduct an evaluation. Depending on the skills and expertise available internally, an organization may identify internal staff to conduct the evaluation, or hire external evaluators to conduct the evaluation or collaborate and provide guidance on design and analysis. External evaluators often lend expertise and objectivity to the evaluation. Regardless of whether the evaluation is conducted by internal or external experts or a combination, ongoing support for the evaluation from internal staff—for example, to obtain claims data and to participate in interviews and surveys—is critical. The amount of time available for the evaluation will affect the outcomes you can measure, due to the time needed for data collection, as well as the time needed for outcomes to change.
Table 1. Inventory of Resources Needed for Testing a Primary Care Intervention
Resource Type
Examples
Resources for Intervention
Leadership buy-in
Motivation and support for trying the intervention.
Financial resources
Funding available for the intervention (including the number of practices that can test it).
Technical assistance
Support available to help practices transform such as data feedback, practice facilitation/coaching, expert consultation, learning collaboratives, and information technology (IT) expertise.
Tools
Tools for practices such as registries, health IT, and shared decision making tools.
Time
Allocated time of staff to implement the intervention; elapsed time for practices to transform and for outcomes to change.
Resources for Evaluation
Leadership buy-in
Motivation and support for evaluating the intervention.
Financial resources
Funding available for the evaluation, including funds to hire external evaluation staff if needed.
Research skills, expertise, and commitment
Skills and expertise in designing evaluations, using data, conducting implementation and impact analyses, and drawing conclusions from findings.
Motivation and buy-in of evaluation staff and other relevant stakeholders, such as clinicians and staff who will provide data.
Expertise in designing the evaluation approach and analysis plan, creating files containing patient and claims data, and conducting analyses.
Data
Depending on the research questions, could include claims, electronic medical records, paper charts, patient intake forms, care plans, patient surveys, clinician and practice staff surveys, registries, care management tracking data, qualitative data from site visit observations and interviews, and other information (including the cost of implementing the intervention). Data should be of adequate quality.
Time
Time to obtain and analyze data and for outcomes to change.
III. How Do I Plan an Evaluation?
Develop the evaluation approach before the pilot begins.
Start planning your evaluation as early as you can. Careful and timely planning will go a long way toward producing useful results, for several reasons. First, you may want to collect pre-intervention data and understand the decisions that shaped the choice of the intervention and the selection of practices for participation. Second, you may want to capture early experiences with implementing the intervention to understand any challenges and refinements made. Finally, you may want to suggest minor adaptations to the intervention’s implementation to enhance the evaluation’s rigor. For example, if an organization wanted to implement a PCMH model in five practices at a time, the evaluator might suggest randomly picking the five practices from those that meet eligibility criteria. This would make it possible to compare any changes in care delivery and outcomes of the intervention practices to changes in a control group of eligible practices that will adopt the PCMH model later. If the project had selected the practices before consulting with the evaluator, the evaluation might have to rely on less rigorous non-experimental methods.
Consider the purpose and audience for the evaluation.
Who is the audience for the evaluation?
What questions do they want answered?
Identifying stakeholders who will be interested in the evaluation’s results, the decisions that your evaluation is expected to inform, and the type of evidence required is crucial to determining what questions to ask and how to approach the evaluation. For example, insurers might focus on the intervention’s effects on the total medical costs they paid and on patient satisfaction; employers might be concerned with absentee rates and workforce productivity; primary care providers might focus on practice revenue and profit, quality of care, and staff satisfaction; and labor unions might focus on patient functioning, satisfaction, and out-of-pocket costs. Potential adverse effects of the intervention and the reporting burden from the evaluation should be considered as well.
Consider, too, the form and rigor of the evidence the stakeholders need. Perspectives differ on how you should respond to requests for information from funders or other stakeholders when methodological issues mean you cannot be confident in the findings. We recommend deciding during the planning stage how to approach and reconcile trade-offs between rigor and relevance. Sometimes the drawbacks of a possible evaluation—or certain evaluation components—are serious enough (for example, if small sample size and resulting statistical power issues will render cost data virtually meaningless) that resources should not be used to generate information that is likely to be misleading.
Questions to ask include: Do the stakeholders need numbers or narratives, or a combination? Do stakeholders want ongoing feedback to refine the model as it unfolds, an assessment of effectiveness at the end of the intervention, or both? Do the results need only to be suggestive of positive effects, or must they rigorously demonstrate robust impacts for stakeholders to act upon them? How large must the effects be to justify the cost of the intervention? Thinking through these issues will help you choose the outcomes to measure, data collection approach, and analytic methods.
Understand the challenges of evaluating primary care interventions. Some of the challenges to evaluating primary care interventions include (see also the bottom box of Figure 1):
▲ Time and intensity needed to transform care. It takes time for practices to transform, and for those transformations to alter outcomes.1 Many studies suggest it will take a minimum of 2 or 3 years for motivated practices to really transform care.2, 3, 4, 5 ,6 If the intervention is short or it is not substantial, it will be difficult to show changes in outcomes. In addition, a short or minor intervention may only generate small effects on outcomes, which are hard to detect.
▲ Power to detect impacts when clustering exists. Even with a long, intensive intervention, clustering of outcomes at the practice levela may make it difficult for your evaluation to detect anything but very large effects without a large number of practices. For example, some studies spend a lot of time and resources collecting and analyzing data on the cost effects of the PCMH model. However, because of clustering, an intervention with too few practices might have to generate cost reductions of more than 70 percent for the evaluation to be confident that observed changes are statistically significant.7 As a result, if an evaluation finds that the estimated effects on costs are not statistically significant, it’s not clear whether the intervention was ineffective or the evaluation had low statistical power (see Appendix A for a description of how to calculate statistical power in evaluations).
▲ Data collection. Obtaining accurate, complete, and timely data can be a challenge. If multiple payers participate in the intervention, they may not be able to share data; if no payers are involved, the evaluator may be unable to obtain data on service use and expenditures outside the primary care practice.
▲ Generalizability. If the practices that participate are not typical, the results may not be generalizable to other practices.
▲ Measuring the counterfactual. It is difficult to know what would have occurred in the intervention practices in the absence of the intervention (the “counterfactual”). Changing trends over time make it hard to ascribe changes to the intervention without identifying an appropriate comparison group, which can be challenging. In addition, multiple interventions may occur simultaneously or the comparison group may undertake changes similar to those found in the intervention, which can complicate the evaluation.
Adjust your expectations so they are realistic, and match the evaluation to your resources. The goal of your evaluation is to generate the highest quality information possible within the limits of your resources. Given the challenges of evaluating a primary care intervention, it is better to attempt to answer a narrow set of questions well than to study a broad set of questions but not provide definitive or valid answers to any of them. As described above, you need adequate resources for the intervention and evaluation to make an impact evaluation worthwhile.
With limited resources, it is often better to scale back the evaluation. For example, an evaluation that focuses on understanding and improving the implementation of an intervention can identify early steps along the pathway to lowering costs and improving health care. We recommend using targeted interviews to understand the experiences of patients, clinicians, staff, and other stakeholders, and measuring just a few intermediate process measures, such as changes in workflows and the use of health information technology. Uncovering any challenges encountered with these early steps can allow for refinement of the intervention before trying out a larger-scale effort.
The evaluator and implementers should work together to describe the theory of change. The logic model will guide what to measure, and when to do so.
Describe the logic model, or theory of change, showing why and how the intervention might improve outcomes of interest. In this step, the evaluators and implementers work together to describe each component of the intervention, the pathways through which they could affect outcomes of interest, and the types of effects expected in the coming months and years. Because primary care interventions take place in the context of the internal practice and the external health care environments, the logic model should identify factors that might affect outcomes—either directly or indirectly—by affecting implementation of the intervention. Consider all factors, even if you may not be able to collect data on all of them, and you may not have enough practices to control for each factor in regression analyses to estimate impacts. Practice- or organization-specific factors include, for example, patient demographics and language, size of patient panels, practice ownership, and number and type of clinicians and staff. Other examples of practice- specific factors include practice leadership and teamwork.8 Factors describing the larger health care environment include practice patterns of other providers, such as specialists and hospitals, community resources, and payment approaches of payers. Intervention components should include the aspects of the intervention that vary across the practices in your study, such as the type and amount of services delivered to provide patient-centered, comprehensive, coordinated, accessible care, with a systematic focus on quality and safety. They may also include measures capturing variation across intervention practices in the offer and receipt of: technical assistance to help practices transform; additional payments to providers and practices; and regular feedback on selected patient outcomes, such as health care utilization, quality, and cost metrics.
Find resources on logic models and tools to conduct implementation and impact studies in theResource Collection.
A logic model serves several purposes (see Petersen, Taylor, and Peikes9 for an illustration and references). It can help implementers recognize gaps in the logic of transformation early so they can take appropriate steps to modify the intervention to ensure success. As an evaluator, you can use the logic model approach to determine at the outset whether the intervention has a strong underlying logic and a reasonable chance of improving outcomes, and what effect sizes the intervention might need to produce to be likely to yield statistically significant results. In addition, the logic model will help you decide what to measure at different points in time to show whether the intervention was implemented as intended, improved outcomes, and created unintended outcomes, and identify any facilitators and barriers to implementation. However, while the logic model is important, you should remain open to unexpected information, too. Finally, the logic model’s depiction of how the intervention is intended to work can be useful in helping you interpret findings. For example, if the intervention targets more assistance to practices that are struggling, the findings may show a correlation between more assistance and worse outcomes. Understanding the specifics of the intervention approach will prevent you from mistakenly interpreting such a finding as indicating that technical assistance worsens outcomes.
As an example of this process, consider a description of the pathway linking implementation to outcomes for expanded access, one of the features the medical home model requires. Improved access is intended to improve continuity of care with the patient’s provider and reduce use of the emergency room (ER) and other sites of care. If the intervention you are evaluating tests this approach, you could consider how the medical home practices will expand access. Will the practices use extended hours, email and telephone interactions, or have a nurse or physician on call after hours? How will the practices inform patients of the new options and any details about how to use them? Because nearly all interventions are adapted locally during implementation, and many are not implemented fully, the logic model should specify process indicators to document how the practices implemented the approach. For practices that use email interactions to increase access, some process indicators could include how many patients were notified about the option by mail or during a visit, the overall number of emails sent to and from different practice staff, the number and distribution by provider and per patient, and time spent by practice staff initiating and responding to emails (Figure 2). You could assess which process indicators are easiest to collect, depending on the available data systems and the feasibility of setting up new ones. To decide which measures to collect, consider those likely to reflect critical activities that must occur to reach intended outcomes, and balance this with an understanding of the resources needed to collect the data and the impact on patient care and provider workflow.
Figure 2. Logic Model of a PCMH Strategy Related to Email Communication
Source: Adapted from Petersen, Taylor, and Peikes. Logic Models: The Foundation to Implement, Study, and Refine Patient-Centered Medical Home Models, 2013, Figure 2.9
Expected changes in intermediate outcomes from enhanced email communications might include the following:
▲ Fewer but more intensive in-person visits as patients resolve straightforward issues via email
▲ Shorter waits for appointments for in-person visits
▲ More continuity of care with the same provider, as patients do not feel the need to obtain visits with other providers when they cannot schedule an in-person visit with their preferred provider
Ultimate outcomes might include:
▲ Patients reporting better access and experience
▲ Fewer ER visits as providers can more quickly intervene when patients experience problems
▲ Improved provider experience, as they can provide better quality care and more in-depth in-person visits
▲ Lower costs to payers from improved continuity, access, and quality
You should also track unintended outcomes that program designers did not intend but might occur. For example, using email could reduce practice revenue if insurers do not reimburse for the interactions and fewer patients come for in-person visits. Alternatively, increased use of email could lead to medical errors if the absence of an in-person visit leads the clinician to miss some key information, or could lead to staff burnout if it means staff spend more total time interacting with patients.
Some contextual factors that might influence the ability of email interactions to affect intended and unintended outcomes include: whether patients have access to and use email and insurers reimburse for the interactions; regulations and patient concerns about confidentiality, privacy, and security; and patient copays for using the ER.
What outcomes are you interested in tracking? The following resources are a good starting point for developing your own questions and outcomes to track:
▲ The Commonwealth Fund’s PCMH Evaluators Collaborative provides a list of core outcome measures.www.commonwealthfund.org/Publications/Data-Briefs/2012/May/Measures-Medi...
▲ The Consumer Assessment of Healthcare Providers and Systems (CAHPS®) PCMH survey instrument developed by the Agency for Healthcare Research and Quality (AHRQ) provides patient experience measures. https://cahps.ahrq.gov/surveys-guidance/cg/pcmh/index.html.
IV. How Do I Conduct an Evaluation, and What Questions Will It Answer?
See the Resource Collection for res