Evaluation

Q20: What is the Prescription for Health Analysis Team and what is its role and function?

A: RWJF separately funds an Analysis Team (A-Team) to assess the program’s progress in developing and applying novel, practical approaches to promoting healthy behaviors in primary care practices. Under the leadership of Dr. Deborah Cohen, the A-Team will facilitate the learning of overarching lessons that transcend any single project and provide regular feedback to each funded PBRN that can be used to enhance projects and evaluate their outcomes. As a condition of accepting grant funds, all funded PBRNs must commit to work with the A-Team and be responsible for collecting the data relevant to its cross-site analysis.

Q21: What evaluations are we expected to do ourselves, and what is the responsibility of the independent evaluation team?

A: Evaluation of project-specific outcomes is the responsibility of each funded project. Each proposal must include an evaluation strategy written in such a way that it aligns with the RE-AIM framework, and stands independently from the cross-site evaluation conducted by the Prescription for Health Analysis Team.

Required data collection for the cross-site evaluation will include:

Practice-Level Survey Data

• At baseline and at the end of the study, projects are required to complete a Practice Information Form (PIF) for each practice participating in their project. The PIF is completed by PBRN staff in collaboration with the office manager and/or physician manager of each participating practice. PIF data include key characteristics of the practice as well as a list of all clinicians and staff, their roles, length of service, and hours worked per week.
  

Online Diaries

• A confidential “room” will be established for each project on a secure web site for posting project updates and for “real time” correspondence with the Analysis Team. Each project is required to identify investigators and staff who will post regular diary entries. Following notification of funding, the Analysis Team will work with each project to identify those members who will make the diary entries. Members of the team will read diary postings as they are entered and will respond, posing questions and identifying areas for clarification. These online diaries help document the process and context of each project’s implementation. Additionally, as each project has access to its own diary, diaries also serve as a resource to projects as they evaluate their initiatives.
  

A-Team members will perform ongoing analyses of the diary data as a means of capturing what is being learned by individual projects and by Prescription for Health projects collectively. The team will summarize PIF, site visit and diary data to create a detailed confidential report that will be shared with the principal investigator of each project. While project-specific insights will be shared only with individual principal investigators, overall learning from the Prescription for Health initiative will be shared among all projects, the NPO and RWJF.

Q22: What kind of technical assistance will be provided for evaluation?

A: The independent evaluation team will provide technical assistance in collecting ongoing process data and maintaining a project diary. Technical assistance related to data collection to meet project goals will be provided by the NPO and PBRN Resource Center.

Q23: What common measures/cross-site data will be collected and how?

A: In addition to the data that will be collected by the A-Team , a common set of behavioral measures (link to The Grant Program under “About Program”) will be used in Round 2 of the program. The process for the collection and analysis of these data will be discussed with grantees early on in the program.

Q24: Why are you recommending the common patient-oriented measures when some of them have not been widely validated and lack scoring and cut-point standards? How should applicants proceed in light of these limitations?

A: The criteria and rationale for selection of common measures are explained in “Practical and Relevant Self-Report Measures of Patient Health Behaviors for Primary Care Research,” by Glasgow et al, Annals of Family Medicine 2005;3:73-81. The goal was to derive a set of measures that would be practically achievable and appropriate for most primary care settings. The limitations of candidate measures necessitated judgments that will not be universally agreeable. Nonetheless, Prescription for Health (P4H) concluded that the advantages of selecting common measures outweighed the disadvantages. Applicants are strongly encouraged to use the common measures but are not precluded from including additional measures. All applicants are equally advantaged and disadvantaged by the strengths and weaknesses of the recommended measures. Recognizing that investigators might have important questions and strategies with which the recommended measures might not comport, the Call for Proposals (CFP) establishes the option of carefully explaining why the common measures cannot be used and why others must be. When measures lack key information needed for power calculations, an effect size can be used or other approaches taken to cope with uncertainty. As you probably know, for this approach, all you have to do is estimate small, medium or large effect sizes (which should be possible to do from the general literature on behavior change interventions, even if you don't have effect sizes for these specific measures).

Q25: Is scoring information available for the recommended patient measures "Starting the Conversation - Diet" and "Physical Activity and Nutrition Behaviors Monitoring Form"?

A: We understand that some of you are aware of different/longer variations of these instruments and that their developers intended them primarily to guide counseling and monitor population changes, which is causing confusion about which instruments we are recommending and for what purposes. P4H’s official recommendation is for their use as measurement tools as described in Figure 1, and Table 2 of Glasgow RE, Ory MG, Klesges LM, Cifuentes M, Fernald DH, Green LA. Practical and Relevant Self-Report Measures of Patient Health Behaviors for Primary Care Research. Ann Fam Med 2005; 3:73-81.

The most recent versions of these instruments will be reassessed by the NPO and its expert consultants before the July 2005 meeting. Should the recommendations change, we will notify you as soon as possible.

In the article, you are directed to contact Dr. Alice Ammerman for scoring information on the recommended adult eating patterns instrument “Starting the Conversation – Diet” and the recommended adolescent eating patterns instrument “Physical Activity and Nutrition Behaviors Monitoring Form”. So that multiple requests to Dr. Ammerman can be minimized, we have obtained any/all the scoring information available at this time on these instruments and are including it below for your reference.

Scoring the STC tool (taken from communication with Alice Ammerman)

We have a very simple scoring system for the STC tools. Any answer in the far right column for any of the questions is scored 2, the middle column is 1 and the left column is O. Therefore the higher the score, the more atherogenic/unhealthy the diet. We validated the longer DRA using this scoring system against the Keys equation (ref below) and are in the process of comparing STC scores with longer diet assessments and biomarkers.

Ammerman A, Haines P, DeVellis R, Strogatz D, Keyserling T, Simpson R, Siscovick D. A brief dietary assessment to guide cholesterol reduction in low income individuals: design and validation. JADA 1991;91:1385-1390.

Scoring the Physical Activity and Nutrition Behaviors Monitoring Form (taken from communication with Janice Sommers)

No Scoring information available at this time.

I'm sorry to say that at this point in time we do not have a scoring system for the PAN form. The tool was developed as an enhancement to the NC Nutrition and Physical Activity Surveillance System (NC-NPASS) to monitor population changes in key nutrition and physical activity behaviors that would assist community and state public health leaders with program planning and evaluation. There is some interest in developing one but not sure when that might happen.

^ top of page