Evaluation of the Research to Policy Collaboration Model

Child Abuse Clinical Trial

— RPC  

Official title:

Testing an Approach to Improve the Use of Evidence

Clinical Trial Details — Status: Completed

Administrative data

• NCT number: NCT03671434
• Other study ID #: 00010061
• Status: Completed
• Phase: N/A
• Start date: January 8, 2019
• Est. completion date: February 17, 2021

Study information

• Verified date: October 2021
• Source: Penn State University
• Contact: n/a
• Is FDA regulated: No
• Study type: Interventional

Clinical Trial Summary

This work aims to evaluate an approach for improving federal legislators' use of evidence-known as the Research-to-Policy Collaboration (RPC) - which seeks to address known barriers to policymakers' use of research, including a lack of personal contact between researchers and policymakers and limited relevance of research translation efforts to current policy priorities. The RPC involves structured processes for identifying policymakers' priorities, building researchers' capacity for nonpartisan responses to current policy priorities, and facilitating ongoing and productive researcher-policymaker interactions. This implementation of the RPC will focus on child and family policies relevant to child maltreatment.

This study assesses both processes for collaboration and policymakers' use of research within a randomized controlled trial (RCT) employing a mixed methods approach-including quantitative and qualitative evaluation of impact. The proposed project will be guided by three overarching questions:

1. How does the RPC impact researchers and legislative staff?

2. How does the RPC impact legislative activity?

3. How might perceptions and experiences of collaboration through the RPC relate to different forms of evidence use among researchers and policymakers?

The RPC's effectiveness will be tested through experimental design (randomization) using qualitative and quantitative assessments of researcher-policymaker interactions and impact. This includes surveying congressional staff and researchers, reviewing records of policymaker's public statements and introduced legislation, and conducting qualitative interviews around researchers' and legislative staffs' experiences with researcher-policymaker collaboration prior to and during the RPC.

Description:

Objective: The aim of this study is to examine the effectiveness of a model for supporting policymakers' use of research evidence within a randomized controlled trial (RCT) employing both qualitative and quantitative analyses of survey, interview, and observational data to understand the use of research in policy and collaboration. Legislators' research use will be assessed with multiple methods, including (1) self-report via a survey protocol, (2) legislative activity (i.e., introduced bills and official statements), and (3) qualitative interviews. Additionally, interviews and surveys will assess researchers' reported efficacy, skill and engagement in public policy, and the extent to which their work is informed by the policy process.

Intervention: The Research-to-Policy Collaboration (RPC) builds upon theoretical and empirical literature that suggests the need to (a) translate research relevant to current policy priorities, and (b) facilitate productive interactions between policymakers and researchers. The RPC is a nonpartisan, manualized model implemented by an intermediary organization that cultivates relationships between researchers and legislative offices. Implementation occurs in two phases involving a series of seven interrelated activities. During the capacity-building phase, the RPC simultaneously aims to support researcher's development of honest knowledge broker skills with training and coaching, while also conducting iterative needs assessments with congressional offices regarding their current priorities and desire for research evidence. In the collaboration phase, researchers with relevant expertise are coalesced into rapid response teams that are matched with legislative offices. Through a scaffolded series of interactions, the RPC model is used to cultivate productive researcher-policymaker relationships and support responses to legislative requests. Evaluating the effectiveness of the RPC is the focus of the current study.

Study Design:

This study employs both qualitative and quantitative analyses of survey, interview, and observational data within the context of an RCT. Three overarching research questions guide this work and align with the study's three different components - self-report via a quantitative survey, observation of research use via coded legislative activity, and ethnography via in-depth interviews and participant observations.

1. How does the RPC impact researchers and legislative staff? (Self-Report)

1. Do researchers report improved policy competencies and motivation for conducting policy-relevant research?

2. Will legislative staff report increases in positive attitudes toward, knowledge of, intended application, and actual use of research evidence?

3. Might the impact on legislative and research participants will vary as a function of aspects of their collaborative relationship (e.g., enhanced by mutual trust)?

2. How does the RPC impact legislative activity? (Observation)

a) Will congressional offices increase their use of research evidence in legislative activities (i.e., bills, official statements)?

3. How might perceptions and experiences of collaboration through the RPC relate to different forms of evidence use among researchers and policymakers? (Ethnography)

1. What are the barriers and facilitators of participation in the RPC?

2. What types of interactions between researchers and policymakers does the RPC influence?

3. How does the RPC affect participants' assumptions about each other, interactions with each other, and their intentions?

4. What types of evidence use are supported by RPC participation?

Sampling Framework All study components draw from an initial sampling framework through which researchers and congressional staff who opt into participating in the RPC will be asked to participate in the study; those who agree to participate in the study will be randomized for either the full RPC intervention or a light-touch control group that receives little direct engagement. This study will oversample to combat anticipated attrition. The final sample includes 96 congressional offices (48 in RPC, 48 in control group) and 226 researchers (151 in RPC, 75 in control group). These samples were derived from an identified population of eligible intervention participants.

• A population of relevant legislative offices (Senate and House) will be identified based on the number of child/family bills legislator have (co)sponsored. The random selection process will be stratified to ensure proportional partisan representation and representation of legislators who range in activity related to child welfare. Prior to randomization, the population of relevant legislators will be assigned an Activity Ranking based on the number of times a legislator has sponsored a child/family policy bill. Activity Rankings will range from 1-10 such that 10% of legislators who (co)sponsored the fewest bills will be ranked 1, and 10% of legislators who co-sponsored the most bills will be ranked 10. Recruitment and stratified sampling across a continuum of activity on child/family policy will improve bipartisan representation of the sample. Also aiding with bipartisan representation are targeted outreach efforts that intensify along party lines if participation becomes skewed.

• Researchers who work on relevant child/family research will be identified via research institutions, professional networks and listservs, literature searches, and referrals from other researchers. Researchers include research-oriented practitioners and program evaluators from both academic and non-academic settings who are interested in translating their expertise with policymakers. Subsequent to opting-in to participate in a policy engagement intervention, researchers are asked to participate in the study. Study participants will be randomized to receive the RPC or the control condition.

Study Component 1: Quantitative Evaluation of Self-Reported RPC Impact A structured survey will be used to assess researchers' capacity for engaging in public policy processes. A corresponding survey protocol will assess legislative staff's attitudes, knowledge, intent, and reported behavior regarding the use of research evidence. Both researchers and legislative staff will be asked about the nature of their collaborative interactions with one another. Surveys will be conducted approximately every three months. Survey constructs are described in outcome measures. Missing data will be handled with multiple imputation because this is best practice when data are not missing at random. Multilevel modelling will be used to examine individual trajectories in outcome variables. Multiple regression will examine changes between select time points; for instance, whether researchers' reported policy skills improve between baseline and subsequent to trainings. These analyses will allow us to examine whether RPC participation is associated with increases in (1) researchers' reported efficacy to engage with policymakers, (2) legislative staff's interest and willingness to use research evidence, and (3) both researchers' and policymakers' positive sentiment toward working together. Longitudinal analyses will further allow us to explore the trajectory of outcome variables over time (e.g., improved researcher competencies, legislators' use of research)-specifically modelling change in evidence use and other outcomes of interest. Analyses of researcher and legislative staff outcomes are all powered at least at a 90% level to detect effect sizes of 0.2 or greater (power analysis conducted with Optimal Design plus Empirical Evidence).

Study Component 2: Quantitative Evaluation of Observed RPC Impact on Legislative Activity To complement the self-reported quantitative assessment of change, use of research evidence in legislative activities will be observed before and following RPC participation. Data will be selected, extracted, and coded prior to quantitative analyses with indicators derived from deductive codes. Legislative data will be selected based upon the use of keywords indicating relevance to child/family. Since each bill will comprise more text than is feasible to code with traditional qualitative methods, relevant provisions will be extracted prior to coding (those containing keywords). Further, all subsection titles for each sampled bill will be reviewed for potential relevance and extracted if research evidence is used but not captured otherwise by keywords. Data on legislative activity will be coded based on a detailed codebook informed by prior findings of how research has been used by policymakers and in legislation. This coding scheme will be applied by two coders and the codebook will be revised and further specified until the intercoder reliability reaches at least .70 and 30% of the data for the first time period has been coded. Subsequently, trained coders will code individually and a lead investigator will randomly sample 10% of these to ensure consistency in coding quality. These codes will be converted into quantitative indicators of research use, which are described in outcome measures. Legislative activity will be assessed annually. Multivariate Poisson regression analyses will assess change in research use in thematic codes of (1) bills and (2) official statements between the RPC intervention and control conditions. Moderation analyses will further explore the extent to which changes in legislative activity depend on the nature of collaboration experiences, other survey data (e.g., policymakers' perceptions of the value of research), and legislator characteristics (e.g., party affiliation, years of age).

Study Component 3: Ethnographic Evaluation of the RPC model The qualitative component of this study is closely linked with the quantitative work and seeks to better understand the experiences of RPC participants and processes underlying the use of research evidence in the RPC context. Using participatory and ethnographic methods, data will be collected from two sources: (1) semi-structured interviews of trial participants and RPC staff, and (2) observations of researcher trainings and researcher-staff meetings. Interview participants will be sampled from the pool of study participants randomly assigned to receive the RPC, including 11 RPC researchers and 11 RPC congressional office staff that will be interviewed prior to their involvement in the RPC, and 11 RPC researchers and 11 congressional office staff that will be interviewed following collaboration activities via the RPC. Additionally, 3 RPC staff will be interviewed prior to and following the RPC. To gain additional insight into the RPC operations and impact, observations will be conducted of trainings for researchers, formal meetings and informal interactions between researchers and congressional staff, and meetings that occur as part of any RPC collaboration. Following ethnographic best practices, "open-observations" of key meetings and training events will be conducted to enrich the interviews with more information about how attendees interact and how they respond to the RPC. Field notes and transcripts will be analysed with thematic analysis and an interpretive policy analysis that explores different rationalities and rhetoric when discussing evidence used in the collaboration.

Recruitment information / eligibility

• Status: Completed
• Enrollment: 322
• Est. completion date: February 17, 2021
• Est. primary completion date: February 17, 2021
• Accepts healthy volunteers: Accepts Healthy Volunteers
• Gender: All
• Age group: 18 Years and older

Eligibility

Inclusion Criteria:

• Participants who voluntarily enlist in the RPC will be asked to participate in the trial.

Exclusion Criteria:

• Participants who choose to stop participating in the study or the RPC itself. All study participants can choose to opt-out of the study at any time.

Related Conditions & MeSH terms

• Child Abuse
• Family and Household
• Legislation
• Policy

Intervention

Behavioral:
• Research-to-Policy Collaboration
The RPC is a behavioral intervention through which RPC Researchers and RPC Congressional Offices are prepared and matched for collaboration. Specifically, congressional offices are asked to identify opportunities for researcher engagement in policy efforts, researchers with expertise related to policy opportunities are identified and prepared to collaborate with congressional offices, researchers and congressional staff are matched for ongoing collaborative partnerships, and both researchers and congressional staff receive ongoing support to facilitate research translation.
• Light Touch Policy Training
Control Researchers are provided information on policy engagement via email.

Locations

Country	Name	City	State
United Kingdom	London School of Hygiene & Tropical Medicine	London
United States	Child Trends	Bethesda	Maryland
United States	Penn State University	University Park	Pennsylvania

Sponsors (5)

Lead Sponsor	Collaborator
Penn State University	Child Trends, London School of Hygiene and Tropical Medicine, Penn State Social Science Research Insititute, William T. Grant Foundation

Countries where clinical trial is conducted

United States,  United Kingdom, 

References & Publications (2)

Crowley M, Scott JTB, Fishbein D. Translating Prevention Research for Evidence-Based Policymaking: Results from the Research-to-Policy Collaboration Pilot. Prev Sci. 2018 Feb;19(2):260-270. doi: 10.1007/s11121-017-0833-x. — View Citation

Scott JT, Larson JC, Buckingham SL, Maton KI, Crowley DM. Bridging the research-policy divide: Pathways to engagement and skill development. Am J Orthopsychiatry. 2019;89(4):434-441. doi: 10.1037/ort0000389. — View Citation

Outcome

Type	Measure	Description	Time frame	Safety issue
Primary	Change in Reported Use of Evidence	Congressional study participants will be asked how often they have accessed research and used research in decision-making processes in the past 3 months.	Assessed multiple times through study completion, an average of one year.
Primary	Change in Evidence Sources	Congressional study participants will be asked what sources (e.g., personal contacts, academic journals) use to obtain evidence on policy issues.	Assessed multiple times through study completion, an average of one year.
Primary	Change in Interactions with Researchers	Congressional study participants will be asked how often they have interacted with researchers in different settings in the past 3 months.	Assessed multiple times through study completion, an average of one year.
Primary	Change in Attitudes and Readiness Regarding Evidence Use	Congressional study participants will be asked how valuable research is perceived by the staff and within the congressional office as a whole, as well as perceived benefit of social science specifically.	Assessed multiple times through study completion, an average of one year.
Primary	Change in Policy Engagement	Researcher study participants will be asked how frequently they have engaged with policymakers in different activities and at different stages of the policy process.	Assessed multiple times through study completion, an average of one year.
Primary	Change in Policy-Informed Research	Researcher study participants will be asked their perceived value in engaging policymakers in the research process, the extent to which their research activities are informed or guided by policymakers' needs, and the extent to which researchers have actively engaged policymakers in the research process in the last 3 months.	Assessed multiple times through study completion, an average of one year.
Primary	Change in Policy-Related Self-Efficacy	Researcher study participants will be asked how confident and prepared they feel about engaging with policymakers.	Assessed multiple times through study completion, an average of one year.
Primary	Change in Reported Policy Knowledge	Researcher study participants will be asked the extent to which they understand policy processes and norms, and the degree to which they perceive a need for additional policy training.	Assessed multiple times through study completion, an average of one year.
Primary	Change in Official Statement Research Use	Legislators' public statements will be dichotomously coded to indicate any type of research use. These statement-level indicators will be aggregated by legislator.	Bills that were introduced one year prior to the RPC implementation, during the implementation period and one year following the RPC completion.
Primary	Change in Research Use in Legislation	Child/Family bills will be dichotomously coded to indicate any type of research use. These statement-level indicators are then aggregated by legislator.	Bills that were introduced one year prior to the RPC implementation, during the implementation period and one year during or following the RPC completion.
Primary	Change in Intensive Research Use in Legislation	Each section in child/family bills will be dichotomously coded to indicate any type of research use. Each bill will be scored to indicate a proportion of sections in which evidence is used out of the sum of sections in the bill. These statement-level indicators are then aggregated by legislator.	Bills that were introduced one year prior to the RPC implementation, during the implementation period and one year during or following the RPC completion.
Primary	Change in Type of Use in Legislation	Child/family bills will be dichotomously coded to indicate the observation of different types of research use (e.g., conceptual, instrumental). These statement-level indicators are then aggregated by legislator.	Bills that were introduced one year prior to the RPC implementation, during the implementation period and one year during or following the RPC completion.
Secondary	Satisfaction with Collaboration	Both researcher and congressional study participants will be asked about the quality of the collaborative interactions, such as the adequacy of communication and working styles of fellow collaborators.	Assessed during the collaboration component of the intervention or up to one year after study initiation.
Secondary	Perceived Value of the Partnership	Both researcher and congressional study participants will be asked the extent to which the partnership was valuable in relation to time investment and would be worth recommending or doing again.	Assessed during the collaboration component of the intervention or up to one year after study initiation.
Secondary	Perceived Impact of the Collaboration	Both researcher and congressional study participants will be asked the extent to which the collaboration improved aspects of participants' work, mutual understanding, or addressed needs and common goals.	Assessed during the collaboration component of the intervention or up to one year after study initiation.
Secondary	Trust and Respect	Both researcher and congressional study participants will be asked about their perceptions of the quality of their interactions with collaborative partners, including comfort, trust, shared commitment, and conflict management.	Assessed during the collaboration component of the intervention or up to one year after study initiation.

External Links

•   Describes the intervention