A How-To Guide for the RCEM QI Assessment
Executive Summary
This briefing document provides a comprehensive guide to the Royal College of Emergency Medicine (RCEM) Quality Improvement (QI) assessment, a core component of the 2021 curriculum. Specialty Learning Outcome (SLO) 11, "Participate in and promote activity to improve the quality and safety of patient care," represents a fundamental shift from a single examination hurdle to a continuous, longitudinal assessment of QI and patient safety skills throughout training.
Critical Takeaways:
- Continuous, Longitudinal Assessment: QI is no longer an isolated event but is "spiralled" through training, requiring evidence of engagement at every stage (Core, Intermediate, and Higher). This approach, supported by educational principles of "interleaving" and "spacing," embeds QI into daily practice and prevents post-exam disengagement.
- The Quality Improvement Assessment Tool (QIAT): The primary mechanism for recording and assessing annual QI activity is the QIAT, a standardized reporting form available on the Kaizen ePortfolio. A well-constructed QIAT is detailed, typically covering the equivalent of 7–8 pages.
- Focus on Methodology: The assessment has shifted from traditional audit to a focus on mastering and applying systematic QI methodologies, such as the Model for Improvement (MFI) with Plan-Do-Study-Act (PDSA) cycles. The emphasis is on demonstrating an understanding of the improvement journey.
- Broadened Project Scope: QI projects are no longer limited to narrow clinical topics. Projects addressing education, environmental sustainability, staff wellbeing, cost-saving, pre-hospital care, or overseas settings are now suitable, significantly expanding the range of available topics.
- Progressive Competency: Expectations evolve with the trainee's level of seniority. Core trainees focus on participation and understanding basic principles; Intermediate trainees progress to data analysis and evaluation of change; and Higher trainees are expected to demonstrate project leadership.
- Assessment and Sign-Off: The QIAT is reviewed annually by the trainee's Educational Supervisor (ES). For trainees in Higher Specialist Training (ST4-6), there is additional oversight to ensure standards are met, which for some transitional trainees involves a regional QI panel.
--------------------------------------------------------------------------------
1. The Strategic Shift to SLO 11
The integration of QI throughout the RCEM 2021 curriculum aligns with a General Medical Council (GMC) mandate for all specialties, reflecting that quality and safety are core components of Good Medical Practice (1). The previous model, which isolated QI assessment to a single FRCEM examination, has been replaced to ensure these skills are continuously developed and applied.
From Quality Assurance to Quality Improvement
SLO 11 marks a significant move away from the traditional audit model of quality assurance towards a more dynamic quality improvement framework. While audits play a role in measuring compliance against standards, QI focuses on continuously improving processes to achieve higher quality care.
Feature | Quality Assurance (Traditional Audit) | Quality Improvement |
Motivation | Measuring compliance with standards | Continuously improving processes |
Means | Inspection | Prevention |
Focus | Outliers and "bad apples" | Processes, systems, and patient focus |
Method | Data collection, comparison to standard, re-audit after intervention | Iterative Plan-Do-Study-Act (PDSA) cycles, continuous measurement, and adaptation |
Timescale | Often cumbersome and slow | Responsive, with rapid feedback loops |
Responsibility | Held by a few individuals | Shared by all team members |
This new approach requires trainees to record and reflect upon a breadth of experience across different settings, identifying their own strengths and weaknesses in QI throughout their training journey.
2. The QI Journey: Competency Progression by Stage
Expectations for SLO 11 are "spiralled," meaning they increase in complexity as a trainee progresses. It is crucial to select and engage with projects commensurate with the current training stage.
Training Stage | Focus & Key Capability | Requirements & Expectations |
ACCS / Core Training (ST1-ST2) | Participation & Understanding <br> "…able to contribute effectively to a departmental quality improvement project" | • Demonstrate a basic understanding of key QI principles and methodology. <br> • Show evidence of engagement in a departmental audit or QI process for each year of training. <br> • Reflect on the team-based nature of QI work. |
Intermediate Training (ST3-ST4) | Data Analysis & Evaluation <br> "…able to describe their involvement and show an understanding of QI methods and reflect on a Quality Improvement Project they have been involved in" | • Participate in a QI project involving a multi-disciplinary team. <br> • Demonstrate additional data analysis and an evaluation of the changes made. <br> • Provide a satisfactory account of the QI methods used and reflection on the project's conduct. |
Higher Specialist Training (ST4-ST6) | Leadership & Completion <br> "…able to provide clinical leadership on effective Quality Improvement work" <br> "…able to support and develop a culture of departmental safety and good clinical governance" | • Lead on a QI project, ideally from inception to completion by the end of training. <br> • Provide evidence of multi-disciplinary team leadership and effective communication of results. <br> • Demonstrate perseverance in making change and insight into the challenges of change management. |
3. The QI Toolkit: Essential Methodologies and Tools
Successful QI work depends on the application of established methodologies. The curriculum expects trainees to move beyond simple data collection and demonstrate the use of recognized QI tools to analyze problems, develop interventions, and measure impact.
3.1. Identifying a Project and Analyzing the Problem
A QI project should start with a problem or issue, not a pre-determined solution. Common sources for identifying projects include audit data, patient feedback (complaints or compliments), incident reports, and direct observation of clinical practice. Once an area for improvement is identified, several tools can be used for analysis:
- Process Mapping: A visual representation of a process (e.g., patient flow) that enables the identification of redundant or inefficient steps.
- Ishikawa (Fishbone) Diagram: A visual tool for brainstorming and categorizing the potential causes of a problem to identify its root causes.
- Driver Diagram: A logic chart that helps define the project's aim and identifies the primary and secondary drivers (factors) that will contribute to achieving that aim. This tool links the overall goal to specific interventions.
- Stakeholder Analysis: A process of identifying key individuals or groups affected by the project and planning how to engage them effectively.
3.2. Key QI Methodologies
While numerous QI methodologies exist, the Model for Improvement (MFI) is one of the most common and effective frameworks for healthcare settings.
The Model for Improvement (MFI)
The MFI is based on answering three fundamental questions:
- What are we trying to accomplish? (This defines the aim).
- How will we know that a change is an improvement? (This defines the measures).
- What changes can we make that will result in an improvement? (This identifies potential interventions).
The MFI uses Plan-Do-Study-Act (PDSA) cycles to test changes on a small scale before wider implementation.
The PDSA Cycle
The PDSA cycle is the iterative engine of improvement:
- Plan: Define the objective, ask predictive questions, and plan the change to be tested (who, what, where, when).
- Do: Carry out the plan, document problems and unexpected observations, and begin to analyze the data.
- Study: Complete the data analysis, compare the results to the predictions, and summarize what was learned.
- Act: Based on the learnings, decide whether to Adapt the change, Adopt it as a new standard, or Abandon the idea and try something different.
SMART Aim Mnemonic
A project's aim statement should be clearly defined using the SMART framework:
- Specific: Clearly state what will be improved.
- Measurable: Include a numerical goal.
- Achievable: Ensure the goal is realistic.
- Relevant: Align the aim with departmental or patient priorities.
- Time-bound: Set a deadline for achieving the goal.
Example Aim Statement: "Reduce the number of missed drug doses for inpatients under ED care by 25% within 6 months."
4. Measurement for Improvement: Using Data Effectively
Measurement is vital in QI to determine if a change has resulted in an improvement. Data for improvement differs from data for research or assurance; it requires "just enough" data collected sequentially to inform the next steps, rather than large datasets collected "just in case."
4.1. Understanding Variation
All systems have natural variation. A key task in QI is to distinguish between this background noise and a true signal of change.
- Common Cause Variation: Natural, predictable variation inherent to a stable system (e.g., the slight day-to-day fluctuation in patient attendance).
- Special Cause Variation: Unpredictable, unexpected variation from external factors that signals a fundamental change in the system (e.g., a major incident or the effect of a QI intervention).
4.2. Types of Measures
A robust QI project uses a family of measures to provide a complete picture:
- Outcome Measures: The "voice of the patient." These measures focus on the results of care, such as patient satisfaction, harm rates, or clinical outcomes.
- Process Measures: The "voice of the system." These measures track whether parts of the system are performing as planned (e.g., time to analgesia, percentage of patients receiving a specific intervention).
- Balancing Measures: These measures look at the system from a different perspective to identify potential unintended consequences. For example, a project to speed up triage for one patient group should measure whether it creates delays for others.
4.3. Visualizing Data: Run Charts and SPC Charts
Plotting data over time is essential for understanding variation and impact.
- Run Chart: A simple line graph of a measure plotted over time with the median value shown as a horizontal line. Run charts help visualize trends and shifts in data resulting from interventions. A minimum of 10 data points is recommended.
- Statistical Process Control (SPC) Chart: A more advanced version of a run chart that includes a mean and statistically calculated upper and lower control limits. SPC charts are powerful tools for differentiating between common and special cause variation. Data points falling outside the control limits indicate that the process is out of control and requires investigation. A minimum of 20 data points is preferred for an SPC chart.
5. The Core Assessment: Mastering the QIAT
The Quality Improvement Assessment Tool (QIAT) is the mandatory electronic form on the Kaizen ePortfolio used to report annual QI activity. It is not merely a summary but a detailed account of the project's journey, analysis, and the trainee's learning. There are different versions of the QIAT commensurate with the training stage (ACCS, Intermediate, Higher).
5.1. Structure of the QIAT
The QIAT is divided into three main sections. A comprehensive submission requires detailed, reflective answers in each part.
Part 1: The Project
This section details the core components of the QI work.
- Analysis of Problem: Describe the problem the project was designed to tackle. Provide evidence (e.g., local audit data, incident reports) for why it was a problem in the department and how solving it would improve patient care.
- Use of QI Methods: Justify the QI methodology chosen (e.g., MFI). Describe the specific tools used (e.g., driver diagram, process map) and how they helped analyze the problem and develop interventions.
- Aim of the Project: State the project's aim clearly, ideally using the SMART format.
- Measurement of Outcomes: Detail the outcome, process, and balancing measures used and explain why they were chosen. Present key results, ideally including graphical representations like run charts or SPC charts (which should be uploaded as evidence).
- Evaluation of Change: Describe the interventions tested, referencing specific PDSA cycles. Evaluate the changes, including analysis of the data and a summary of what was learned from both successful and failed tests.
Part 2: Working with Others
This section assesses collaboration and engagement skills.
- Team Working: Describe the multi-disciplinary team involved, the trainee's role within it, how contributions were encouraged, and how any conflict was managed.
- Stakeholder Engagement: Identify key stakeholders (individuals or groups outside the core team) and describe how they were prioritized and engaged. Explain their impact on the project.
- Patient and Carer Involvement: Describe how the project improved care for patients or carers. Detail any active engagement of the patient/carer voice in the project's co-design or evaluation.
Part 3: Reflection on Leadership and Learning
This is a critical section that demonstrates insight and personal development.
- Self-Awareness: Reflect on personal qualities (strengths, weaknesses, values) and how they affected the project. Discuss managing workload under pressure, seeking feedback, and personal wellbeing. For higher trainees, the prompt is: "What is it about you that enabled this project to improve patient care, or why did you struggle?"
- Longitudinal Learning (from previous year): Outline what the year's activity has contributed to the development and knowledge of QI.
- Personal Development (future years): Describe plans for the next stage of the QI journey. What is the plan to learn or achieve next year to contribute to improving patient care?
5.2. Accompanying Evidence
The QIAT form itself has limited space. Supporting evidence is crucial and should be uploaded to a dedicated folder in the trainee's document library on Kaizen and linked to the QIAT form. Suggested evidence includes:
- Driver diagram
- Fishbone (Ishikawa) diagram
- Process map
- Stakeholder analysis
- Run charts or SPC charts
- Guidelines, pathways, or Standard Operating Procedures (SOPs) developed
- Posters or presentation slides
6. The Assessment Process and ARCP
The review and sign-off process is staged according to seniority.
- CT1 – ST5 Trainees: The completed QIAT is sent to the Educational Supervisor (ES) for review and sign-off. The ES provides feedback and an overall rating.
- ST6 Trainees: For trainees nearing CCT, a two-step process is in place to ensure a consistent standard.
- The QIAT is first sent to the Educational Supervisor (ES).
- The ES then forwards it to a Regional QI Panel for final review and sign-off. This panel consists of consultants with QI expertise.
The QIAT is graded as Excellent, Meets expectations, or Does not meet. An unsatisfactory QIAT will result in an Outcome 5 at ARCP. If it is the final ARCP, an Outcome 3 will be awarded, delaying CCT until the requirements are met.
7. Practical Guidance and FAQs
Project Selection and Common Pitfalls
- Start with a Problem, Not a Solution: A common mistake is "solutioneering"—starting with a predetermined solution (e.g., "we need a new guideline") without deeply analyzing the underlying problem. Use analysis tools to understand the system first.
- Scope and Scale: Choose a focused, achievable project. It is better to make a measurable impact on a small, well-defined problem than to fail at a large, overly ambitious one. Discuss feasibility with your supervisor.
- Success and Originality: A QI project does not need to be "successful" in achieving its aim to be a valid learning experience. A well-conducted project that demonstrates learning from failed interventions is valuable. Originality is not required; tackling a common ED problem within your local context is perfectly acceptable.
- Educational and Non-Clinical QIPs: Projects focused on education, wellbeing, or cost-saving are now permitted. However, the link to improving patient care, experience, or safety must be clear. For example, an educational QIP must demonstrate how improved staff knowledge translates into better patient outcomes, which must be measured.
Managing Change
QI is fundamentally about change management. More than 70% of organizational changes fail, usually due to a lack of staff engagement.
- Engage the Team: Involve the people affected by the change in the analysis and design of solutions. Ownership is key to success.
- Communicate Effectively: Use multiple channels (meetings, newsletters, emails) to share the project's aims, progress, and results. A clear message is vital.
- Visible Leadership: Be present on the shop floor to support the change, gather immediate feedback, and troubleshoot problems. This is often described as "Management by Walking About."
Advice for Specific Situations
- Previous FRCEM QIP Pass: Trainees who have already passed the FRCEM QIP exam are exempt from the mandatory submission of a QIAT until 2028. However, in the spirit of lifelong learning, continuing to engage in and record QI activity is considered good practice.
- Short Posts (e.g., 6 months): The QIAT covers the entire ARCP period. While completing a full project in a short post is not expected, active engagement in departmental QI in some form is required and should be documented.
- CESR Pathway: Trainees on a CESR pathway must now follow the 2021 curriculum and are expected to produce a QIAT annually for review at their appraisal or ARCP-equivalent meeting.
References
- General Medical Council. Good Medical Practice. London: GMC; 2013.
.png)
.png)