QA Engagement during User Acceptance Testing

Quality Assurance Testing and User Acceptance Testing have obvious similarities, yet each have unique objectives.  Ideally, QA and UAT are performed by different teams and at different intervals in the project timeline.  QA are the SMEs in defect detection, removal, and testing process governance, while oftentimes the UAT team are the functional SMEs in their respective business area and “not so much” in the testing space – and could benefit from QA’s guidance to make it a success as well as a well-defined repeatable process.  This blog today will focus on  how QA can effectively engage with the UAT team and project stakeholders before and during the UAT test  to maximize the effectiveness and efficiencies of both teams.

So lets Learn 

What is User Acceptance Testing?

“Formal testing conducted to determine whether or not a system satisfies its acceptance criteria based on the business requirements, which enables the functional users to determine whether or not to accept the system”

What Exactly Happens when a UAT is being performed

·       A thoughtfully planned and well-executed UAT will enable users to verify that the system under development meets business requirements.

·       Associates/users will  become more familiar with the new functionality that is being introduced – developing the expertise required for roll-out and training

·       Expect defects to be found! This adds to the overall quality of the solution.

Throughout a project’s lifecycle, there are various test events and each has its unique objective.  This is depicted in the following diagram (commonly referred to as the V-model).  The intent here is to show that each test event is focusing on a different set of specifications.  For example, QA performs System Testing, which focus is on the functional specifications whereas UAT is focused on business requirements.  By having each team focused primarily on their respective specifications, all of the specifications will be covered with minimal redundancy.

QA vs UAT HOW TO MANAGE EXPECTATIONS:

•        Clarify, document, review, and approve expectations for QA and UAT during the early planning stages of the project:

•        Test Strategy: describes process, standards, and tools used

•        Test Plan: identifies scope, assumptions, risks, dependencies, entry/exit/success criteria, data and environment requirements

•        RACI chart for all QA and UAT test activities

•        High level review at project core team meeting and with resource managers – get on the agenda!

Oftentimes, the lines get blurred between System Testing and User Acceptance Testing from the perspective of the project stakeholders and even the project manager (“It’s all testing, right?”).  The QA  team makes assumptions about UAT, UAT makes assumptions about QA, and the project manager often sees QA owning it all. 

Properly preparing for the transition from System Testing (performed by QA) to User Acceptance testing (performed by functional users) is all about managing expectations.   Those expectations need to be clarified, documented, and reviewed/approved by all project stakeholders during the early planning stages of the project.  I found the most effective way to do this is to develop a RACI chart of all QA and UAT test activities and review these with the project stakeholders (primarily, the leads for QA, UAT, and development) as well as the project manager.  This chart -- listing who is Responsible, Accountable, Consulted, and Informed for each task -- should cover everything from test case planning, data required, test environment management, defect management as well as test event logistics (who manages rooms, dates, times, schedule, etc).  A sample for UAT activities is included here.

HOW A UAT IS PLANNED

Functional testers are business SMEs, who know their business processes inside and out.  Planning for UAT may be outside their core skills, and that’s where help from QA is needed most.  Define a repeatable process for UAT and a UAT Test Plan Template that can be reused over and over again and ensure it covers the 4 W’s (Who, What, Where, When) and How.

Define When and Where to Test

•        Centrally co-locate the test executors (same room is best!).

•        Reserve room(s) and communicate the plan.

•        Have the development lead and the QA lead also present during UAT for support and quick turnaround of blocking issues.

•        Ensure test environment is available and test executors have proper access.

Define How to Test

•        Define and communicate how UAT will look in terms of:

•        Process

•        Tools

•        Test case management

•        Defect management

•        Reporting

•        Provide logins and training for any test case management tool or defect tracking system before UAT. 

•        Otherwise, have a standard test execution template that is accessible for everyone and that the process for filling it out is understood.

Providing Training

It’s best to schedule these trainings the week prior to the start date of the milestone for which that training is targeted for – for example, if high level scenarios are due to start in week 8 of a project, provide the training near the end of week 7 so it’s fresh in their minds.

Provide separate and just-in-time training throughout the UAT planning phase:

UAT Overview – the Why, What, Who, When, Where, and How of UAT

UAT Test Planning Part 1 (Scenarios) – what is a scenario, how to document high level scenarios (how to put them in a test management repository, scenario standards, etc.), approving scenarios

UAT Test Planning Part 2 (Test Cases) – breaking down scenarios into positive/negative test cases with detailed steps, prerequisites, test objectives, test case standards, and traceability back to business requirements.

UAT Test Execution – how to execute test cases and document test results, reporting and logging defects, defect retest process, defect prioritization, UAT metrics, test case scheduling and assignment, logistics

It’s important to determine and agree upon Entry/Exit Criteria for UAT and what constitutes Success.  This should be documented in the UAT Test Plan (and reiterated at the project core team meeting), reviewed and approved so that at the end of the UAT test event, there will not be any debate over when UAT should end and if it was deemed successful.

UAT ENTRANCE CRITERIA:

•        Document Entrance/Exit/Success in UAT Test Plan and reiterate at Core Team Meeting

•        Obtain approval prior to the start of the UAT test event

Example:

•        Critical and High functionality is complete and deployed to the test environment

•        Configuration guide is complete and all configurations are completed in the test environment

•        System Test (QAT) is 95% complete, with 95% pass rate and no blocking/critical open issues.

•        UAT test cases are complete and approved, and test case schedule is known to all participants.

UAT EXIT CRITERIA:

Example:

•        96% of all planned tests are executed and reflected in test event metrics.

•        All failed test cases have a defect logged and all business functionality defects are associated with a failed test case.

•        Business review and agreement of all open defects by severity and priority by <date>. 

•        No unresolved Critical defects and all unresolved H/M/L defects have a resolution plan.

UAT SUCCEESS CRITERIA:

Example:

•        98% of executed test cases pass

UAT EXECUTION 

•        Communicate daily plan/schedule

•        Prepare and distribute Daily UAT Metric Report (use a template)

•        Conduct daily standup with all participants:

•        Communicate and prioritize issues found during the day

•        Ensure defect fixes are retested and when next set of fixes will be available

•        Monitor planned vs. actual test execution to ensure testing stays on track (if not, provide mitigation plan)

Ideally, UAT testers are in a designated room with the QA lead and development lead.  QA is there to help with environment issues as well as support for the application under test and the test and defect management tools.  A daily standup meeting with all participants is recommended to communicate and prioritize issues found.   QA should be ensuring defect fixes are retested and looking at planned vs. actual execution metrics so that the testing stays on track.  Lastly, QA should prepare the daily metric report which should be discussed during the UAT daily standup meetings. Having a standard template for reporting UAT metrics is recommended and sets the expectation of what information is going to be reported.

UAT Metrics and Reporting

•        At a minimum, UAT Metrics Report should include:

•        Defects: #open/closed by severity and status (if many, then in priority order)

•        Test execution: plan vs. actual and pass rate

•        Break down by business area or team, if applicable Overall

As with QA System Test, metrics on defect detection and test execution should be done throughout the UAT test event.  It should be concise and at a level so that at the daily standup, it’s clear where things are at in terms of defects found, blocking issues, and how test execution is going per the plan.  If multiple business areas are participating in UAT, it’s helpful to not only show overall progress, but the breakdown per team.

Post Go-Live: Lessons Learned and Continuous Improvement

•        Issues will occur!

•        Perform root cause analysis on post go-live issues to determine what process improvements need to be made and whether test cases or data need to be added or updated to the test suites.

•        Conduct Lessons Learned session with all UAT participants and project stakeholders on what went well and what could have been done better

CONCLUSION:

User acceptance testing helps create a competitive product that meets your business and product goals. Follow this guide and adjust it to your needs to achieve the best results.