Preface
In
this blog, I have attempted to summarize the important aspects of test
automation tools selection process, based on my personal experience, and industry
insights. I hope you will find this read interesting, and unique in nature, and
it will help you enrich your knowledge and widen your views. Please share your
opinion using the comment section and share it with your friends and colleagues.
Introduction
If
you are working or had worked in an IT organization, and ever worked/interacted with a
QA/Test team, you will often hear these terms “Automation POC”, “Tools
selection”, “tool evaluation” etc. The “test automation tools evaluation and
selection process” is an integral part of any organization who is either a
consumer or producer of IT services or products.
Initiation
Before initiating a test automation tools
evaluation and selection process, first we need to brainstorm and understand
what are the business needs, what are the goals and objectives, what is the
budget, what are the timelines, how many team's representatives needed
(depending upon stake in the tools evaluation), are there any existing
organizational process to obtain the trial licenses from the tools vendors, the
degree of freedom given to the Test Architect/Automation Architect / Automation
Manager / Test Manager (to shortlist the tools, procuring trial licenses,
selecting and entrusting the team members for the evaluation and reporting,
steering SLA's with inter-operating teams, i.e. internal Cyber security teams
to get timely compliance clearance for installation of required software's /
cloud tenant subscription etc., DevOps team to set up integration with CI, CD,
or tools like JIRA, setting up demo test servers/data etc.), are any of these
tools used by our parent or sub-organizational entities in other geographies at
present or past, do we have any of these tool vendors in our existing/preferred
suppliers list etc.
Next, draft a plan that will detail about the goals and objectives, list the test automation tools that will be evaluated, mention the test scenarios to be covered, document the trial license procurement procedure, specify any needed test infrastructure/software, specify security compliance procedure, commitment needed from various teams, activities to be carried out, by whom and when, synchronizing or parallelization of tasks, defining the communication channels and control flow, timelines, reporting mechanism, frequency of review meetings, data management process in compliance with appropriate data-privacy law (including sensitive data, such as bespoke documents, pricing details, test data etc), risks and mitigation, expected outcomes (Questionnaires, Checklists, Reports, Activity Diagrams, Review logs etc.). Then circulate it among the stakeholders for their review and collect feedback. Ideally always place the plan under a configuration management tool for versioning.
Once all parties are satisfied with the plan and they sign it off, then an official kick off for the tool evaluation project can begin. I have categorically used the term "project" here, as it will be a time-bound piece of work with specific objectives and goals to achieve and will be used as a strategic instrument. In general, test tool evaluation is considered as a process as it is repetitive in nature, but each iteration may have different objectives to achieve.
Team selection and team building (can overlap with planning)
This is pivotal to the success of a tool evaluation
and selection project. Ideally 2 to 3 committed personnel are required, additionally
keep 1 person as buffer to cover up any absenteeism or reassignment of any
personnel for any other priority work. The entrusted personnel should have
passion for learning new tools, experienced enough to deal with the technical
complexities, strong analytical, communication and reporting skills, self-motivated,
know when and how to deep dive to gather crucial insights keeping in mind the
objectives the tools are expected to meet and the end goal of the project, can deal
with timeline pressure, identify and communicate the risks/issues on time.
It is always a good practice to allocate some
personnel with diversified skills and experience to shadow the committed personnel
on this project. This will help to develop the workforce and will spread positivity.
Remember, in Agile projects, it’s hard to find such available personnel.
Tools shortlisting (can overlap with planning)
This must be aligned with the overall
organizational strategy of an organization’s digital journey and the vision for
quality. Test automation tools can be broadly categorised into these categories:
UI, API, CLI, Mobile, RPA, AI/ML (including BOT), Embedded, Platform Automation (aligned towards DevSecOps), IoT, Visual Testing, Accessibility.
Depending on the category, prepare a questionnaire, keeping in mind the objectives and goals, to gather and capture important and necessary information about these tools. Create two copies. Allocate a team with a timeline, handover one copy to them, to do research and find top 3 tools that satisfies majority of the questions. In parallel, take opinion from other experts within the organization, who might have been used any such tools in the past. Once this activity is concluded, collect the report from your team, and compile all gathered information. Next, have a meeting with your team and assert their confidence levels on the 3 tools that they have shortlisted and recommended.
Remember, sometimes, tools are shortlisted at a
strategic level and passed on to a team for evaluation and feedback, based on
that final procurement decision is made.
Procuring automation tools trial licenses
Depending upon whether the organization has
existing procurement process or not, this can be initiated by entrusting
Automation Architect / Automation Manager / Test Manager to steer the
communications with either the procurement department or the tools vendors.
In this process, the first step is to contact the tools vendor through their official website's request a demo license page. Once you submit a form, the vendor organization's representative will connect with you to gather details about the business needs and expectations. At this point, you could provide them a copy of the questionnaire to capture their responses. Focus on any hardware, software, network requirements for the tool. Involve representatives from your organization’s cyber security compliance, DevSecOps, and procurement departments, and include Automation Architect/Test Manager and Technical Architect/Principal Engineer in these discussions. Be rational during these discussions, don't ask for too many free licenses, as some may be wasted due to lack of commitment/unavailability of people, set right expectations, request for bespoke offer and pricing details. Sometimes a non-disclosure agreement needs to be signed, as pricing is a sensitive information. If you have strict timelines, tell them upfront, build upon mutual trust and establish some SLA for communication turnaround time, it will be helpful in expediting procurement of demo licenses.
Installation, exploration, and reporting
These days, many automation tools are offered as
SaaS (‘Software as Service’ cloud offering). So, you may not be required to
download and install something, like traditionally. But if it is required for
the chosen tool, then read their installation guidelines carefully, do the
installation and setup. Most vendors allocate a SPOC to reach out for help or a
ticket-based service desk or a BOT to contact support, use it wisely.
Once done, start the exploration by automating the scenarios as per the test plan. At this stage, team members are expected to do multitasking, they should use their committed time in reading/watching videos of the tutorials/user docs and then they should apply the gained knowledge immediately to progress automating the scenario. It’s a good practice to fill in a query tracker and send it to the TA/TM, so that they can communicate with the tools vendor and expedite removing any impediments. Also, keep making notes, from the experience you gain out of this activity, as you will need it at the end to write up a report. Highlight any risks immediately to the TA/TM so that appropriate actions can be taken in good time.
Once, the activity is concluded, send a report in the predefined format, to the TA & TM, including your recommendation.
Note, sometime multiple tools are explored in parallel. It is the TA / TM's responsibility to combine the reports and producing a master report for the higher management.
Final procurement decision
Multiple factors influence this decision, majorly pricing model, and budget trade-off on a long term, it plays a crucial role. Apart from it, Vendors reputation and credibility, cyber and data security aspects, turnaround time for procurement process and all other important communication, the tools fitment for the pipeline projects across the organization portfolios, existence of talent pool in local job market, ease of use, training and support, software & patch updates, ease of migration from old to newer version etc.
Sometimes, the decision gets deferred, it favours
an open-source solution, or the organization outsource that automation work to
an entrusted service-provider for short term to observe the potential benefits coming
out of using the selected automation tool.
