QA Quonversations: AI’s Place in Software Testing

At Livefront, the Quality Assurance team likes to get together every couple of weeks for a “QA Quorum” to talk about project updates, interesting bug finds, tactics we’re experimenting with, and new things in the world of QA. One of the topics we recently discussed was how we use AI in our day-to-day work, and we thought, “This would be a good conversation to share with others!” So, welcome to the first of what we like to call, “QA Quonversations.”

Joining this conversation were Livefront QA team members: Ben Kimball, Haley Bowler, Lauren Whitesell, Lukus Cich, Jeremy Eventyr, John Caplinger, and Sonia Merten. The team has a combined experience of 30+ years in the Quality Assurance field, focused on API testing, UI/UX testing, accessibility testing, manual testing, and test automation.

We started the conversation with the question:

What are some ways you are using AI for Quality Assurance?

Across our QA team, AI has become an everyday tool that helps us think through problems, write better documentation, broaden our test coverage, and troubleshoot tricky technical issues. All of us have unique use-cases depending on the client we’re working with, but each of us have found ways to make it a meaningful part of our workflow.

Ben said he often leans on AI at the beginning of his process to help him get his thoughts moving. He likes to use AI for “rubber-ducking” problems he may be running into and to help bounce around ideas. “Usually the responses I get back will help me form my thoughts and move down to whatever workflow I want to go,” he said. That same approach has helped him with other tasks like creating a tutorial on merge conflicts, where he walked through the challenge with ChatGPT until he found a reliable method.

Jeremy said, “I’ve just started using Cursor and I’ve found it really useful for writing documentation.” He realized that he could ask Cursor to analyze his automation tool repo’s code and generate a Readme file for the repository. “It went through the repo, analyzed the tools, then gave some short descriptions and some examples of how to use them.” He said there was quite a bit of editing necessary afterwards, but it was a solid basis to start coming up with the documentation for the code.

Lauren works on an open-source project where a “description of work” is required when approving a ticket. “I will take the contents of the ticket before I start testing and ask AI to provide a list of scenarios to cover.” She said this helps her catch any test cases she might not have initially thought of and guides her on what she will be testing. She will also use AI to help write up the required comment once she is done testing the ticket. “That’s the biggest thing I use it for day-to-day.”

Sonia mentioned, “I feel like this is an obvious one, but utilizing Copilot when writing automated tests. It’s been really helpful with refactoring some of my tests by making them more efficient, generally making the code better, and helping me understand the code better too!” She also gave a shout-out to Ben for showing her how to use AI to summarize what has changed in a current PR instead of having to write it manually.

Lukus also works on an open-source project where there is a lot of complexity. “This may be a one-off situation, but I’ve found it really useful for creating diagrams.” He said there was a complex flow for a particular test coverage area, “so I fed it some information and got a super helpful diagram that I could share with other QA folks on the team and it helped our testing a lot!” And when he needed a simulated rooted Android device and ran into issues the forums couldn’t solve, he turned to ChatGPT. “It actually ended up helping me get to where I needed to go!”

What are other things do you find AI useful for?

John finds AI tools helpful for taking meeting notes, especially for gathering details he might have missed. “That’s definitely been a useful tool for me!”

Sonia has found it very useful for understanding particular accessibility guidelines. And the information provided allows her to be sure her testing is accurate. “It can point me to the WCAG (Web Content Accessbility Guidelines) and I don’t have to spend time searching for the information I need.”

John chimed in, “Using it to compare things is really handy too!” He works on a project that includes both a web and native app. To be sure the test suites were the same for each platform, he fed ChatGPT both test suites and prompted it to check the differences. “It was better than one of those ‘spot the diff’ programs because some test cases were named differently, but the AI was smart enough to figure out they were the same test!”

Ben was working on a spreadsheet project at the time and found it useful for providing Excel or Google Sheets formulas. “For example, combining two strings and then comparing them to something on a different worksheet, it’s really good at that!” Determining SQL queries or JQL queries for Jira is another thing he likes to use it for.

What is your process for creating test cases using AI?

John described a progressive prompting approach. He starts with broad context, then gradually refines prompts with specific requirements and scenarios — such as device orientation, authentication state, or poor network conditions. “Afterwards I’ll manually refine the list of test cases provided into what I feel are the appropriate tests to test.”

Lukus takes a different route, “I’ll create my first pass of test cases and then feed those in and ask it if there are any other scenarios that I didn’t consider.” While not all suggestions he received were useful, the exercise helped surface edge cases he hadn’t thought of.

Lauren said she follows a similar approach as John, but since the project uses Gerkin-styled test cases she says, “I’ll ask for the test cases to be written in that style. Then I’ll edit the test cases since it usually provides test cases that aren’t necessary or splits them up in a way I don’t like.” She then continues to iterate and refine her prompts until the test cases meet her standards.

Sonia voiced some concern about using AI to create test cases. “I find it’s a little verbose whereas I like to make my test cases as concise as possible.” For her, AI is better suited to identifying gaps than generating test artifacts.

Lauren and Ben both felt there were ways to fine tune the AI’s responses. Lauren said, “I’ve had some success with prompts like ‘make this more concise’ but the results can still be hit or miss.” Ben agreed, “you could tell [the AI] to always give you test cases in an action-expectation format or Gerkin and it should stick to those instructions.”

What are some ways we could improve our AI results?

At Livefront, one of the company mantras is, “I leave things better” which is why we wanted to discuss this question. Lukus brought up the idea that we could create a repository with the list of prompts that we know work well.

Jeremy responded, “That would be nice if we had some sort of library that you could use to say, here’s the prompts you could use to create a readme file or here’s the prompt you could use to look for edge-cases in this test scenario.” We all agreed!

Are there things you’ve found that AI consistently messes up?

Ben chimed in, “Until about 40 minutes ago I thought it didn’t know what animals were, but it turns out I don’t know what animals are!” He was working on creating an array to select a random animal and the AI included animals like “Zorilla” and “Zorse”. “…but it turns out those are real animals!” he exclaimed. John replied, “Ha! I didn’t know those were animals either!”

Jeremy pulled us back on track and said, “One of the things we’ve seen consistently with Copilot or Cursor is it’s solutions a) have hallucinations and come up with functions that don’t exist or b) it will generate code that is more complicated than it needs to be.” We all agreed, it is still important to know the coding syntax you are asking for help with in order to parse out these types of AI mistakes.

Ben mentioned he’s had a lot of issues when using Copilot when writing tests where the AI would edit the code in places he didn’t ask it to. “I’ve had to move to a ‘don’t do anything unless I say yes’ mindset with it and that works pretty well,” he said.

Haley remarked that she has trouble getting Copilot to provide useful responses. She asked Ben, “are there certain settings you’re using to control that?”

Ben described a process of creating an “.instructions” markdown file inside of “/.github/prompts/”. “That ‘.instructions’ file is the best thing you can do for yourself to use Copilot,” he said.

Our conclusions

The consensus we’ve arrived at is that AI is a helpful tool for troubleshooting, repetitive tasks, summarizing and documenting tests, and filling in gaps in our test suites. The things that you could teach almost anyone, but the AI learning curve is off-the-charts.

On the flip-side, we as humans still need to be vigilant about the data we’re feeding into the models and the trust we have in the results we get back. Blindly accepting what AI suggests only proves the limits of our own skills and inconsiderate prompts will only give you inconsiderate solutions.

What are your thoughts? Where do you find AI useful in your day-to-day? What do you watch out for when using it? Let’s keep the conversation going!

We have many more topics to discuss as a Quality Assurance team, so stay tuned for more QA Quonversations!

• John Caplinger

  QA Analyst

  More from John