Tester 2.0: becoming indispensable in the age of AI

Imagine taking a person from 17th-century Europe and bringing them into today's world. You could show them the internet, our computers, our means of travel or AI. But my favorite place to take them would be your average supermarket spice aisle.

Nutmeg, cinnamon, pepper, cloves. Things we barely think about putting in our shopping basket were once expensive luxuries. A whole aisle full of them would look like incredible wealth to someone looking through that person's lens.

I think we're witnessing something similar with code. Something that used to be expensive has become incredibly abundant. We can produce it at a rate that would have sounded ridiculous just a few years ago. But having a lot of code doesn't automatically mean we've become code rich.

That's where I started my keynote, Tester 2.0: becoming indispensable in the age of AI. I wanted to talk about our lens as testers and quality engineers, because I think some of the assumptions we're bringing into this change need adjusting.

There's a saying associated with Abraham Maslow: if all you have is a hammer, everything looks like a nail. I sometimes see a similar bias in testing. Every problem in technology starts to look like a testing problem, and every answer looks like more of the testing we already know how to do.

Over the past few years, I've spent a lot of time talking to testers and developers about AI. I've seen a lot of hope, a lot of cope, but also invention and innovation. I care about this community, so I want to share some convictions that might make you uncomfortable. There are three lies I think we tell ourselves, three things we need to abandon, and three strengths we can build on.

Three lies we tell ourselves

The golden age of testers is coming

I've heard this said from a conference stage, and I understand the logic. Code has become cheap, so we'll produce more of it. More code means more testing, which means companies will hire more testers and QA teams will grow.

I'm as guilty as anyone of believing this. I've even written a version of it myself. But more code to test doesn't necessarily mean bigger QA teams.

Hiring another tester is one possible response to a growing workload. An employer can also invest in tools so the existing team can handle more, or spread testing responsibility across engineering. The amount of work and the number of people hired to do it are different things.

I'm meeting testers who have been laid off and are struggling to find another role. That doesn't fit neatly into the story that more code will automatically create more demand for the roles we have today.

The hardest part of this lie is that there's a grain of truth in it. I don't think we're wrong to expect more quality problems and more things to validate. But expecting those problems to be solved with the same roles, team structures and ways of working is a much bigger assumption.

Without testers, things will inevitably get worse

This one hurts my ego. As someone who spent years in testing, I refused to let go of this conviction for a long time. Finding bugs was part of my contribution, so it seemed obvious that removing testers would make quality worse.

But while we test everything, (sometimes including our colleagues' patience, haha) we haven't always put this particular assumption under test.

Teams can change how they deliver quality. Developers can take more responsibility for verification. Automation and observability can replace manual handoffs. A team can work with a "you build it, you run it" culture, with shared standards and the autonomy to execute.

We've talked about this much longer than we've talked about AI. It's shift left.

At QA conferences, I often hear shift left described as getting testers involved earlier in development. But in practice, it can mean getting testing involved earlier.

When this works, it's a cultural and architectural transformation. The team looks at outcomes, such as defects reaching production and how long delivery takes, rather than assuming quality follows from having a QA team.

The comforting response is to believe they'll eventually fail and come crawling back to hire us. But that leaves our future dependent on someone else's failure.

We can't build the future of testing on the hope that teams will fail without us. I truly believe we can make a better case for ourselves.

Things will stay pretty much the same

"They said automation would replace testers, and we're still here."

I understand the skepticism. We've seen technologies promise to change everything before. But I think this argument misses what happened inside our profession. Automation didn't eliminate testers, but our tools, challenges and approaches changed tremendously.

A profession surviving a technological shift doesn't mean the work stayed the same.

We also don't work in isolation. If the daily work of everyone around us changes, it's hard to expect that none of it will affect us. Development changing is already part of the environment we work in.

More importantly, "things will stay the same" is a prediction. It deserves the same scrutiny we'd give any other assumption. Just because it's soothing to believe doesn't make it true. It may be a way to cope with our fears.

Three things to abandon

Avoiding technical expertise

I firmly believe we need to stop debating whether testers should become technical. Of course they should. Full stop.

I still encounter hesitation about understanding the technical details of the software we test. The argument is that it might create tunnel vision, make us lose our unique perspective on the system, or turn us into developers. I wholeheartedly disagree.

I sometimes look at quality work in other professions. Quality engineers in motor sports need to have deep understanding the car's engineering. The same applies to materials and safety systems in an amusement park. Aviation, automotive industry, and other engineering professions all need to have a technical understanding of systems they test. Knowing how the thing works is part of being qualified to assess it.

I think we need to take our profession seriously enough to treat it as technically demanding.

This matters beyond the job title we happen to have. Titles change, but the need for expertise remains. And in my experience, LLMs reward expertise too. Watch a senior developer interact with AI and compare that with someone casually vibe coding a solution to a problem they don't understand.

If I can offer one direction to explore, it's AI itself. Learn how agents work, how to evaluate them, and how a harness helps them do their work. Teams need people who understand how to make these systems reliable.

We want to become AI-first, jump head-first into tokenmaxxing, and then meet every Tuesday for a post-mortem because Claude Code ignored half the requirements in the Jira ticket.

This is an opportunity for us. We can help teams figure out ways that work, but we need the technical knowledge to contribute to those solutions.

Calling critical thinking our only important tool

This is a bit of a pet peeve of mine. At conferences, I often ask people what tools they use, and it annoys the hell out of me when the answer is that their brain is their most important tool.

I'm sure your brain is amazing. Your colleagues have brains too. Hopefully.

Usually, the conversation then turns to critical thinking, as if it's a specialty that comes with being a tester and makes everything else optional. But there are plenty of critical thinkers on your team. Some of them are probably your best developers.

More importantly, critical thinking is something you build. You learn new things, gather new perspectives and uncover your own biases and blind spots. That's why I connect it so strongly to technical expertise. Broadening your understanding gives you more ways to question what's in front of you.

If your critical thinking stops at the boundary of your own role, it's self-preservation dressed up as a professional skill.

Treating test automation as our only tool

To balance things out a little, I don't think the answer to everything is writing more automated tests either. I expect test automation to change, and I think we need to be willing to explore what that could mean.

In the keynote, I used a thought experiment. Imagine we had no automated tests at all, but we got to keep AI. We'd need to look for other signals to help us assess whether a change is safe to release.

There's actually quite a lot to work with:

• Type checking, linters and the build process.
• Code review, commit history and related repositories.
• Tickets, requirements, documentation and designs.
• Compliance checks, engineering rules and agent skills.

These can uncover requirement gaps, regressions and problems with following our standards. Agents can help us examine those signals and give us useful information before we run a single end-to-end test.

In my own workflow, I loosely split the work into generation and verification. The coding agent takes in the ticket, documentation, rules and skills, and creates a feature. But it has a limited context window, makes mistakes and sometimes ignores things written right there in the ticket.

So I use a separate review agent, or a set of agents built on a different model. The review checks whether the ticket was fully implemented, whether the rules were followed, and whether there are bugs or security issues. It also looks at the blast radius of a change. Two changed lines can affect something elsewhere in the repository, or break another repository connected to it.

Doing this only at the pull request gives us quite a wide feedback loop. We can bring the same review into uncommitted changes, then earlier into the implementation plan, and even into the ticket. We can examine how the requirements fit the codebase before we start implementing them.

By the time tests run, the change has already been through multiple checks. I'm getting feedback earlier, while there's less work to undo.

The reason I think this thought experiment is useful is that our end-to-end suites might be becoming a bottleneck. We have excellent tools for creating tests, and sometimes we've overdone it. We end up with huge suites, nightly builds and runs that take hours.

Imagine a developer opening twelve pull requests a day where they used to open one. Running the same enormous suite for every change may push us to rethink how we get feedback. I don't have a neat prediction for what test automation becomes, but I think we need to stop treating it as our only tool.

Three strengths to build on

A quality-first approach

As roles overlap, it's useful to look at work that requires testing abilities, even if the job title doesn't say "tester".

Take the forward-deployed engineer role. The part I want to point to is taking a customer's problem and delivering a solution, including responsibility for its quality. Our experience investigating user problems and thinking about how a solution might fail is useful in that kind of work.

Another example is an eval engineer. They design evaluations to measure how well an AI system works, where it fails, and whether a change improves it or makes it worse. In simple terms, they're like a teacher for an AI system. They create exercises, decide what counts as a good answer and check its work.

During a tutorial alongside the keynote, we compared an agent working with a skill file against the same agent working without it. We checked the output and measured token usage and response time. You'd expect adding a skill to improve the result, but we mostly burned a whole lot of tokens without really improving the response.

There's a lot here that should feel familiar to us. Understanding the user's problem. Defining what success looks like. Exploring missing information, unexpected results, retries and failure modes. Those can become repeatable evaluation cases.

Imagine an assistant says it booked a meeting. We need to check whether the meeting exists, whether it's at the right time and whether the right people are invited. We also need to know whether our evaluator would catch it if the assistant hadn't done what it claimed.

The same applies when changing models. Updating to the latest model doesn't tell us whether our system got better. Someone has to measure that.

These roles require us to learn new things. But our testing experience gives us a solid foundation. I meet testers who have been laid off and feel they have nothing to offer, and that's just not true. Get comfortable looking at other job titles that need the talents you've been developing.

Being a generalist

There's a lot of broad knowledge involved in testing. We need to understand the business, our clients and their needs. We look at user experience, behavior, performance, accessibility and the code. We look at the same product from different perspectives.

I think this generalist mindset is something to lean into.

A year before the keynote, I was working for a company building a product for vibe coders. If you know Lovable, Bolt or Replit, you know the kind of product I mean. We had a talented development team, and our product could spin up an application that worked as described, with all its tests passing.

And yet nobody would use it.

A working technical solution didn't mean we'd built something a customer wanted, needed, liked or would buy. Just because we could build something didn't mean we should.

Someone still needed to understand the user, choose the right problem, manage tradeoffs and figure out whether the result actually helped. This is why being able to look at a problem from different perspectives matters.

If you ask me, we've created a lot of code that should never have been created. Our broad understanding gives us a way to contribute before we get to checking whether that code works.

Turning uncertainty into experiments

This is one of the things that made me fall in love with testing. We don't know something, so we find a way to learn about it.

With AI, we're facing plenty of uncertainty. But the three lies I started with are ways of feeling certain during uncertain times. They keep us comfortable without necessarily helping us understand what's happening.

We have the capacity to innovate and run experiments. We can turn uncertainty into knowledge, build our critical thinking and become experts. Some people frame testing as constant skepticism. I like to think of it as constant curiosity, and I don't think we're tapping into that strength enough.

So the homework I gave in the keynote was to get your hands dirty. Build an agent. Try something new. Create something for yourself or somebody else. Do that especially if you don't know where to start.

Striving for excellence

I know I'm calling for a lot of change. I've talked to testers and developers who are having a difficult time because a skill they've spent years perfecting suddenly feels like it's being taken away. As models get more capable, we question what's left for us.

I'm not sure I have a clear or satisfying answer. I only have hints of one.

When we see a great sporting performance, an amazing piece of art or a beautiful piece of engineering, it makes us feel something. A game between two chess grandmasters can mean something different to us than watching two computers play. We marvel at the ceiling of the Sistine Chapel. We read The Lord of the Rings and feel something that generated text may not give us.

I think that when we see another person achieve something great, a little part of us believes we're capable of great things too.

Whether you pick up a brush, kick a football or create a product, your humanity will speak through what you do. We're spending a lot of energy thinking about what AI might take away, but whatever tools we use, we still have something to contribute.

So I think you should never let go of striving for excellence. Pour it into everything you do. That's how you become truly indispensable.