Hmm, I didn’t test this many tests. But you can run all tests in parallel — two at a time — and we should be good. The limitation isn’t the framework; it’s your budget and the tier you have on OpenAI. If you have Tier 2, which allows more requests to the OpenAI API, you could potentially run 1,000 tests in a single run.
Basically, it’s built on top of Playwright — kind of like Selenium. The pros are that it’s easy to write tests in English and you don’t need to maintain them. The cons are that you spend tokens, so it costs money for every test run.
Unfortunately, yes — you need to spend money for every test run. But you’re paying either way. Let me explain: I work at a big company with a large QA team. People write tests in JS/TS/Python. It takes time to write them, and then more time to maintain them when they break due to UI changes. Our team spends a lot of time fixing flaky and broken tests. Time is money. So leadership has to make a decision: should people spend less time writing and maintaining tests and focus more on quality, even if it means paying for test runs?
6 comments