TestingOps.ai

Test execution · Execution provider

RunningOps.ai and TestingOps: run test cases and collect evidence

TestingOps sends your test cases to RunningOps, follows each run and brings the verdict, step screenshots, video and logs back, so results, defects and reports work from the same evidence.

  • Execution provider
  • Available
Request a demo
Connection
Execution provider
Writes back
Runs on your provider account
Last checked
7 Nov 2026

In short

RunningOps.ai and TestingOps in short

RunningOps.ai is the execution provider TestingOps runs test cases on.

TestingOps sends each case's written steps to the RunningOps public API with your organisation's API key; RunningOps plans and runs them in a browser or on a mobile device with its own engine and returns its own pass or fail verdict.

Step screenshots, a video recording, a console log and the Playwright or Maestro script RunningOps generates come back to the TestingOps run, where defects and reports use them.

On testingops.ai it is the only automated runner the execution policy allows, and each customer runs in its own RunningOps organisation.

What it does

What it does for your testing

  1. 1

    Written steps become a run

    Input
    A test case with steps and expected results in plain language
    TestingOps
    Sends the steps to RunningOps' public v1 API as a prompt for its engine
    Output
    A RunningOps run that RunningOps plans and executes with its own engine
  2. 2

    Evidence comes back

    Input
    A finished RunningOps run
    TestingOps
    Downloads the step screenshots, the MP4 recording, the console log and the generated Playwright or Maestro script
    Output
    A run with RunningOps' verdict for every step and the evidence to judge it
  3. 3

    Broken runs stay apart from failed tests

    Input
    Engine-side problems such as a step time-out, an agent output error or an app blocked by a forced update
    TestingOps
    Records them as errors with their real cause instead of failed checks
    Output
    Pass and fail figures, and defects, that describe the product rather than the runner
  4. 4

    Mobile batches check the app first

    Input
    Android or iOS cases for one app
    TestingOps
    Sends them as one batch; RunningOps opens the app once and looks for a forced update, maintenance or a service error
    Output
    A blocked batch waits for your decision instead of failing every case

Step by step

How it works

  1. Connect your RunningOps organisationUnder Integrations → Project tools → Test execution → RunningOps.ai, select Get the key from RunningOps when your account was created together with RunningOps, or paste an API key (rops_…) from your organisation, then Test connection.
  2. Start a test planOn Test plans, RunningOps.ai is the Run target. You can choose which AI model drives the run in RunningOps and how many cases run in parallel, within your package.
  3. TestingOps sends the casesWeb cases go to RunningOps' public v1 API one by one; mobile cases for one app go as a batch, so RunningOps can check the app before the run.
  4. RunningOps runs them and decidesRunningOps plans the steps with its own engine and returns pass or fail for each case. TestingOps records that verdict and does not judge the run a second time.
  5. Results and evidence arriveStep screenshots, the video, the console log and the generated script are attached to the run. Watch live and Open in RunningOps lead to the same run on RunningOps' side.

Which runners a server may use is set by its execution policy, an environment setting that applies to every company and customer on that server. On testingops.ai it allows RunningOps and manual runs; other runner cards show "Disabled by execution policy: all runs go through RunningOps".

Connect

Setup

  1. On testingops.ai, your customer account needs its own RunningOps organisation; if it is not linked yet, a TestingOps admin links it from the customer's record with Link RunningOps.
  2. Open Integrations → Project tools → Test execution and select RunningOps.ai.
  3. Select Get the key from RunningOps, or paste an API key (rops_…) from your RunningOps organisation, then select Save and Test connection.
  4. Optionally set what happens when a mobile app is blocked before a batch, whether the app's data is cleared before each mobile test, and the step and scenario time limits.
  5. On Test plans, keep RunningOps.ai as the Run target and select Run.
Where you connect it in TestingOps: Integrations → Project tools → Test execution → RunningOps.ai

Use cases

Use cases

Run a regression plan on a schedule

Trigger
A test plan starts from its schedule or by hand
Uses
The plan's cases, its environment and the parallelism your package allows
Agent activity
TestingOps sends each case to RunningOps, follows every run and collects the verdict and evidence
Result
A run with a verdict, screenshots, video and log for each case
Condition
Parallel runs are bounded by your RunningOps package and quota

Open a defect a developer can act on

Trigger
A case fails on RunningOps
Uses
The failed step, its screenshot and recording, and the case's recent runs
Agent activity
Fills the defect from the run: steps with the failed one marked, expected and observed result, environment, run history, evidence and a likely cause class
Result
A defect with the evidence attached instead of a bare error message
Condition
The likely cause is a direction to look, not a verdict

Choose the model that drives your runs

Trigger
A team wants to settle on one model for a plan
Uses
Compare on AI models on the test plan
Agent activity
RunningOps runs the plan's cases on two to eight models and ranks them
Result
Models ranked by success, cost and speed for that plan
Condition
Comparison runs are real runs on RunningOps, up to 500 per comparison

Start a mobile suite while the app demands an update

Trigger
A mobile suite starts while the app shows a forced update
Uses
RunningOps' app check before the batch and your blocker setting
Agent activity
Holds the batch and asks you, retries with the latest uploaded build, updates from the store or continues, as the setting says
Result
No batch of failures caused by an update screen
Condition
Applies to mobile cases, which are sent as a batch

Capabilities

Capabilities and limits

What is read

  • Run status and RunningOps' own pass or fail verdict for each case, with the result of every step
  • Step screenshots, the MP4 recording, the live view and the console log
  • The Playwright spec (web) or Maestro flow (mobile) RunningOps generates for a run, kept as the run's script
  • The engine's capabilities, capacity, quota and device list for your organisation

What is produced

  • Test run results in TestingOps with per-step evidence, used on run pages, in defects and in reports
  • Engine-side failures and app blockers recorded as errors, not as failed checks

Where it can write

  • Runs, batches and scenario runs in your RunningOps organisation; cancellations; decisions on blocked mobile batches; uploaded app builds
  • With the partner connection: the customer's RunningOps organisation, its API key, invited users and the customer's AI provider keys

How it connects

RunningOps public v1 API with the organisation's API key (rops_…) as a Bearer token; the key is stored encrypted

Account and permissions

  • On testingops.ai, a RunningOps organisation of your own: a customer project without one cannot run tests
  • An API key from that organisation, issued by TestingOps (Get the key from RunningOps) or pasted
  • An admin role to manage the connection; other users are told that test execution accounts are managed by their organisation admin

Limits

  • Web cases tagged requires-auth, which need the environment's stored credentials, are refused so those credentials never go into a prompt; on testingops.ai no other automated runner takes them
  • Parallel runs are bounded by your RunningOps package and quota and by your TestingOps package
  • A step times out after 5 minutes unless the organisation sets its own step limit (0.5 to 60 minutes) or a scenario limit (1 to 360 minutes)
  • TestingOps records RunningOps' verdict; it does not re-judge the run
  • Browser and device options inside RunningOps, such as Browser Use Cloud, Browserbase, BrowserStack or Sauce Labs, are set up in RunningOps, not in TestingOps

Example

Example: how a team uses it

Illustrative scenario, not a customer story.

A retail team runs its checkout regression plan on RunningOps every night. One morning a case fails at the payment step.

The run shows RunningOps' verdict for each step, the screenshot of the declined-payment message, the video and the console log. The tester opens a defect from the failed result, and it arrives with the steps, the failed step marked and the evidence attached.

Frequently asked questions

Why is RunningOps the only automated runner on testingops.ai?

Each TestingOps server has an execution policy, an environment setting that lists the runners it may use. On testingops.ai it allows RunningOps and manual runs, so other runner cards are shown as disabled. The setting applies to the whole server, not to a plan or a company.

Do I need an AI model key in TestingOps for runs?

No. RunningOps plans and runs cases with the models available to your RunningOps organisation, which can include the AI provider keys TestingOps syncs to it. You can choose the model per run; left empty, RunningOps uses its default.

What evidence does a run bring back?

RunningOps' verdict for each case and step, step screenshots, an MP4 recording, the console log and, when RunningOps provides one, a generated Playwright spec for web or a Maestro flow for mobile. While a run is going, Watch live opens RunningOps' live view.

Can RunningOps test mobile apps?

Yes. Android and iOS cases for one app go to RunningOps as a batch. RunningOps first opens the app once and looks for a forced update, maintenance or a service error; if something blocks the batch, your setting decides: Ask me, Use the latest build, Update from the store or Continue.

What happens to runs made on a runner the policy no longer allows?

Rerunning them sends the cases to RunningOps instead, and a rerun never falls back to a manual run. A new run that explicitly asks for a disallowed runner is refused.

Can RunningOps run on BrowserStack, Browserbase, Browser Use or Sauce Labs?

Those are options inside RunningOps: it lists Browser Use Cloud as an engine, Browserbase and BrowserStack as cloud browsers, and BrowserStack App Automate and Sauce Labs as mobile device providers. They are set up in RunningOps, and TestingOps shows the provider RunningOps reports for each run. Connecting those services to TestingOps directly is a separate path that the execution policy switches off on testingops.ai.

Why was a case refused with requires-auth?

The case is tagged requires-auth, which means it needs the environment's stored credentials. RunningOps receives the steps as a prompt, so TestingOps refuses such web cases rather than writing credentials into that prompt.

Next step

See a run end to end: request a demo and bring a test case from your own application.

Request a demoMore test execution integrations