TestingOps.ai

Issue trackers, wikis and code · Native connection

Azure DevOps and TestingOps: AI test agents on Azure Boards work items

TestingOps reads the work items of an Azure DevOps project into its knowledge base, lets the Tester Agent act when a work item changes state and reports defects as Bug work items. Test cases themselves go through the separate Azure Test Plans connection.

  • Native connection
  • Available with limits
Request a demo Microsoft Learn: use personal access tokens
Connection
Native connection
Writes back
Within what you allow
Last checked
7 Nov 2026

In short

Azure DevOps and TestingOps in short

The Azure DevOps connection links a team project in your organization to a TestingOps project with a personal access token. Its work items become requirements that agents search when they write and review test cases.

With service hooks for work items created, updated and commented on, the Tester Agent can refine an item, propose acceptance criteria, write draft cases and run the cases linked to it when it reaches your test state, and replies in HTML comments.

Two gaps to know: commits in Azure Repos are not searched for impact analysis, and the agent changes a work item's state rather than its board column.

What it does

What it does for your testing

  1. 1

    Work items become requirements

    Input
    Work items of the team project, of every type, with title, description, state and type
    TestingOps
    Keeps the 300 most recently changed per sync as requirements, and the README of the project's Git repository as know-how
    Output
    Context agents use when they write and review cases
  2. 2

    Acceptance criteria become draft cases

    Input
    The Acceptance Criteria field, or an acceptance criteria section in the description
    TestingOps
    Writes draft cases so that each criterion is checked by at least one case
    Output
    Draft cases linked to the work item, ready for review
  3. 3

    A state change starts agent work

    Input
    A work item moved to the state your board rule names
    TestingOps
    Runs the rule: refine, propose criteria, write cases or run the linked cases
    Output
    HTML comments, approved description sections and tags on the work item
  4. 4

    A defect becomes a Bug

    Input
    A TestingOps defect
    TestingOps
    Creates a Bug work item in the linked project through Report to…
    Output
    The work item link stored on the defect

Step by step

How it works

  1. Connect with a PATUnder Integrations → Project tools, open Azure DevOps and enter your organization URL (https://dev.azure.com/<organization>) as the base URL and a personal access token.
  2. Link a team projectIn Knowledge Base, add the project under Issue trackers. Each sync reads the 300 most recently changed work items, hourly and on Sync now.
  3. Add service hooks for the Tester AgentGenerate a webhook address on the connection, then in Project settings → Service hooks create Web Hooks subscriptions for Work item created, Work item updated and Work item commented on.
  4. Describe what happens in each stateIn Board rules, write one sentence for each state; the wizard reads the states of your work item types.
  5. Agents write backWith the link's write switch on, the agent comments in HTML, writes approved sections into the description, adds tags and changes the state.

Write @testingops in a discussion comment to talk to the agent. The agent moves work items between states, not between board columns.

Connect

Setup

  1. In TestingOps, open Integrations → Project tools, select Azure DevOps, enter https://dev.azure.com/<organization> as Base URL and a personal access token, then select Test connection and Connect.
  2. In Knowledge Base, pick the project, select + Add source under Issue trackers, choose the connection, select the team project and select Link to RAG.
  3. Turn on Agents may write to this tracker on the source if agents should comment, tag or change states.
  4. Optional: select Generate webhook address on the connection and add three Web Hooks subscriptions in Project settings → Service hooks, for Work item created, Work item updated and Work item commented on, all with that address.
  5. Open Board rules and write what the agent does in each state.

Documentation: Microsoft Learn: use personal access tokens · Microsoft Learn: webhooks with Azure DevOps

Where you connect it in TestingOps: Integrations → Project tools → Issue trackers, wikis and code → Azure DevOps

Use cases

Use cases

Refine a user story before the sprint

Trigger
A User Story reaches your refinement state
Uses
The work item, its Acceptance Criteria field and the nearest knowledge-base entries
Agent activity
Proposes a user story, Given/When/Then acceptance criteria and open questions where the item does not say enough
Result
An HTML comment that a person approves, edits or rejects in the discussion
Condition
Approve mode is the default

Run the linked cases when an item reaches test

Trigger
A work item moves to your test state
Uses
The cases linked to the work item
Agent activity
Runs them on your execution provider and comments the counts with a link to the run
Result
A comment with passed and failed counts and, if your rule allows, a state change
Condition
Azure Repos commits are not searched, so selection uses the cases linked to the item

File defects into Azure Boards

Trigger
A tester selects Report to… on a defect
Uses
The defect's fields, from severity and environment to its steps and expected versus actual result
Agent activity
Creates a Bug work item in the linked project
Result
A Bug linked from the TestingOps defect
Condition
The token must be allowed to create work items

Run the plan after a release deployment

Trigger
Azure DevOps reports a completed release deployment
Uses
A Run on deploy trigger on a TestingOps test plan and a Release deployment completed service hook
Agent activity
Starts the plan's run when the deployment succeeded
Result
A run on RunningOps with a pass or fail verdict
Condition
The service hook sends the trigger key as an Authorization: Bearer header

Capabilities

Capabilities and limits

What is read

  • Work items of the linked team project, every type: title, description (HTML as text), state, type and change date; the 300 most recently changed per sync
  • The README of the project's Git repository: the repository named like the project, or its only repository
  • On demand for agents: a work item with comments, search in the project, the states of its type, and files of the repository
  • The Acceptance Criteria field, read by the Tester Agent

What is produced

  • Requirements and a README document in the project's knowledge base
  • Refinement proposals and draft cases linked to a work item
  • Test plan runs started by release deployments

Where it can write

  • Bug work items through Report to… on a defect
  • With "Agents may write to this tracker": comments, state changes, new work items, and title, tag, assignee or description changes
  • Tester Agent: HTML comments, approved sections in the description and tags
  • Not written: board columns, test cases (use the Azure Test Plans connection) and code

How it connects

  • Organization URL and a personal access token; there is no sign-in option
  • Tester Agent: Web Hooks service hook subscriptions to an address generated on the connection
  • Run on deploy: a Release deployment completed service hook

Account and permissions

  • TestingOps reads and writes with the permissions of the token's owner
  • A project administrator creates the service hooks
  • Comments, tags and state changes by agents need the link's "Agents may write to this tracker" switch

Limits

  • Azure Repos commits are not searched for impact analysis
  • The agent changes states only, not board columns
  • 300 work items per sync; comments and attachments are not stored
  • Personal access token only

Example

Example: how a team uses it

Illustrative scenario, not a customer story.

A team plans in Azure Boards. When a Product Backlog Item moves to Ready for QA, the Tester Agent reads its Acceptance Criteria field, notices that the criteria do not say what happens when a discount code has expired, and comments a proposal with that open question in the discussion.

After the product owner approves in the discussion, the agent writes the approved sections into the description and drafts three cases linked to the item. A later regression failure is reported with Report to… and appears as a Bug in the same project.

Frequently asked questions

Which acceptance criteria does the agent read?

The work item's Acceptance Criteria field and an acceptance criteria section in its description. Approved criteria are written back into the description as managed sections, in HTML.

Does it use Azure Repos commits to pick tests?

No. The Azure Repos client cannot list commits, so the agent selects the cases linked to the work item. Agents can still read the README and open files of the project's repository.

Can the agent move cards on the board?

It changes the work item's state. It does not set the board column itself.

Where do test cases go?

Into Azure Test Plans, through its own connection. The Azure DevOps connection covers work items, comments, tags and bugs.

Can I sign in instead of using a token?

No. The connection uses your organization URL and a personal access token.

How do I talk to the agent on a work item?

Write @testingops in a discussion comment. The agent answers there, and an approval or edit of its pending proposal works the same way.

Next step

Planning in Azure Boards? Request a demo and bring a project to try the connection on.

Request a demoMore project tools integrations