Skip to content

/00 — Business-rule testing for engineering teams

Write your product’s rules in plain English, in chat. On every pull request, two AI agents try to break them. A person on your team makes every final decision.

For engineering and product teams. Works in Slack, Microsoft Teams, Discord, and Mattermost.

canary://shaft-01
Shaft 01Level B-7Depth −412 m
Rules typed in chat
03
Pull request that broke rule 1
#42
Caught before merge
−$4,500
Rulings flagged twice
0

/01 — The gap

Security scanners look for known patterns, like SQL injection and XSS. They do that well.

But they can’t know your product. Only your team knows that a refund can never be more than the original charge, or that a promo code works once per customer.

When a pull request breaks one of those rules, every check stays green. The bug ships, and customers or attackers find it first.

src/refunds/create.ts
export async function createRefund(chargeId, amount) {  const charge = await db.charges.find(chargeId)  if (amount > charge.amount) throw new BadRequest()  return db.refunds.insert({ chargeId, amount })}

Checks one refund at a time. Three refunds at once each pass this check.

  • Security scanner0 issues found
  • Unit tests48 of 48 passed
  • CanaryRule 1 broken: $4,500 refunded on a $1,500 charge

/02 — How it works

No config files and no test code. You talk to Canary the same way you talk to your team.

  1. Write rules in chat

    Type a rule in plain English in your team’s channel. Canary saves it and numbers it.

    @canary-red rule: A refund can never be more than the original charge.

  2. Open a pull request

    Canary reads the diff and finds which rules the change could affect.

    PR #42 · Split refunds into partial payments · touches rule 1

  3. Red tries to break them

    @canary-red writes an exploit for each rule, runs it in a sandbox, and posts the evidence in the room.

    3 parallel refunds · $4,500 refunded · rule 1 broken

  4. Blue fixes, your team decides

    @canary-blue writes a fix and Red runs the exploit again. A person on your team makes the final call, and the room remembers it.

    Fix pushed · exploit re-run · rule 1 holds · ruling saved

/03 — The room

Two agents and your team share one channel. Everything happens in the open, so anyone can follow the evidence and the decisions.

#canaryNov 4, 2026

  1. Amara Okafor

    @canary-red rule: A refund can never be more than the original charge.

  2. canary-red

    Saved as rule 1. I will test it on every pull request.

  3. GitHub

    Lucas Moreau opened PR #42: Split refunds into partial payments.

  4. canary-redRule broken

    Rule 1 broken in PR #42. I sent 3 refunds of $1,500 at the same time against one $1,500 charge. All 3 succeeded. Total refunded: $4,500.

    14:04:51.120Z POST /refunds {charge: "ch_8Kq2", amount: 150000} → 20114:04:51.121Z POST /refunds {charge: "ch_8Kq2", amount: 150000} → 20114:04:51.121Z POST /refunds {charge: "ch_8Kq2", amount: 150000} → 201refunded 450000 cents · charge 150000 cents
  5. canary-blue

    Fix pushed to PR #42. It locks the charge and checks the refund total before it saves. I added a regression test.

    + const charge = await tx.charges.lockForUpdate(chargeId)+ if (charge.refunded + amount > charge.amount) throw new Conflict()
  6. canary-redRule holds

    I ran the exploit again on the fix. 1 refund succeeded and 2 were rejected with 409. Rule 1 holds.

  7. Marcus Bell

    Question: support sometimes refunds the shipping fee on top. Is that allowed?

  8. Priya RamanRuling

    Ruling: no. Shipping is part of the charge, so the total refund can’t go above it.

  9. canary-red

    Ruling saved. I won’t flag this case again.

  10. Aiko Tanaka

    Approved and merged. Thanks, both.

Shown in Slack. It works the same in Microsoft Teams, Discord, and Mattermost.

/04 — Rules

If you can say it in one sentence, Canary can test it. These are common first rules.

  • Wallets

    Rule in chat: “A wallet balance can never go below $0.”

    How Red attacks it: Two withdrawals at the same moment, each for the full balance.

  • Promo codes

    Rule in chat: “A promo code works once per customer.”

    How Red attacks it: The same code from two sessions, then from a second account with the same card.

  • Refunds

    Rule in chat: “A refund can never be more than the original charge.”

    How Red attacks it: Several partial refunds on one charge, sent in parallel.

  • Seats

    Rule in chat: “A team on the Starter plan has at most 5 seats.”

    How Red attacks it: Invite 5 people, then accept 2 old invites that never expired.

  • Free trials

    Rule in chat: “Each organization gets one free trial.”

    How Red attacks it: Cancel, delete the organization, and sign up again with the same billing details.

  • Inventory

    Rule in chat: “Stock can never go below zero.”

    How Red attacks it: Ten checkouts for the last item, all at once.

/05 — vs Scanners

Canary does not replace your security scanner. Run both. They catch different bugs.

  • Finds SQL injection, XSS, and known CVEs

    Security scanner
    Yes
    Canary
    No, keep your scanner
  • Knows your product’s rules

    Security scanner
    No
    Canary
    Yes, you write them in chat
  • Proves each bug with a working exploit

    Security scanner
    Sometimes
    Canary
    Yes, run in a sandbox
  • Writes the fix and tests it again

    Security scanner
    No
    Canary
    Yes, Blue fixes and Red re-tests
  • Remembers your team’s decisions

    Security scanner
    No
    Canary
    Yes, every ruling is saved
  • Works where your team talks

    Security scanner
    In a dashboard
    Canary
    Slack, Teams, Discord, Mattermost

/06 — Quickstart

Canary is open source. You need Node.js 20+, Git, the GitHub CLI, and a chat workspace where you can install an app. The full guide is in agents/README.md in the repo.

  1. 01Clone the repo

    bash
    git clone https://github.com/adeyanjufuhad/canary.gitcd canary
  2. 02Start Switch on your computer

    Install Switch Console, then choose Add a server → Run a server on this computer. Switch connects the agents to your chat app.

  3. 03Connect your chat app

    Create a chat app from the Switch guide, connect it in Switch Console, and link your own account. Then make a channel, such as #canary, and invite Switch to it.

  4. 04Add the two agents

    bash
    node agents/setup-dirs.mjs

    This makes a working folder for canary-red and canary-blue. In Switch Console, add both agents, pick each one’s folder, and invite them to the channel.

  5. 05Write your first rule in chat

    chat
    @canary-red rule: A wallet balance can never go below $0.

    Red saves the rule and numbers it. Add one rule for each thing that must always be true.

  6. 06Start the watcher, then open a pull request

    bash
    node agents/tools/pr-watcher.mjs

    The watcher checks GitHub every 30 seconds and tells Red about new pull requests. Red tests each one and posts the results in the channel.

/07 — FAQ

Does Canary replace my security scanner?

No. Scanners find known patterns like SQL injection. Canary tests the rules that only your team knows. Use both.

Where do the exploits run?

In a sandbox: the pull request’s build of your app, with a test database that is reset before every run. Canary never runs exploits against production. Today the sandbox runs on your own computer, so only point Canary at pull requests from people you trust.

Can Canary find bugs nobody wrote a rule for?

No. Canary tests the rules your team writes. Keep your scanner and your tests for everything else, and add a rule whenever your team agrees on something that must always be true.

Which chat apps does it work with?

Slack, Microsoft Teams, Discord, and Mattermost. Canary runs on Switch, which connects the agents to the chat app your team already uses.

Who decides when an agent is wrong?

A person on your team. Anyone can dispute a result in the thread. The ruling is saved, and Canary follows it on every future pull request.

How do I write a good rule?

Use one sentence that states what must always be true. For example: “A promo code works once per customer.” Avoid rules about how the code works inside.

Is Canary free?

Canary is open source under the MIT license. The agents run on your own AI model account, so you only pay your model provider for what you use.

/08 — Get started

Canary is open source under the MIT license. Clone it, connect your chat app, and write your first rule today.