On Belay

Custom agents

Custom agents are AI workers you define once and run on demand or on a schedule. Each one runs as a principal whose group memberships gate exactly what it can touch — so an agent can do real work without ever exceeding the access you've granted.

What custom agents are

A custom agent is a reusable AI task. It has a name, a system prompt, a model, a set of permitted integrations, and a run-as principal that determines what it can access. You build it in a guided wizard, then run it yourself or let it run on a schedule.

Agents live at Dashboard → Agents. Every run is recorded, so you always have a history of what the agent did, how long it took, and what it produced.

Before you start

Two things need to be in place:

  1. At least one connected integration. The wizard can't build a useful agent with no tools to call. Connect what the agent needs first — see Integrations overview.
  2. A principal for unattended runs. To deploy a scheduled agent, you (as an admin) first create a Service Identity at Dashboard → People → Service Identities and add it to the groups that hold the integration access the agent needs. Without a run-as principal, deploy is blocked. (See Core concepts for what a Service Identity is.)

The build wizard

Building an agent walks through five steps. It does not ask for the agent's name or goal up front — those come at the Build step.

Step 1 — Outcome category

Pick what kind of work this agent does: Reporting, Analysis, Recommendations, or Actions. This is a category choice only — you do not name the agent here. The category steers the integrations and prompt templates you'll see next.

Step 2 — Choose your tools

Select which integrations the agent may use. This is an integration-level choice — there's no per-operation picker in the wizard. Integrations your groups can't reach show as locked or greyed out; connect and grant them first if you need them.

Step 3 — Starting prompt

Pick a starting prompt. You can choose a prebuilt template, a template matched to your outcome and selected tools, or an AI-generated suggestion. You'll refine the prompt in the next step.

Step 4 — Build

This is where the agent takes shape. You set:

  • Name — what the agent is called.
  • System prompt — the agent's instructions. An AI Prompt Helper panel can draft or improve it. A safety check blocks deploy if it detects credentials (keys, tokens, passwords) pasted into the prompt — credentials belong in integrations, never in prompt text.
  • Model — see Models below.
  • Run-as principal — the identity the agent acts as (admins only; see below).
  • Output delivery — where results are sent (optional).
  • Run settings — Rich HTML reports, Max price per run, and Visibility.

Step 5 — Schedule & deploy

Choose a trigger and schedule, run Test this agent (a one-off dry run so you can see the output before committing), and confirm deploy.

To begin, go to Dashboard → Agents and start a new agent to launch the wizard.

Run-as principal

A principal is the identity the agent acts as — the User or Service Identity whose permissions the agent borrows when it runs. Every active agent runs as exactly one principal: a real User or a Service Identity, never both and never neither. On Belay enforces this at the database level — an agent can't deploy without precisely one set.

The principal's group memberships are what gate the agent's access. When the agent calls an integration, the proxy authorizes the call as that principal — so the agent can only reach the integrations and operations the principal's groups permit. Put the principal in the right groups and the agent inherits exactly that scope, no more.

Only admins can choose the run-as principal. A member who builds an agent has it forced to run as themselves. Assigning a Service Identity automatically adds that identity to your functional groups that own the agent's selected integrations, so the access lines up.

Auto-add skips On Belay's own background groups. Some groups are created and managed by the platform for its own background jobs — the cache sync, and each fieldset's runtime identity. They are named things like "Cache Sync Agent Background" and hold a grant for every integration you have connected, so joining an agent's identity to one would silently hand that agent your entire integration surface. Auto-add therefore never targets them. Your own groups are unaffected, including any you happen to have named "… Background".

One consequence worth knowing: if the only group granting an integration is one of those platform background groups, auto-add has nowhere to put the identity and adds it to nothing. The agent then can't call that integration. The fix is to grant the integration to one of your own functional groups at Dashboard → Groups and save the agent again.

Service Identities are non-human principals built for unattended runs. Create and manage them at Dashboard → People → Service Identities.

Models

The wizard lets you pick the model the agent runs on:

  • Haiku (claude-haiku-4-5-20251001) — "Fast & economical." Good for high-frequency, lightweight runs.
  • Sonnet (claude-sonnet-4-6) — "Balanced." The recommended default for most agents.
  • Opus (claude-opus-4-8) — "Most capable." Use it for complex analysis or multi-step synthesis.

Pick the lightest tier that does the job well; you can change it later.

Triggers and schedules

  • Manual — run the agent on demand from its page whenever you need it.
  • Scheduled — run it automatically on a schedule.
  • Event-triggered — coming soon; this option is currently disabled.

Schedule presets are: Every hour, Daily at 9am, Weekdays, Weekly, or a custom cron expression.

Schedules run in your organization's own timezone. "Daily at 9am" means 9am where you are. You can see and change that timezone at Settings → General → Timezone; if your store is connected, On Belay inherits the timezone from it, and setting one here overrides that. Daylight saving is handled for you — a 9am schedule stays at 9am local across the spring and autumn changes.

Agents also resolve dates against that same timezone. When an agent reads "yesterday" or "the last 30 days" in your prompt, it means yesterday where you are, and it states the exact date range it used in its output so you can check it.

Every time On Belay shows you a run time it is stamped in that timezone and names it — the "Next run" line on the agent's page and on a teammate job's settings page, and the "When" line of the Slack alert you get if a run fails. So a next run reads like Aug 21, 2026, 9:00 AM MDT (America/Denver) rather than a bare time you have to guess the clock for. If you have no timezone set and no store connected, those times are shown in UTC and say UTC.

Changed 2026-08-21. Schedules previously ran on US Eastern time for every organization, regardless of location. Existing schedules were reinterpreted in your own timezone, so an agent set to "Daily at 9am" now runs at 9am local rather than 9am Eastern. For most customers that moved the run later in the morning. Nothing else about the schedule changed.

Approvals

There is no per-integration "ask me first" toggle on an agent. An agent requests a human decision by asking for one explicitly — using the request_approval tool from its prompt — not through a configuration switch. Write that instruction into the system prompt if you want the agent to check in before a particular kind of action.

Approvals across On Belay are reviewed by your Owner group, at Dashboard → Approvals or with the Approve / Deny buttons in Slack when the Slack bot is installed. For exactly what an approval does — and what it doesn't do to a task already in flight — see Governance & approvals.

Output delivery

An agent can deliver its output to one or more destinations. Delivery is optional — with no destination set, results are viewable only in the Run Log.

Prerequisites

For delivery to work, two things must be true:

  1. The integration must be connected to your organization at Dashboard → Integrations.
  2. The agent's run-as identity must have access to it through a group. If the agent runs as a Service Identity, that identity must be added to a group that grants the integration. (This is an admin action — see Functional groups & permissions.)

If either condition isn't met, the run itself still completes — the agent does its work and its output is saved — but the delivery is recorded as failed. Check the Output delivery section of the agent's Configuration tab, which shows the error from the last run.

Setting up each destination

  • Slack — A channel picker is provided in the form; select the channel you want. You'll need the Slack integration connected and the run-as identity granted to it.

  • Microsoft Teams — Right-click the channel in Teams and select Get link to channel. That link contains both IDs you need:

    • The URL parameter groupId= is your Team ID
    • The URL parameter threadId= is your Channel ID (it looks like 19:…@thread.tacv2)

    Both fields must be filled before the "Add destination" button becomes active. This is the most common place to get stuck — if the button is greyed out, check that you've pasted both IDs into their fields.

  • Notion — You need a page ID. To find it: open the page you want, click the three-dot menu at the top right, select Copy link, and paste it somewhere. The URL will look like https://www.notion.so/My-Page-<PAGE_ID>?v=.... The PAGE_ID is the long hex string at the end. Paste that string into the form.

  • Google Drive — You need a folder ID. Open the folder you want to upload to and look at the URL bar. It will be https://drive.google.com/drive/folders/<FOLDER_ID>. Paste the FOLDER_ID (the string after /folders/) into the form.

  • Webhook — Paste a HTTPS URL. On each run completion, the agent will POST the output as JSON to that URL. Your endpoint is responsible for handling the payload.

  • Email (owner) — Results are emailed to the agent owner's email address. No configuration needed.

Run settings

Visibility

  • Private — only admins and you (the creator) can see and manage the agent.
  • Company — shared with the whole org, so colleagues can view the agent's run outputs and artifacts (they still can't edit, delete, or run it unless they're the creator or an admin).

Visibility is an admin-only control. A member's agent is forced to Company.

Max price per run

A per-run cost cap, between $0.50 and $100. Leave it blank to use the $10 default. A run hard-aborts when it reaches 100% of the cap, and does a graceful wrap-up at around 80% so you still get partial output rather than an abrupt stop.

Rich HTML reports

Optional. When enabled, each run ends with a branded, self-contained HTML report attached to the run, alongside the plain output.

Shared templates (across organizations)

Some agents are useful to more than one company. On Belay has a shared template library at Dashboard → Agents → Shared templates, where organizations can publish an agent for anyone else on the platform to copy.

Sharing is anonymous and works on snapshots — read both sections below before you publish, because publishing puts your text in front of every other organization on On Belay.

Using a shared template

Open Dashboard → Agents → Shared templates, browse or filter by category, and select Use this agent. That opens the normal build wizard with the template's name, goal, system prompt, and tool selection already filled in.

What you get is a draft you own. Nothing runs automatically. You review and edit every field, pick your own run-as principal and delivery settings, then save and deploy it like any other agent — the normal rules still apply, so a member's imported agent is still forced to Company visibility and still runs as that member.

Two things to check before you run an imported agent:

  • Read the prompt. It was written by another organization, and once you deploy it, it runs with your credentials and data access. Treat it the way you'd treat any instructions from outside your company.
  • Check the integrations. When it can determine which integrations your organization has connected, the wizard flags any the template needs that you're missing. The agent can't do that part of its job until you connect them at Dashboard → Integrations. If that check can't run, the wizard stays quiet rather than guessing — so it's worth confirming the template's listed integrations yourself before you deploy.

Any org member can import a template. Publishing is admin-only.

Publishing one of your agents

Admins only. Open the agent, go to the Configuration tab, and find the Sharing panel.

Selecting Share as template opens a composer where you edit exactly what gets published — name, description, goal, and system prompt. This is the important part: On Belay publishes the text you submit in that composer, not your agent's live prompt. Anything you delete there is never published, and editing your agent afterwards does not change the published template.

The composer scans all four fields as you type and tells you what they contain — email addresses, URLs, dollar amounts, Slack channels, and company-looking names — and highlights each one it finds in the system prompt itself. For the name, description, and goal it reports a count rather than highlighting. Anything that looks like a credential is rejected outright, in any of the four fields. You tick a confirmation box before Publish is enabled.

What crosses over: the name, description, goal, and system prompt you approved, plus the agent's integration and operation selection.

What never crosses over: your organization's name, your name, your run-as principal, visibility, schedule, delivery destinations, cost cap, model, memory, and knowledge sources. Other organizations see the template as "shared by another organization" and nothing more.

But be clear about what "anonymous" does and doesn't mean. It hides who published the template. It does not hide what the text says. If your prompt mentions a customer, a supplier, a margin, an internal project name, or a competitor, everyone who reads the template sees it. Edit the text down to something generic before you confirm.

Updating and revoking

Republish with changes takes a fresh snapshot and increments the template's version. Anyone who imported an earlier version sees a note that the source has been updated, with a link to read the current one. Their agent keeps running on the copy they already have — nothing changes underneath them.

Remove from library pulls the template from the library so nobody new can find or copy it. It does not reach into other organizations: anyone who already imported it keeps a fully working agent. Once you've published something, treat it as out of your hands — revoking limits future copies, not existing ones.

Importing an agent spec (from a Claude conversation)

If you built a report conversationally in Claude using the On Belay connector, you don't have to rebuild it by hand in the wizard. Ask Claude for an agent spec at the end of the session, then paste it at Dashboard → Agents → New agent → Import agent spec.

This works because a Claude conversation over the On Belay connector already goes through the same proxy and the same permission model an agent run uses. So the conversation already knows which operations actually returned your data — the part that's tedious to reconstruct from memory.

The prompt to give Claude

The import page carries this prompt with a Copy prompt button — open Import agent spec and expand Get the spec from Claude. It is reproduced here so you can read it before you start.

It only works in a Claude conversation using the On Belay MCP connector. That connector is how Claude knows which operations it actually called; without it, Claude would be guessing.

Emit an On Belay agent spec for the report we just built, so I can import it as a scheduled agent.

Rules:

1. Report the operations you ACTUALLY called in this conversation — not operations you
   considered, and not ones you think would be useful. If you called an operation and it
   returned nothing useful and you dropped it from the report, leave it out. `operationKeys`
   holds the operation keys exactly as the On Belay catalog names them (the `key` field, e.g.
   `list_orders` — not a URL path, not a tool name, not a guess). `integrationSlugs` holds the
   slugs those operations belong to. If you are unsure of an exact key, call
   `list_my_integrations` or `get_operation_schema` and use what it reports rather than
   inferring one.

2. Write the report STRUCTURE into `systemPrompt` as INSTRUCTIONS, in prose. Do NOT emit HTML,
   CSS, markdown tables, or any markup — On Belay applies its own report styling and the output
   is expected to look different from the artifact you produced here. Describe the sections,
   which metrics go in the KPI band, what each chart ranks or trends, the table's columns in
   order, and what the callouts should say. Also carry over the analysis itself: the time
   window, how each metric is derived, the filters and exclusions we settled on, what to do
   when a source returns nothing, and any judgement calls we made along the way. Write it as
   instructions to an agent that has never seen this conversation.

3. `systemPrompt` must contain NO credentials, API keys, tokens or passwords. Reference
   integrations by name only.

4. `categoryKey` is one of: reporting, analysis, recommendations, actions.

5. `artifactEnabled`: true if this report deserves the rich HTML report treatment, false if
   plain text is enough.

6. `notes`: one line of provenance — what this conversation was and roughly when.

7. Emit ONLY the JSON object below. No commentary before or after it. No extra fields — an
   unrecognised field is rejected, not ignored. Do not include a schedule, a run-as identity,
   or output delivery: those are configured on the draft in On Belay, not by the spec.

{
  "specVersion": 1,
  "name": "string — short agent name",
  "goal": "string — one sentence on what this agent produces",
  "categoryKey": "reporting | analysis | recommendations | actions",
  "systemPrompt": "string — the analysis, the framing, and the report structure as instructions",
  "integrationSlugs": ["string"],
  "operationKeys": ["string"],
  "artifactEnabled": true,
  "notes": "string — provenance"
}

Claude replies with a single JSON object. Copy the whole thing — the surrounding ```json fence is fine if it comes with one — and paste it into the box on the import page.

What you get

Importing always creates a draft. It is never active, never scheduled, and has no run-as principal. You still choose the run-as identity, the schedule, and the delivery destinations on the draft before activating it — importing a spec skips the typing, not the governance.

Importing is admin-only, because a spec sets the integration and operation selection in one shot.

Importing never grants anything. A spec selects operations. It does not connect an integration, does not add anyone to a group, and does not widen any permission. If the spec names an integration your org hasn't connected, you get a warning and a working draft — not a new connection.

The output will look different, and that's intended

The report's structure carries over as instructions — which metrics go in the KPI band, what the charts rank, which columns the table has. The visual styling does not: On Belay applies its own report design so every agent report in your organization looks like one product. Expect the same data and the same structure in On Belay's house style, not a pixel copy of the Claude artifact.

Errors vs warnings

The import distinguishes two things, and the difference matters:

Errors block the import and create nothing. Malformed JSON, a missing or mistyped field, an unrecognized field, a duplicate entry, an integration or operation that doesn't exist, or a credential pasted into the prompt. Each error names the field and what was expected, so you can fix the blob and validate again.

Warnings don't block anything. They tell you what will break the first scheduled run if you leave them alone:

  • An integration isn't connected. The draft imports fine. Connect it before you activate.
  • Permission wasn't checked. This one is unavoidable at import: integration access is granted to a principal's groups, and a draft doesn't have a principal yet — so there's nothing to check against until you pick one. When you choose the run-as identity, confirm its groups grant every integration the agent needs. An agent missing a grant activates cleanly and then fails with a permission error on its first scheduled run.

Execution and the run log

Agent jobs are dispatched by a background scheduler roughly every minute. Each execution becomes a run — On Belay calls Claude with the agent's system prompt and permitted tools, then records the result.

Open an agent at Dashboard → Agents and select it to see its Run Log. Each run records its status, start and finish time (which gives you the duration), the output it produced, and its cost. The exact per-run dollar cost is shown to the platform owner or on failed runs; customers otherwise see a "Billed" amount.

What an agent has learned

Every agent has a Learnings tab at Dashboard → Agents → (your agent) → Learnings. It shows, in plain sentences, the facts that agent has accumulated — and the most recent of them are sent to the agent at the start of every run, so it doesn't spend a turn rediscovering them.

Learnings come from four places, and each row says which:

| Source | What it means | |---|---| | CONFIG CHECK | Read from your own configuration — which groups grant which operations to this agent's run-as identity. No run needed; it happens whenever you open the Configuration tab. | | RUN FAILURE | Recorded automatically when a run failed, with the failure category and the suggested fix. | | AGENT-SAVED | The agent chose to write this down during a run — usually a working API path or a parameter it had to discover. | | OBSERVED CALL | Recorded when the agent actually called an integration, with the operation it used and the status it got back. |

Each row carries when it was last confirmed. Configuration checks never go stale — they are re-read from your current setup every time you open the agent, so what you see is always current. A run failure, an agent note, or an observed call older than 14 days is marked Old: historical, not wrong. An observed call stops being sent to the agent once it passes that mark, so an old failure can't keep discouraging the agent from retrying something you have since fixed.

A learning replaces the one it contradicts. If a check says an operation is blocked and a later check says it is permitted, you see one row with the current answer, never two contradicting rows.

What gets stored here. Only structural facts: the integration, the operation, the method, the path, the status, and the reason. Response data is never written, and nothing that On Belay recognises as an API key or token is either — every learning is screened on the way in and dropped if it matches, which is enforced in code rather than by policy alone.

The Learnings tab reports; it never changes your permissions. When a check finds a missing grant it names the integration and the operation and stops there. Widening a grant is an admin's deliberate act at Dashboard → Groups, and nothing on this tab will do it for you.

The badge on a run row links here, and it names where the knowledge came from — "learned from 3 prior runs", "learned from 4 config checks", or both together. The number is the count of learnings that run started with.

Related