Plugin Tools

Plugin tools are registered with the function manager and the AI calls them like any built-in tool. The format is the same across all plugins (see core/plugins/memory/tools/ for examples of core tools, or core/plugins/email/tools/ for a plugin tool).

For simple tool creation without a full plugin, see TOOLMAKER.md.

Tool File Format

ENABLED = True
EMOJI = '🔧'
AVAILABLE_FUNCTIONS = ['my_tool_do_thing']

TOOLS = [
    {
        "type": "function",
        "is_local": True,
        "function": {
            "name": "my_tool_do_thing",
            "description": "Does the thing",
            "parameters": {
                "type": "object",
                "properties": {
                    "target": {
                        "type": "string",
                        "description": "What to do it to"
                    }
                },
                "required": ["target"]
            }
        }
    }
]

def execute(function_name, arguments, config, plugin_settings=None, credentials=None):
    """Called by function manager.

    The function manager inspects your signature and passes what you accept:
    3 args (function_name, arguments, config), or add a 4th (plugin_settings —
    this plugin's stored settings dict) and/or a 5th (credentials — the
    credentials manager). Declare only what you need; 3-arg is the minimum.

    Args:
        function_name: Which function was called
        arguments: Dict of parameters
        config: System config
        plugin_settings: This plugin's saved settings (4th arg, optional)
        credentials: Credentials manager for resolving secrets (5th arg, optional)

    Returns:
        (message: str, success: bool) tuple
    """
    if function_name == "my_tool_do_thing":
        target = arguments.get("target", "")
        return f"Did the thing to {target}", True
    return "Unknown function", False

Required Exports

Categories — the words an agent's toolset is made of (toolset: 'web, files'), the groups of the Toolsets picker, and the scope a tool needs: web · memory · knowledge · people · goals · files · system · devices · comms · media · meta_danger · agents. memory, knowledge, people and goals tools need that scope; a lane that closed it (an agent with memory_scope: none) sheds them instead of carrying tools that answer "scope disabled". A tool in any other category is assumed to need the ONE scope its plugin registered, if it registered exactly one (an email tool needs email_scope). meta_danger is for tools that change Sapphire herself — prompt, voice, model, toolset, scene — the name is the warning. Pick the plain truth; a plugin that fits nowhere uses its own name (bitcoin, wordpress do).

Manifest Declaration

"capabilities": {
  "tools": ["tools/my_tool.py"]
}

Schema Flags

Inside each tool's schema dict:

TOOLS = [{
    "type": "function",
    "is_local": "endpoint",   # calls an external API — refused in private chats
    "network": True,           # shown as a network tool in the UI
    "function": { ... }
}]

Neither flag routes traffic. Routing is decided by the network facade: when the SOCKS proxy is on, all WAN traffic from the process rides the proxy regardless of these flags, and LAN traffic goes direct. See Networking from Plugins.

Returning Images

A tool that wants the model (and the user) to see a picture returns the images contract instead of a string:

from core import images as ci

def execute(function_name, arguments, config):
    raw = capture()                       # bytes of any image format
    return ci.result("Here's the camera.", [ci.for_chat(raw)]), True

• ci.result(text, images, display_only=False) builds {"text", "images": [{data, media_type, display_only}]}. display_only=True = the user sees it, the model never does.
• The user sees every image forever; the model sees an image the turn it arrives, and again for the next Images › Image memory turns turns (default 3 — the vision window replays exactly what it saw live). Older images it re-views by handle.
• Core saves each image to the chat's tool_images table (encrypted for vaulted chats), prepends the UI marker, and appends an (image img:<id>) receipt line to your text. Any image tool accepts that handle — ci.resolve("img:<id>") gives you the bytes; she hands it to memory_view_image(image_id=), memory_save_image, telegram_send_image.
• ci.for_chat(raw) is the one resize (EXIF-upright, ≤1536px, JPEG); ci.contact_sheet(list_of_raw) the one numbered grid. Don't write your own.
• ci.resolve(source) understands img:<id>, doc:<N> (Mind Palace library image), an absolute path, or an http(s) URL (fetched through core.net, 20MB cap). Raises ci.ImageError with a user-readable message.
• Many pictures, one image: show the model ONE contact sheet and keep the individuals as handles — h = ci.stash(raw) stores the bytes in the chat's image store now (same table, vault, cascade and route as every tool image; visible=False so the vision window never replays it) and returns img:<id>. Name the handle in your numbered text (3. title — img:abc.jpg) so she can look again (memory_view_image(image_id=)) or memory_save_image any one of them. Never cache image bytes anywhere else.
• Tiles for the user without spending model tokens: append ONE marker, <!--GALLERY:{"title": "…", "items": [{"handle": "img:<id>", "full": "<lightbox url>", "title": "…", "page": "<source url>"}, ...]}-->. Tiles with a handle are served from the store (vault-aware, offline); an item may carry thumb instead for a URL you already serve. Items render as a numbered row — number them the same way as your sheet. The older array forms ([{thumb, full, title, page}], ["url", …]) still render. Browser-facing external image URLs go through ci.proxied(url) — the browser never hot-links a third party. The marker is stripped from the model's copy.
• All of the above in one call: images, tail = ci.gallery(title, entries, view=True) — entries in sheet order, each {raw, title, page, full?, thumb?, show?, stash?}. It stashes every raw (sets entry['handle']), builds the marker and the sheet (or the one image), and returns her pixels plus the note-and-marker tail. Write your own numbered lines with the handles in hand and return ci.result(text + "\n" + tail, images) when images is non-empty, else the plain string. stash: False + thumb = a row you already serve (library); thumb alone = the fallback tile when the stash refuses. The web image search, the library query and the local-folder view all ride it — don't hand-roll a fourth.

Returning Files

A tool that wants the user to have a file (a song, an export, a recording) serves it from one of its own routes and appends ONE marker line to its result:

from core import attachments

def execute(function_name, arguments, config):
    row = attachments.marker("Copper Light", [
        {"url": "/api/plugin/my-plugin/song/ab12.mp3", "name": "copper-light.mp3"},
        {"url": "/api/plugin/my-plugin/song/ab12.mid", "name": "copper-light.mid"},
    ])
    return "Saved Copper Light.\n" + row, True

• The chat renders the marker as a row under the tool result: audio files (mp3, wav, ogg, oga, m4a, flac) get a player, every file gets a download button named by name. The player fetches nothing until play is pressed.
• URLs are your own plugin routes only (/api/plugin/<name>/...). attachments.marker raises ValueError on anything else, and the browser renderer drops it too. A tool result can carry text from the open web, so the marker can never point the browser at a third party.
• The marker is UI-only. It stays in history for the browser and is stripped from every copy the model reads, so say in plain words what you saved. Up to 12 files per marker.
• The same marker works in the text of a turn your plugin starts itself (see Starting a turn yourself). The row then shows on that message.
• Serve the bytes with a FileResponse so seeking works, and keep them where the user's data lives (user/plugin_state/<name>_*). Files outside the chat database are not covered by a private chat's vault, so refuse to write them when scope_private is set.

Starting a turn yourself

A plugin that has something for the chat outside a tool call — a daemon's result, a device's take, a finished background job — puts it in the chat's inbox and the AI answers it as a turn of its own, in order, as soon as she has finished writing whatever she is on now (a streaming-TTS tail is cut over, not waited for). A machine's item waits its turn — through a busy chat, a locked database, even a vault lock (it is held in memory until the unlock) — and is dropped only when the chat is gone, when it was queued while the chat was private and the chat is public by the time it would run (privacy rolls downhill), when its ttl passed, when the user took it out, or on a restart. Each of those calls your on_drop(why) with item.drop_kind set (gone, privacy, stale, removed, restart, unreachable), so you can leave your own record where that is right — never for privacy/gone/sealed. No browser needs to be open.

from core.chat import inbox

inbox.tell(chat_name, "Take 3 is in: 42 notes, 9.8 s.",
           source='my-plugin',
           header_line=inbox.header('My plugin', 'take', 'a recording finished'),
           speak='speakers')          # or 'browser', or None for text only

• header_line is the one line above your text that says who wrote it and that the user did not type it — [My plugin (take) — a recording finished; not typed by the user]. Every door that isn't the user typing leads with one (agents, satellites, MIDI, phone calls); use inbox.header(who, kind, what) so yours reads the same.
• inbox.ask(chat, text, source, timeout=...) is the same door when you need her reply back as the return value (it blocks until she answers or the timeout passes).
• Returns from a machine never fold with each other: each gets her own reply. Only the user's own typed turns fold. Don't pass coalesce=True.
• Private chats take inbox items like any other (the turn runs on the chat's privacy at that moment — a private chat answers locally). Only a chat that does not exist refuses with inbox.InboxRefused; catch it. The one door that refuses private chats is MCP (ask/tell over the network), which is not yours.
• core.cadence.fire_once(chat, text, ...) is the older door that raises ChatBusy instead of waiting. Use it only when a stale result is worse than no result (a timer tick that will fire again anyway). Before 2026-10-06 every plugin used it and lost results whenever she was mid-sentence.

Networking from Plugins

Make HTTP calls through the network facade instead of bare requests:

from core import net

r = net.get(url, timeout=10)       # requests-shaped: get / post / put / delete / request
s = net.session_for(url)           # pooled session for loops (lane-fixed — one per URL)
s = net.wan_session()              # WAN-lane session when you have no URL yet (browser profile)

The facade classifies each URL's host as LAN or WAN — syntactically, without ever resolving DNS:

• LAN (loopback, private-range IPs, *.local / *.lan / *.home.arpa, single-label hostnames, registered direct hosts) goes direct: no proxy, environment ignored. Redirects are refused by default on the LAN lane so a LAN device's 30x can't hop off-proxy silently — pass allow_redirects=True to opt in.
• WAN (every real FQDN and public IP) honors the proxy environment: proxied while SOCKS is on, plain direct when it's off. Errors never fall back across lanes — a dead proxy fails loudly instead of leaking direct.

If your plugin talks to LAN gear through a public-looking hostname (a DDNS name, an FQDN that resolves inside the house), register it so it classifies as LAN:

from core.socks_proxy import register_direct_hosts

register_direct_hosts(lambda: ["ha.my-house.example.net"], owner="my-plugin")

The provider is a zero-arg callable returning hostname strings, keyed by owner (use your plugin name) — re-registering replaces rather than accumulates, and core unregisters it when your plugin unloads. Raw requests callers still work (a process-wide proxy-environment belt covers them while SOCKS is on), but the facade is the exact lane; the belt is the floor.

Multi-Account Scope Support

Tools that support multiple accounts (email, bitcoin, etc.) can read the active scope:

from core.chat.function_manager import scope_email

def execute(function_name, arguments, config):
    account = scope_email.get()  # returns active account name (ContextVar)
    creds = load_credentials(account)
    # ... use account-specific credentials

Available scope ContextVars: scope_rag and scope_private are always present (core). The rest — scope_email, scope_bitcoin, scope_knowledge, scope_memory, scope_people, scope_goal, scope_github, etc. — resolve via __getattr__ against the scope registry and only exist while the owning plugin (memory, email, bitcoin, github…) is loaded, so importing one is safe from a tool in that same plugin.

Reading Plugin Settings

Tools can load their own plugin's settings:

import json
from pathlib import Path

def _load_settings():
    path = Path("user/webui/plugins/my-plugin.json")
    if path.exists():
        return json.loads(path.read_text())
    return {}

Or via the plugin loader (merges with manifest defaults):

import json
from core.paths import PROJECT_ROOT

DEFAULTS = {"timeout": 30, "max_results": 10}

def _load_settings():
    path = PROJECT_ROOT / "user" / "webui" / "plugins" / "my-plugin.json"
    settings = DEFAULTS.copy()
    if path.exists():
        try:
            user = json.loads(path.read_text())
            settings.update(user)
        except Exception:
            pass
    return settings

Finding the project root

Import paths from core/paths.py — never count .parents up from __file__:

A counted chain that comes up one level short still lands on a real folder (core/ or user/), so nothing errors; your files just quietly go to the wrong place. Standalone scripts and child-env modules that can't import core should walk upward until they find sapphire.py and use that folder as the root.

Dynamic Tool Descriptions

A tool's description is what the AI reads to decide how to use it — so it's often useful to build it from the plugin's own settings (a user-configured name, mode, target, etc.). Define an optional get_tools() function that returns the same shape as TOOLS, built from current settings:

def _build_description(cfg):
    base = "Generate an image. Describe the scene or action in ~20 words."
    name = (cfg.get("ai_name") or "").strip()
    if name:
        base += f" Write '{name}' for yourself — the appearance is filled in automatically."
    return base

def get_tools():
    cfg = _load_settings()                      # your settings reader (see above)
    return [{"type": "function", "function": {
        "name": "generate_image",
        "description": _build_description(cfg),
        "parameters": { ... },
    }}]

# Static fallback — used if get_tools() is absent or raises.
TOOLS = get_tools()

How it behaves:

• At load, the function manager calls get_tools() (when present) instead of reading the static TOOLS list, so the schema is correct from the first request.
• On a settings save, the function manager re-runs get_tools() and copies the fresh description / parameters onto the live tool objects in place — the AI sees the new description on its next turn with no plugin reload and no restart. Nothing is re-exec'd, so module-level state (DB handles, locks, ContextVars) is preserved.
• Only description and parameters are refreshed; the tool name never changes (toolset membership and dispatch are keyed on it). Adding or removing tools still requires a full reload.

Always keep a static TOOLS as the fallback. Plugins without get_tools() are unaffected — the settings-save refresh is a clean no-op for them.

Plugin State

Each plugin gets a persistent JSON key-value store at user/plugin_state/{name}.json:

from core.plugin_loader import plugin_loader

state = plugin_loader.get_plugin_state("my-plugin")
state.get("counter", 0)        # read
state.save("counter", 42)      # write (auto-persists)
state.delete("counter")        # remove key
state.all()                    # entire dict
state.clear()                  # wipe everything

PluginState is thread-safe — daemon threads, continuity tasks, and API handlers can all read/write the same plugin's state concurrently without data loss.

Chat-Scoped State

For data that belongs to one chat — a playthrough, a per-conversation cache, a journal — use the chat-scoped store instead. Its rows live in the chat database and follow the chat: renamed with it, encrypted with it when it goes private, and deleted with it.

state = plugin_loader.get_chat_state("my-plugin")

state.put(chat, "save", {...})       # single slot per key
state.get(chat, "save", default)
state.append(chat, "journal", event) # ordered rows — O(1) append, returns seq
state.append_many(chat, "journal", evs) # several rows in ONE transaction — all land or none; returns their seqs
state.read_all(chat, "journal")      # every row for that key, in order
state.replace(chat, "journal", rows) # atomic renumber (revert/rewrite)
state.delete(chat, "journal")        # or delete(chat) for everything
state.keys(chat)
state.get_all_chats("save")          # {chat: value} across visible chats
state.meta(chat, "journal")          # {'rows': n, 'updated_at': iso} or None

Values are JSON-serializable objects. On a hidden chat (private + vault sealed), reads come back empty and writes raise — never a silent drop, so handle the exception rather than falling back to a file. Use this instead of writing chat-keyed files under user/: file paths leak chat names and don't encrypt.

Deleting a chat deletes these rows with it, for public and private chats alike — core does it inside the delete transaction, before chat_deleted fires. Don't rebuild an archive-on-delete around it: a private chat's data must not outlive the chat. Keep never-erase behaviour inside a living chat (extra keys, extra rows) instead. See Private chats & privacy_aware.

For heavier storage, plugins can create their own SQLite database.

Advanced Patterns

Privacy-First Design

Never expose raw credentials (emails, keys, addresses) to the AI. Resolve at execution time:

# BAD — AI sees raw email addresses
def execute(function_name, arguments, config):
    return f"Contacts: alice@example.com, bob@example.com", True

# GOOD — AI only sees names and IDs
def execute(function_name, arguments, config):
    contacts = [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
    return json.dumps(contacts), True

Command Blacklists

For tools that execute commands (SSH, shell):

BLACKLIST = ["rm -rf /", "mkfs", "dd if=/dev", ":(){ :|:& };:"]

def _check_blacklist(command):
    for pattern in BLACKLIST:
        try:
            if re.search(pattern, command):
                return f"Blocked: matches '{pattern}'"
        except re.error:
            if pattern in command:
                return f"Blocked: contains '{pattern}'"
    return None

Caching with Scope Keys

For tools that fetch external data, cache per-scope with TTL:

_cache = {}
CACHE_TTL = 60

def _get_cached(scope):
    entry = _cache.get(scope)
    if entry and time.time() - entry["timestamp"] < CACHE_TTL:
        return entry["data"]
    return None

def _invalidate(scope):
    _cache.pop(scope, None)

Tools are added to toolsets and the AI calls them contextually. See TOOLS.md for the user-facing tools guide.

Reference for AI

• Tool file exports: ENABLED, EMOJI, AVAILABLE_FUNCTIONS, TOOLS, execute(function_name, arguments, config, plugin_settings=None, credentials=None) → (message: str, success: bool). Signature is inspected — declare 3, 4, or 5 params; extras are passed only if accepted. Optional get_tools() returns TOOLS-shaped schemas from current settings. Optional shutdown() runs before the module is dropped (disable, reload, uninstall, Sapphire stopping): release ports, threads, child processes; never call tools from it.
• Files for the user: from core import attachments; append attachments.marker(title, [{url, name}, ...]) to the result text. URLs must be the plugin's own routes (/api/plugin/<name>/..., else ValueError); audio renders a player, every file a download button; max 12; UI-only (stripped from the model's copy). Also valid in the text of a turn the plugin starts through the inbox.
• Starting a turn yourself: from core.chat import inbox; inbox.tell(chat, text, source='my-plugin', header_line=inbox.header('My plugin', kind, what), speak=None|'browser'|'speakers', on_drop=fn) — waits its turn (busy, locked, sealed alike), own reply (no folding); dropped only for gone / privacy-downhill / ttl / removed / restart, with item.drop_kind saying which. inbox.ask(...) returns her reply. Missing chat → inbox.InboxRefused. core.cadence.fire_once raises ChatBusy instead of waiting — only for a result that goes stale.
• Schema flags: is_local True|False|"endpoint" gates PRIVATE chats only (True runs; False/"endpoint" refused; unset refused unless PRIVATE_ALLOW_UNFLAGGED_TOOLS); network: true = UI "network tools" labeling only; hidden: true = out of the Toolsets picker but still registered and callable. No flag routes traffic.
• Tool metadata: file-level TOOL_CATEGORY = 'web' (one of web, memory, knowledge, people, goals, files, system, devices, comms, media, meta_danger, agents — or the plugin's own name; missing = the plugin's name), per-tool "category" override, per-tool "writes": True on anything that changes state or acts. A category resolves like a toolset word (an agent's toolset: 'web, files', extra_toolsets), groups the Toolsets picker, and names the scope its tools need (memory/knowledge/people/goals by category; any other tool = the one scope its plugin registered). FunctionManager.tool_meta(name), categories(), tools_in_category(cat, writes=None), tool_scope(name); resolve_tools(..., closed_scopes={...}) sheds tools whose scope is closed.
• Networking: from core import net — net.get/post/put/delete/request(url, ...) (requests-shaped), net.session_for(url) (pooled, lane-fixed), net.wan_session() (WAN lane, browser profile). Host classification is syntactic, never resolves DNS: LAN = loopback / RFC1918 / link-local / *.local / *.lan / *.home.arpa / single-label names / registered direct hosts → direct, redirects refused by default; WAN = everything else → proxy env when SOCKS is on. Register LAN FQDNs: core.socks_proxy.register_direct_hosts(zero_arg_callable, owner=plugin_name); keyed by owner (replace-on-reregister), auto-unregistered on plugin unload.
• get_tools() live refresh: re-run on settings save; only description/parameters update in place; tool names never change; add/remove needs a reload. Keep a static TOOLS fallback.
• PluginState (plugin_loader.get_plugin_state(name)): get/save/delete/all/clear; thread-safe; backed by user/plugin_state/{name}.json.
• Paths: from core.paths import PROJECT_ROOT, SYSTEM_PLUGINS_DIR, USER_PLUGINS_DIR, PLUGIN_STATE_DIR — never count .parents (a short chain lands on a real folder, fails silently). Scripts that can't import core: walk up to sapphire.py.
• PluginChatState (plugin_loader.get_chat_state(name)): put/get (slot), append (returns seq), append_many (one transaction, all-or-none, returns seqs), read_all, replace (atomic renumber), delete(chat[, key]), keys, get_all_chats(key), meta. Rows ride rename/vault/delete with the chat; on a hidden (sealed) chat reads return empty and writes RAISE. Rows survive a chat clear (register the chat_cleared hook to drop turn-anchored ones).
• Scopes: from core.chat.function_manager import scope_email (etc.) — resolves via __getattr__ against the scope registry; exists only while the owning plugin is loaded. scope_rag and scope_private are always present (core).