MCP / AI Agent

1. Status & Launch

The MCP server now starts itself the first time it's needed - either from the "Launch" button below, or the chatbox's Generative AI dialog - so -mcp at startup is optional (useful only if you want it running before you open this page, e.g. for Claude Desktop). If starting it fails (for example fastmcp isn't installed), the error shows up right where you triggered it.

The in-app chatbox has no separate "Workflow" or "Plugin" mode anymore - it's a single free-form conversation, backed by this same MCP server, that decides on its own which tools a message calls for (build/edit a graph, generate a plugin, drive AI Studio, inspect state, control playback, ...). It keeps real multi-turn history too: each conversation is its own saved thread with its own persistent claude session, switchable from the sidebar, so context isn't lost between messages or across an InfiniWorkflow restart.

Launch an agent here

Opens a real terminal window running the command below, in the InfiniWorkflow project directory, starting the MCP server first if it isn't already running. The command is remembered (in a cookie) for next time.

2. Claude Code

Register the server once per machine (or per project, with --scope local):

claude mcp add --transport sse infiniworkflow http://127.0.0.1:8000/sse

After that, any claude session started in this project picks up the connection automatically - including one launched from the button above. No special prefix is needed - just describe what you want in plain language and Claude calls the right tools on its own.

3. Claude Desktop

Add this to Claude Desktop's MCP server settings:

{
  "mcpServers": {
    "infiniworkflow": {
      "type": "sse",
      "url": "http://127.0.0.1:8000/sse"
    }
  }
}

4. AnythingLLM

Install AnythingLLM Desktop, then go to Settings > Extensions > MCP and add:

{
  "mcpServers": {
    "infiniworkflow": {
      "type": "sse",
      "url": "http://127.0.0.1:8000/sse"
    }
  }
}

Then, in Workspace Settings > Agent Configuration, paste a system prompt telling the agent how to map requests to tools (see the manual's MCP section for a starting point) - unlike Claude, AnythingLLM's agent doesn't already know the tool names, so it needs to be told.

5. Example Prompts

Once connected, try asking things like this (copy one into your agent's chat - prefix with @agent if you're using AnythingLLM, no prefix needed for Claude):

6. Available Tools

Also kept in sync in the manual's MCP Server section. A node can be addressed by its uuid or by its human-readable name shown on the canvas; if a name matches more than one node, tools that take a node name will report the ambiguity instead of guessing.

7. AI Studio / Video Studio Pipeline

These five tools let an agent drive the whole "segment this movie and train a model" workflow that AI Labeling Studio / Video Studio otherwise require many manual clicks for. They call the exact same underlying functions the UI itself uses - nothing about AI Labeling Studio or Video Studio changes because this exists, and anything the pipeline produces (extracted frames, keyframe masks, tracked labels) shows up exactly the same as if a human had done it, if you open that folder in AI Studio afterward.

The five stages

1. extract - pulls frames out of the source video. Videos over 1024 frames are split into chunks first automatically; only the first chunk is processed by this pipeline.
2. seed - segments one keyframe, two real ways: a foreground point you supply (seed_prompt={"frame": N, "points": [[x,y]]} - look at the frame first with get_node_image to pick a good point), or a plain-language description (seed_prompt={"frame": N, "auto": true, "prompt": "welding pool"}). There's no way to guess "what to segment" from a single sentence about the whole video, so one of these is required - a bare {"auto": true} with no prompt isn't reliable enough to run on its own (it's just SAM's single most-confident guess) and is treated the same as no seed_prompt at all: the pipeline stops and waits, and the app's own progress dialog offers a click-to-seed UI to finish the job by hand (click one or more points, preview the resulting mask, then confirm). A points/prompt attempt that fails falls back to the same click-to-seed UI rather than just failing outright.
3. track - propagates that seed mask across the whole frame sequence with SAM2. The live progress dialog shows a thumbnail of the latest tracked frame as it goes.
4. variations (optional) - generates data-augmentation copies of the labeled video for a larger training set.
5. train (optional) - fine-tunes a model on the labeled (and optionally augmented) frames.

Example: segment a video, then train, in two calls

run_ai_studio_pipeline(
    movie_path="welding_clip.mp4",
    root="demos",
    seed_prompt={"frame": 0, "points": [[420, 260]]},
    stop_after="track"
)
# -> {"pipeline_id": "..."}  poll with get_ai_studio_pipeline_status(pipeline_id)
# once stage == "done", the video is fully labeled - open it in AI Labeling
# Studio to check it, or go straight to training:

run_ai_studio_finetune(
    root="demos",
    folder="welding_clip/.videos/welding_clip",
    export_name="welding_v1",
    epochs=30
)
# -> {"pipeline_id": "..."} - InfiniWorkflow stays open; keep polling
# get_ai_studio_pipeline_status(pipeline_id) until stage is "done"/"failed",
# then read the results (onnx under "welding_v1", metrics under "welding_clip"):
get_ai_studio_pipeline_result(export_name="welding_v1")
get_ai_studio_pipeline_result(export_name="welding_clip")

Example: seeding with a text description

run_ai_studio_pipeline(
    movie_path="welding_clip.mp4",
    root="demos",
    seed_prompt={"frame": 0, "auto": true, "prompt": "welding pool"},
    stop_after="track"
)
# if this description-based match fails, the pipeline falls back to
# stage="awaiting_seed" automatically - same as giving no seed_prompt at all
# (see below) - rather than just failing outright.

Example: no seed point yet, or auto-seeding failed (two-call resume flow)

Omitting seed_prompt - or giving one that doesn't pan out - stops the pipeline right after extraction so you (or the agent, after looking at a frame) can decide where to seed. In the app itself, the progress dialog offers a click-to-seed UI for exactly this case instead of needing a second tool call at all.

run_ai_studio_pipeline(movie_path="welding_clip.mp4", root="demos")
# -> stage: "awaiting_seed", frames_folder: "welding_clip/.videos/welding_clip"
# look at a frame, then resume with the same movie_path/root/folder plus a seed:
run_ai_studio_pipeline(
    movie_path="welding_clip.mp4",
    root="demos",
    seed_prompt={"frame": 0, "points": [[420, 260]]}
)

Only one AI Studio pipeline runs at a time; run_ai_studio_pipeline/run_ai_studio_finetune return an error immediately if one is already active. Check with get_ai_studio_pipeline_status, or cancel_ai_studio_pipeline if you need to free the slot before it finishes on its own.

8. Plugin Generation

Write a brand-new compiled C++ node from a natural-language description. The design is deliberate: rather than the MCP server calling another LLM internally, these tools hand you (the calling model) the SDK headers and reference examples, and you write the plugin's 4 files yourself - you get to see and fix real compiler errors directly, instead of a second model doing it opaquely inside the tool. This is exactly what the in-app chatbox does too, since it's itself just a Claude session wired to this same MCP server (see Status & Launch) - there's no separate generator anymore.

The call flow

1. get_plugin_sdk_context() - read the SDK interface (setup/update/superviseInputs/superviseValue/execute/isCached/flushCache/teardown/destroy) and 4 full reference plugins (Canny2, boxblur, abandoned_object, topple_detection).
2. get_plugin_catalog_schema() - read the catalog JSON contract before writing <plugin_name>.json.
3. Write the 4 files yourself - plugin_impl.h, plugin_impl.cpp, <plugin_name>.json, CMakeLists.txt (base CMakeLists.txt on abandoned_object's or topple_detection's - not Canny2's or boxblur's, which hard-fail CMake on Windows (they ship a companion .sln/.vcxproj instead, which this flow can't use). Its POST_BUILD step, which copies the compiled binary (renamed to .plugin, via a PREFIX/SUFFIX target property) and catalog JSON into Extensions/, is what actually makes the node show up.
4. Either build it yourself with build_and_install_plugin(plugin_name, files) (writes and compiles them - on failure, read the log, fix your draft, retry with overwrite=True), or hand it to the human with preview_plugin_in_ui(plugin_name, files), which opens the same code-viewer dialog the chatbox uses so they can read the code and click Compile themselves (that button posts to the same underlying build endpoint).
5. activate_plugin(plugin_name) - hot-registers the compiled node with the running engine. Only needed after build_and_install_plugin; the dialog's own Compile button does this automatically.
6. add_node(tool_type="<plugin_name>.plugin", ...) - confirm it's actually usable in a graph.

Example

This is the kind of request the flow is built for - describe the node in plain language, and the four calls above turn that into a real, running node:

Create a plugin that takes a source image2D and a detection matrix2D
(class ID, x, y, w, h, confidence per row). The output is any detected
objects that stay stationary for some time. Have inputs of time and an
event trigger to clear.

Which the calling model turns into a real build, e.g.:

context = get_plugin_sdk_context()
schema = get_plugin_catalog_schema()
# Using context + schema, write the 4 files yourself. abandoned_object's
# reference plugin is the closest starting point (source + detections in,
# stationary-tracking logic, preview out) - copy its per-track centroid
# matching, not its trackId assignment (that example hands out a new
# trackId to every detection every frame, so stationaryFrames_ never
# actually accumulates - match against previousCentroids_ by nearest
# distance first, the way topple_detection does, or "stay stationary for
# some time" can never trigger).
files = {
    "plugin_impl.h": "...",                    # your PluginApi subclass header
    "plugin_impl.cpp": "...",                  # your PluginApi subclass implementation
    "StationaryObjectDetector.json": "...",    # identifier MUST be "StationaryObjectDetector.plugin"
    "CMakeLists.txt": "...",                   # base on abandoned_object's/topple_detection's, rename the project
}

result = build_and_install_plugin(plugin_name="StationaryObjectDetector", files=files)
# -> {"success": true, "log": "..."} or a build_failed log to fix and retry
# with overwrite=True

activate_plugin(plugin_name="StationaryObjectDetector")
# -> {"success": true, "identifier": "StationaryObjectDetector.plugin"}

add_node(tool_type="StationaryObjectDetector.plugin", node_name="Stationary Object Detector")
# confirms the node is actually usable in this session - open_viewer() on
# its "preview" output to see it live in the front end

To let a human review/download/compile the same draft instead of building it autonomously, swap the last three calls for one:

preview_plugin_in_ui(plugin_name="StationaryObjectDetector", files=files)
# -> opens the "Generated Plugin Files" dialog in any connected browser tab -
# same viewer the chatbox uses. The human reads the code,
# clicks Download to save the 4 files, or clicks Compile to build it right
# there (posts to the same /compile_plugin_to_disk endpoint, then hot-
# registers with the engine the same way activate_plugin does).