beaver_project/docs/superpowers/specs/2026-06-02-external-sidecar-connectors-design.md

# External Sidecar Connectors Design

Date: 2026-06-02

## Goal

Add real Weixin personal-account QR login and Feishu/Lark plugin onboarding to Beaver through a docker-compose predeclared sidecar service, without binding Beaver's connector layer to one vendor runtime. Beaver must not dynamically create containers or require Docker socket access.

This design implements the next connector layer after `docs/superpowers/plans/2026-06-02-channel-connectors-foundation.md`.

## Design Corrections

This design intentionally fixes four architecture constraints before implementation:

- The sidecar is generic. Beaver depends on a connector HTTP contract, not on one vendor runtime.
- Pairing is modeled as a broader `ConnectorSession`, because Feishu/Lark install/link flows are not only QR pairing.
- Bridge events include `eventId`, `timestamp`, and `deliveryAttempt`, and Beaver dedupes bridge events before they can trigger duplicate agent replies.
- Bridge authentication is service-level in the first version. The shared connector token lives in environment variables, not per-connection credentials.
- Outbound sidecar sends include a required `requestId` so sidecar retries are idempotent.
- Connected sessions dynamically register runtime channels. A successful Weixin or Feishu/Lark connection must not require a Beaver restart.

## Scope

Included:

- A repo-local `external-connector` sidecar service.
- A docker-compose service declaration for the sidecar.
- A sidecar `ConnectorProvider` abstraction.
- A production `VendorCliProvider` that runs the real vendor CLI/plugin commands required for Weixin personal-account QR login and Feishu/Lark plugin onboarding.
- Sidecar HTTP API for health, connector metadata, connector sessions, logout/remove, outbound send, and inbound event forwarding.
- Beaver `WeixinConnector` and `FeishuConnector` objects registered in `ChannelConnectorRegistry`.
- Beaver connector bridge endpoints that accept normalized sidecar inbound events and submit them to `ChannelRuntime.accept_inbound()`.
- `MessageDedupeStore` for connector bridge event idempotency.
- `ExternalConnectorChannel` runtime object for sidecar-backed outbound sends.
- `ChannelRuntime.add_channel()` and `ChannelRuntime.remove_channel()` for dynamic runtime activation.
- Web UI connection wizard for Weixin QR login and Feishu/Lark plugin onboarding.
- Unit tests using fake sidecar providers and bridge events.

Excluded:

- Dynamic Docker container creation from Beaver.
- Docker socket mounts in Beaver.
- Reimplementing Weixin iLink or Feishu/Lark plugin protocols inside Beaver.
- Building a generic plugin marketplace.
- Multi-user enterprise permission governance beyond local connector ownership and bridge token validation.

## Architecture

Use one predeclared sidecar for external connector providers:

```text
Beaver backend
-> Connector HTTP client
-> external-connector sidecar
-> ConnectorProvider
-> provider-specific runtime or CLI
-> Weixin / Feishu / future platform
```

Beaver owns:

- connection state in `ChannelConnectionStore`
- credential references in `CredentialStore`
- connector session state exposed to the web UI
- service-level connector authentication
- bridge event dedupe
- normalized runtime message admission
- runtime channel lifecycle
- runtime dedupe/session identity
- outbound dispatch into sidecar `/send`

The sidecar owns:

- provider runtime state
- provider install/update commands
- Weixin QR login and login-state persistence
- Feishu/Lark plugin install, bot creation/linking, and provider-side verification
- platform receive loops
- sidecar-to-Beaver inbound event delivery

## ConnectorProvider

The sidecar must isolate provider-specific behavior behind a provider contract. Beaver must not know which provider implementation is active.

```ts
interface ConnectorProvider {
  providerId: string;
  connectors(): ConnectorDescriptor[];
  health(): Promise<ProviderHealth>;
  startSession(input: StartConnectorSessionInput): Promise<ConnectorSessionView>;
  getSession(sessionId: string): Promise<ConnectorSessionView>;
  cancelSession(sessionId: string): Promise<void>;
  logout(connectionId: string): Promise<void>;
  send(input: SendMessageInput): Promise<SendMessageResult>;
}
```

Initial provider:

- `VendorCliProvider`: runs the real CLI/plugin commands required by the current Weixin and Feishu/Lark vendor flows.

`VendorCliProvider` command execution is intentionally constrained:

- Command templates are read only from sidecar startup environment variables.
- Frontend requests and sidecar HTTP request bodies cannot provide command strings.
- Command working directory is fixed to `CONNECTOR_HOME`.
- Per-connection state paths may be passed to commands as formatted arguments.
- Every command has a hard timeout.
- stdout and stderr are redacted before storage or API responses.

Future providers can be added without changing Beaver runtime code:

- `WechatyProvider`
- `NapcatProvider`
- `OneBotProvider`
- `EnterpriseWeixinProvider`

Provider choice is sidecar configuration, not Beaver architecture. `ExternalConnectorChannel` only calls the sidecar HTTP contract.

## Runtime Flow

Inbound:

```text
platform event
-> ConnectorProvider inside sidecar
-> sidecar normalized bridge event
-> POST Beaver /api/channel-connector-bridge/events
-> MessageDedupeStore
-> ChannelRuntime.accept_inbound()
-> MessageBus
-> AgentService
```

Outbound:

```text
AgentService
-> MessageBus outbound
-> ChannelManager.dispatch_outbound()
-> ExternalConnectorChannel.send()
-> POST sidecar /send
-> ConnectorProvider.send()
-> platform
```

`ExternalConnectorChannel` implements the existing runtime channel protocol:

```python
channel_id: str
kind: str
mode: str

async def start() -> None
async def stop() -> None
async def send(message: OutboundMessage) -> None
```

It is not a platform protocol adapter. It is a generic HTTP bridge to a sidecar.

Runtime materialization for sidecar-backed connections always emits:

```python
ChannelConfig(
    enabled=True,
    kind="external_connector",
    mode="http",
    account_id=spec.account_id,
    display_name=spec.display_name,
    config={
        "platformKind": "weixin",
        "connectionId": "conn_...",
        "sidecarBaseUrl": "http://external-connector:8787",
    },
    secrets={},
)
```

The original `ChannelConnection.kind` remains `weixin` or `feishu`; only the runtime transport kind is generic.

`ExternalConnectorChannel` authenticates outbound calls with the service-level connector token configured in Beaver's process environment, not with a per-channel secret. The same first-version deployment may use one shared token value for both directions, exposed as `EXTERNAL_CONNECTOR_TOKEN` to Beaver and `BEAVER_BRIDGE_TOKEN` to the sidecar.

## Dynamic Runtime Activation

A connected connector session must activate without restarting Beaver.

Add runtime methods:

```python
async def add_channel(self, channel_id: str, config: ChannelConfig) -> None:
    ...

async def remove_channel(self, channel_id: str) -> None:
    ...
```

`add_channel()` must run under a runtime lifecycle lock and has deterministic duplicate semantics:

- Same `channel_id` and same effective `ChannelConfig`: no-op.
- Same `channel_id` and changed effective `ChannelConfig`: build and start the replacement adapter before swapping it into the manager; after the swap succeeds, stop the old adapter.
- Replacement start failure: keep the old adapter registered and running, and return the failure to the caller.
- First registration after runtime start: build the adapter, register it, then start only that adapter.

`remove_channel()` must also run under the lifecycle lock. Missing channel ids are no-op; existing channels are stopped and unregistered.

When a connector session reaches `connected`:

```text
Connector session connected
-> connector updates ChannelConnection
-> registry materializes ChannelConfig
-> ChannelRuntime.add_channel(channel_id, config)
-> ChannelManager.register(adapter)
-> adapter.start()
-> channel status becomes running
```

This is a hard requirement for Weixin and Feishu/Lark onboarding. Manual backend restart is not an acceptable success path for this feature.

`remove_channel()` is used when a user logs out or revokes a sidecar connection:

```text
logout / revoke
-> sidecar logout
-> ChannelRuntime.remove_channel(channel_id)
-> connection status revoked or disconnected
```

## Sidecar Deployment

Add a sidecar service that can be enabled in deployment:

```yaml
services:
  external-connector:
    build: ./external-connector
    restart: unless-stopped
    environment:
      BEAVER_BRIDGE_BASE_URL: http://app-instance:8080
      BEAVER_BRIDGE_TOKEN: ${BEAVER_BRIDGE_TOKEN}
      CONNECTOR_API_TOKEN: ${EXTERNAL_CONNECTOR_TOKEN}
      CONNECTOR_HOME: /var/lib/external-connector
      CONNECTOR_PROVIDER: vendor_cli
      CONNECTOR_COMMAND_TIMEOUT_SECONDS: 120
    volumes:
      - external-connector-state:/var/lib/external-connector
```

For the current `create-instance.sh`-style deployment, the implementation adds:

- `docker-compose.external-connectors.yml` for local/development sidecar tests.
- documentation for attaching `external-connector` to the same Docker network as the target app instance.
- instance environment `EXTERNAL_CONNECTOR_BASE_URL=http://external-connector:8787`.
- instance environment `EXTERNAL_CONNECTOR_TOKEN=<service-level shared secret>`.

The implementation must not depend on Beaver mounting `/var/run/docker.sock`.

## Sidecar HTTP API

All sidecar requests and responses are JSON. The sidecar listens on port `8787`.

```text
GET  /health
GET  /connectors
POST /connector-sessions
GET  /connector-sessions/{session_id}
POST /connector-sessions/{session_id}/cancel
POST /connections/{connection_id}/logout
POST /send
```

`GET /connectors` returns:

```json
[
  {
    "kind": "weixin",
    "displayName": "Weixin",
    "authType": "qr",
    "providerId": "vendor_cli",
    "capabilities": ["receive_text", "send_text", "receive_media", "direct_messages"]
  },
  {
    "kind": "feishu",
    "displayName": "Feishu/Lark",
    "authType": "plugin_install",
    "providerId": "vendor_cli",
    "capabilities": ["receive_text", "send_text", "receive_media", "groups"]
  }
]
```

`POST /connector-sessions` request:

```json
{
  "kind": "weixin",
  "connectionId": "conn_...",
  "channelId": "weixin-main",
  "displayName": "Weixin Main",
  "callbackBaseUrl": "http://app-instance:8080",
  "options": {}
}
```

The sidecar authenticates the connector-session request with `Authorization: Bearer <EXTERNAL_CONNECTOR_TOKEN>`. It already has `BEAVER_BRIDGE_TOKEN` from its environment, so Beaver does not send bridge tokens in connector-session bodies.

For Feishu/Lark, `kind` is `feishu` and `options` may include `domain`, `mode`, and optional app credentials when linking an existing bot. If using the official plugin installer to create a bot, the sidecar starts that installer flow and reports QR, instruction, or action status back to Beaver.

`GET /connector-sessions/{session_id}` response:

```json
{
  "sessionId": "cs_...",
  "kind": "weixin",
  "status": "qr_ready",
  "qrCode": "weixin://...",
  "qrImage": "data:image/png;base64,...",
  "instructions": [],
  "accountId": null,
  "displayName": null,
  "error": null,
  "metadata": {}
}
```

Allowed connector session statuses:

- `pending`
- `qr_ready`
- `scanned`
- `confirmed`
- `installing`
- `waiting_for_user`
- `connected`
- `expired`
- `error`
- `cancelled`

`POST /send` request:

```json
{
  "requestId": "out_...",
  "connectionId": "conn_...",
  "channelId": "weixin-main",
  "kind": "weixin",
  "target": {
    "peerId": "wx_user_or_chat_id",
    "peerType": "dm",
    "threadId": null
  },
  "content": "reply text",
  "metadata": {
    "contextToken": "optional"
  }
}
```

`requestId` is required. Beaver must generate a stable request id for each outbound delivery attempt and must reuse the same `requestId` if the same outbound delivery is retried. The first-version rule is:

```text
out_{channel}:{session_id}:{message_id or sha256(content + inbound_message_id + peer_id + finish_reason)}
```

The sidecar dedupes `connectionId + requestId`:

- `completed`: return the original send result and do not send a second platform message.
- `processing` updated less than 60 seconds ago: return `409 Conflict` with `{"retryAfterSeconds": 5}` so Beaver retries later.
- `processing` updated 60 seconds or more ago: treat as stale and retry the provider send.

## Beaver Bridge API

Add a backend bridge endpoint for sidecar inbound messages:

```text
POST /api/channel-connector-bridge/events
```

The sidecar must authenticate every bridge request using the service-level bearer token from `BEAVER_BRIDGE_TOKEN`. Beaver rejects missing or invalid bridge tokens. Bridge tokens are deployment secrets, not connection records.

Bridge event body:

```json
{
  "eventId": "provider-event-id",
  "timestamp": "2026-06-02T09:30:00Z",
  "deliveryAttempt": 1,
  "connectionId": "conn_...",
  "channelId": "weixin-main",
  "kind": "weixin",
  "accountId": "weixin:...",
  "peerId": "wx_user_or_chat_id",
  "peerType": "dm",
  "userId": "wx_sender",
  "threadId": null,
  "messageId": "platform-message-id",
  "messageType": "text",
  "content": "hello",
  "metadata": {
    "contextToken": "optional"
  }
}
```

The bridge endpoint must:

1. validate bearer token
2. load `ChannelConnection`
3. reject unknown or revoked connections
4. dedupe by `connectionId + eventId` through `MessageDedupeStore`
5. construct `ChannelIdentity`
6. construct `InboundMessage`
7. call `ChannelRuntime.accept_inbound()`
8. mark bridge event completed or failed

## MessageDedupeStore

Add a JSON-backed `MessageDedupeStore` under:

```text
<workspace>/state/channel_connections/message_dedupe.json
```

It stores:

```python
@dataclass
class ConnectorMessageDedupeRecord:
    dedupe_key: str
    connection_id: str
    event_id: str
    status: str
    first_seen_at: str
    updated_at: str
    delivery_attempts: int
    message_id: str | None
    last_error: str | None
```

`status` values:

- `processing`
- `completed`
- `failed`

Duplicate handling:

- `completed`: return idempotent success and do not call `ChannelRuntime.accept_inbound()` again.
- `processing` updated less than 60 seconds ago: return `409 Conflict` with `{"retryAfterSeconds": 5}` so the sidecar retries later.
- `processing` updated 60 seconds or more ago: treat the record as stale, increment `delivery_attempts`, update `updated_at`, and reprocess the event.
- `failed`: allow reprocessing on the next delivery attempt, increment `delivery_attempts`, and clear `last_error` before calling runtime.

This store is separate from runtime session dedupe. Runtime dedupe still protects platform message identity, while bridge dedupe protects connector retries.

## Beaver Connectors

### WeixinConnector

Responsibilities:

- discover sidecar health
- start Weixin connector session through sidecar `/connector-sessions`
- poll sidecar connector session status
- create or update `ChannelConnection`
- store sidecar connection state reference in `CredentialStore` when the provider returns one
- validate by checking sidecar connection status
- materialize runtime config for `ExternalConnectorChannel`
- activate runtime via `ChannelRuntime.add_channel()` when connected
- revoke/logout by calling sidecar `/connections/{connection_id}/logout`
- deactivate runtime via `ChannelRuntime.remove_channel()` on logout/revoke

### FeishuConnector

Responsibilities:

- discover sidecar health
- start Feishu/Lark plugin install/link connector session
- optionally pass appId/appSecret/domain/mode for existing bot linking
- poll installer/session status
- create or update `ChannelConnection`
- validate by sidecar session or connection status
- materialize runtime config for `ExternalConnectorChannel`
- activate runtime via `ChannelRuntime.add_channel()` when connected
- revoke/remove plugin connection by calling sidecar logout/remove API
- deactivate runtime via `ChannelRuntime.remove_channel()` on logout/revoke

Feishu is sidecar-backed in this design because the user's supplied Feishu article describes the official plugin flow, not only a static bot-credential runtime adapter.

## Frontend

Replace the old static Weixin and Feishu fields with connector-driven UI:

- fetch `GET /api/channel-connectors`
- show Telegram, Weixin, and Feishu/Lark as connector options
- for Weixin:
  - start connector session
  - show QR image
  - poll status until connected/expired/error
  - show connected account and logout
- for Feishu/Lark:
  - choose create bot or link existing bot
  - collect domain and optional app credentials
  - start sidecar connector session
  - show QR/instructions/status returned by sidecar
  - show connected account and logout

The old `/api/channels` static config editor may remain for advanced runtime config, but connector onboarding should not rely on manual JSON editing or direct token entry for Weixin/Feishu.

## Error Handling

- Sidecar unavailable: show connector as `unavailable`; do not create a running connection.
- Provider install command fails: status `error`, with redacted stderr summary.
- QR expired: status `expired`, user can start a new connector session.
- Bridge token invalid: reject with `401`, record event without platform secret values.
- Unknown connection id in bridge event: reject with `404`.
- Duplicate completed bridge event: return idempotent success and do not call runtime again.
- Duplicate in-flight bridge event: return `409 Conflict` until the 60-second processing TTL expires, then allow one reprocess.
- Outbound send failure: mark outbound delivery failed and record connector error.
- Duplicate completed outbound send `requestId`: sidecar returns the original send result and does not send a second platform message.
- Duplicate in-flight outbound send `requestId`: sidecar returns `409 Conflict` until the 60-second processing TTL expires, then allows one retry.
- Sidecar restart: persisted provider state should survive through sidecar volume.

## Security

- Beaver never logs raw tokens, app secrets, bridge tokens, or sidecar connection tokens.
- Bridge authentication uses a service-level token from environment variables. It is not stored per connection and is never returned by APIs.
- Sidecar can only call bridge endpoints with the service-level bridge token.
- Beaver can only call sidecar control and send endpoints with the service-level connector token.
- Sidecar state volume contains login state and must be treated as sensitive.
- Vendor command strings are deployment configuration, not user input.
- Feishu user-identity mode has stronger privacy risk than bot-identity mode; UI must label it clearly if exposed.

## Testing

Backend unit tests:

- sidecar client fake for Weixin connector session start/status/logout/send
- sidecar client fake for Feishu connector session start/status/logout/send
- `ExternalConnectorChannel.send()` target mapping
- `ExternalConnectorChannel.send()` includes stable `requestId` and connector bearer auth
- `ChannelRuntime.add_channel()` dynamically starts and registers a channel
- `ChannelRuntime.add_channel()` no-ops for identical config, replaces changed config, and keeps the old channel if replacement start fails
- `ChannelRuntime.remove_channel()` stops and unregisters a channel
- bridge endpoint accepts valid events
- bridge endpoint rejects invalid token and unknown connection id
- bridge endpoint dedupes repeated `eventId` and calls runtime once
- bridge endpoint returns `409 Conflict` for non-stale `processing` duplicates and reprocesses stale records
- registry lists `telegram`, `weixin`, and `feishu`
- materialized sidecar connections produce `ChannelConfig(kind="external_connector", mode="http")` compatible with runtime factory

Sidecar tests:

- HTTP API shape for health/connectors/connector-sessions/send
- fake provider status transitions
- provider command runner error redaction
- send idempotency for duplicate `connectionId + requestId`
- send `processing` TTL returns `409 Conflict` before stale retry

Frontend tests:

- Weixin connector option opens QR modal
- polling reaches connected state
- expired/error states are visible
- Feishu flow starts install/link and shows returned instructions/status

Manual verification:

- Build app and sidecar Docker images.
- Start docker-compose sidecar setup.
- In `terminaltest`, open Weixin connector, scan QR, observe connected status without restarting Beaver.
- Send a Weixin text message and verify Beaver receives it once.
- Force sidecar retry of the same event and verify Beaver does not produce a duplicate agent reply.
- Send a Beaver reply and verify sidecar `/send` path.
- Start Feishu connector flow using the official Feishu/Lark plugin install path and verify the vendor-provided start command.

## Rollout

Implement in this order:

1. Sidecar HTTP contract with fake provider.
2. `MessageDedupeStore`.
3. Beaver `ExternalConnectorChannel` and bridge endpoint.
4. `ChannelRuntime.add_channel()` and `ChannelRuntime.remove_channel()`.
5. Weixin connector against fake sidecar client.
6. Feishu connector against fake sidecar client.
7. Frontend connector UI.
8. Production `VendorCliProvider` that shells out to real vendor CLI/plugin commands.
9. Docker build/compose integration.
10. Manual live verification.

The fake provider is test-only. The production provider must use the real vendor CLI/plugin commands for Weixin and Feishu/Lark; the fake provider only makes Beaver and frontend tests deterministic while the live provider handles non-deterministic external login and install flows.