The Problem They All Solve

You have applications that talk to LLMs. Maybe OpenAI for general tasks, Claude for coding, and a private inference cluster for sensitive workloads. Each provider has a different API, different auth, different SDKs, but you want one endpoint that handles routing, translation, and access control so your applications don't have to.

The Options

LiteLLM

LiteLLM is the most widely adopted open source LLM proxy. It supports 100+ providers, has a polished admin UI, and handles translation between API formats. If your primary need is broad provider support, LiteLLM is hard to beat.

Strengths:

• 100+ provider integrations out of the box

• Admin dashboard with spend tracking, rate limits, and team management

• Virtual API keys with per-key budgets

• Well-documented, large community

• Python-based, easy to extend

Tradeoffs:

• Security is at the application layer: API keys in headers, HTTPS for transport

• Designed to be deployed on a network and accessed over HTTP

• Requires a database (PostgreSQL) for the admin features

• Python runtime and dependencies

Portkey

Portkey is a lightweight TypeScript gateway focused on reliability and observability. It supports caching, retries, fallbacks, and load balancing with a clean config format.

Strengths:

• Simple, focused feature set

• Caching and automatic retries built in

• Fallback chains across providers

• Lightweight Node.js runtime

• Good observability with logging and analytics

Tradeoffs:

• Fewer provider integrations than LiteLLM

• No built-in access control or API key management in the open source version (available in their cloud offering)

• Security model is standard HTTPS

Kong AI Gateway

Kong adds AI capabilities as plugins to the Kong API gateway. If you already run Kong, this is a natural extension.

Strengths:

• Builds on Kong's mature API gateway infrastructure

• Rate limiting, auth, and observability from the Kong ecosystem

• Enterprise support available

• Plugin architecture for extensibility

Tradeoffs:

• Requires running the full Kong gateway stack

• Significant operational complexity if you don't already use Kong

• AI features are plugins, not a purpose-built gateway

• Community edition has limitations; some features require Enterprise license

Cloudflare AI Gateway

Cloudflare AI Gateway runs on Cloudflare's edge network. It's fast to set up if you're already in the Cloudflare ecosystem.

Strengths:

• Global edge deployment, low latency

• Built-in caching and rate limiting

• Simple setup if you use Cloudflare

• Analytics dashboard

Tradeoffs:

• Not self-hosted - your traffic goes through Cloudflare

• Vendor lock-in to the Cloudflare ecosystem

• Not open source

• Limited customization compared to self-hosted options

llm-gateway (ours)

llm-gateway is our entry. It's an OpenAI-compatible proxy built on zrok and OpenZiti that adds zero-trust networking as a foundational layer.

Strengths:

• Dark by default: the gateway can run with zero listening ports, invisible to network scanners

• End-to-end encryption through the OpenZiti/zrok overlay (not just HTTPS)

• Private model mesh: connect to private inference instances on other machines without opening ports or VPN

• Semantic routing with a three-layer cascade (heuristics, embeddings, LLM classifier)

• Weighted load balancing across multiple inference instances with health checks and failover

• Single Go binary, no runtime dependencies, no database required

• Apache 2.0

Tradeoffs:

• Fewer provider integrations than LiteLLM (currently OpenAI, Anthropic, OpenAI-compatible local inference)

• No admin UI (CLI and config file only)

• Newer project, smaller community

• The zero-trust features require zrok, which adds a concept to learn

Where Each One Fits

If you need broad provider support and an admin dashboard, LiteLLM is the pragmatic choice. It has the widest adoption and the most integrations.

If you want something lightweight with good retry/fallback behavior, Portkey keeps things simple.

If you already run Kong, the AI Gateway plugins are the path of least resistance.

If you're already on Cloudflare and don't need self-hosting, Cloudflare AI Gateway is quick to set up.

If your concern is security - specifically, if you don't want your LLM gateway to be a network-accessible endpoint, if you need to connect to models across networks without opening ports, or if you need cryptographic identity per client rather than shared API keys - that's where llm-gateway fits. The security model is fundamentally different from the others. The other gateways are good at what they do. We're not trying to be better at those same things. The difference is the connectivity layer underneath - that's where we bring something they don't have.

The Security Gap in LLM Gateways

This is the angle we think matters most, so it's worth expanding on.

Every other gateway on this list is designed to be deployed as an HTTP endpoint on a network. Clients connect to it over HTTPS, authenticate with API keys, and make requests. The security model is: put it behind a load balancer, use HTTPS, and manage API keys carefully.

This may be fine for a lot of scenarios, but it has some gaps. API keys can get committed to repos, leaked in logs, or shared among team members. The endpoint is scannable - anyone who can reach the network can find it and try to authenticate. If you're connecting to models on another network (e.g., an inference cluster on a different subnet), you need to either open firewall ports or set up a VPN.

llm-gateway takes a different approach. When you run it with zrok, the gateway doesn't listen on any port. It connects outbound to the zrok overlay and is reachable only by clients with the right cryptographic identity. You can connect to model instances on other machines the same way - no ports, no DNS, no VPN. The traffic is encrypted end to end through the overlay, not just to the gateway's TLS termination point.

This isn't better for everyone. If you're running a local development proxy, maybe you don't need this. But if you're deploying an LLM gateway for a team, connecting to models across networks, or operating in an environment where "put it on the network and secure it with API keys" doesn't meet your security requirements, the zero-trust model is worth looking at.

Try It

go install github.com/openziti/llm-gateway/cmd/llm-gateway@latest

The getting started guide walks through setup from a minimal Ollama config to a full deployment with semantic routing and zrok integration.

If you try it and have feedback, we'd appreciate it. Post on Discourse or open an issue on the repo. And of course a star is always appreciated.

What is OpenZiti? OpenZiti is an open-source, zero-trust networking platform that embeds security directly into application connectivity. It replaces traditional VPNs and network perimeters with identity-aware, policy-driven service access — no open ports, no implicit trust. It's developed by NetFoundry,

The SKILL-driven auto-Zitification workflow changes that math. Running on OpenCode with Claude Sonnet 4.6 and ziti-mcp-server v0.9.0, it took 82 Go files, found every transport candidate, generated reviewable diffs, transformed the code, and provisioned live network identities and policies, in under 5 minutes, with zero broken files.

TL;DR

• The auto-Zitification SKILL runs 4 phases: Detect, Plan, Transform, Provision, with a mandatory human approval gate before any file is written

• Phase 1 uses zitifier-detect-go, a true AST analyzer (go/analysis framework, full type resolution), not grep

• The review gate in Phase 2 caught two real issues in the demo target before touching a single file

• Phase 4 provisions identities, a Ziti service, and dial/bind policies live via ziti-mcp-server v0.9.0

• Claude Sonnet 4.6 reached the approve gate in 2:50; gpt-4o-mini never finished

• Demo target: stefanprodan/podinfo, 82 Go files, gRPC + HTTP + TCP, used by CNCF Flux and Flagger

What Is the Auto-Zitification SKILL?

The SKILL is a 4-phase orchestration defined in a SKILL.md file and executed by OpenCode, an open-source, provider-agnostic AI coding agent. Each phase runs as a named subagent with declared tool permissions. No phase can exceed its scope: the detection agent can't write files, the provisioning agent can't touch source code.

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#1e293b', 'primaryTextColor': '#e2e8f0', 'primaryBorderColor': '#3b82f6', 'lineColor': '#64748b', 'background': '#0f172a', 'mainBkg': '#1e293b', 'edgeLabelBackground': '#1e293b'}}}%%
flowchart LR
  classDef phase fill:#1e3a5f,stroke:#3b82f6,color:#93c5fd,stroke-width:2px
  classDef gate fill:#3d1515,stroke:#ef4444,color:#fca5a5,stroke-width:2px,stroke-dasharray:4
  classDef prov fill:#14291f,stroke:#10b981,color:#6ee7b7,stroke-width:2px

  D["🔍 DETECT\nAST analysis\nzitifier-detect-go"]:::phase
  P["📋 PLAN\nPer-file diffs\nreview gate"]:::phase
  G["✋ HUMAN\nAPPROVAL\nREQUIRED"]:::gate
  T["✏️ TRANSFORM\nWrite SDK replacements\n.pre-ziti.bak backups"]:::phase
  V["⚙️ PROVISION\nIdentities · Service\nPolicies via MCP"]:::prov

  D --> P --> G --> T --> V

4 phases, one mandatory gate. No file is written until the human approves the plan.

Why Doesn't Manual Zitification Scale?

Embedding the OpenZiti SDK by hand requires knowing the idiomatic patterns for each language, following net.Conn assignments across variable assignments, and catching every call site, including the ones inside helper functions three layers deep. Miss one and you get a subtle fallback bug that only shows under load. Go, Node.js, and Python each have different SDK patterns, so there's no cross-language muscle memory to build on.

At microservice scale the math breaks. Hours per service multiplied by hundreds of services means this work simply doesn't get scheduled. The SKILL makes it schedulable.

What Is True AST Detection and Why Does It Matter?

zitifier-detect-go uses the go/analysis framework, the same infrastructure behind go vet, staticcheck, and gopls. It performs full type resolution via NeedTypesInfo, which means it knows that net.Listen is net.Listen even when it's been aliased, assigned to an interface, or shadowed by a local variable. Grep-based detection cannot do this.

The practical difference shows in the false-positive rate. Grep flags http.Get("github.com/some/import-path") as a HIGH confidence external call. The AST detector reads the URL argument, classifies external domains as LOW confidence, and skips them. It also validates the network argument on net.Dial, Unix sockets and UDP get skipped automatically. Only TCP variants are candidates.

Detection output is structured JSON:

{ "file": "pkg/api/grpc/server.go",
  "line": 88,
  "type": "SERVER",
  "pattern": "net.Listen",
  "confidence": "HIGH" }

For podinfo's 82 Go files, the detector found 4 candidates across 3 transport patterns:

No false positives. No missed candidates.

What Did the Review Gate Actually Catch?

Before a single file is written, Phase 2 generates per-file diffs and surfaces warnings. In the podinfo run, it caught two real issues.

Warning 1, Per-request Ziti context. The initial plan placed initZiti() inside echoHandler(), a per-request HTTP handler. Ziti contexts are expensive to initialize and must be created once at startup, stored as a struct field, and reused. Creating one per request would crater performance under any real load. The gate flagged this before writing.

Warning 2, Dropped transport wrapper. The echo.go client originally used otelhttp.NewTransport(http.DefaultTransport) for OpenTelemetry distributed tracing. The naive replacement would have discarded that wrapper entirely, silently breaking distributed traces. The correct fix composes the transports: wrap the Ziti transport with OTel, don't replace it.

Both issues were caught at the approve gate. Neither required finding a bug after the fact. This is the point of the gate, it's architectural, not advisory.

What Does the Transformation Look Like?

The gRPC server transformation is the cleanest example. net.Listen returns a net.Listener interface, and zitiCtx.Listen returns the same interface. It's a true drop-in, no gRPC-specific code changes needed anywhere else in the stack.

Before:

import (
  "fmt"
  "net"
)

func (s *Server) ListenAndServe() {
  listener, err := net.Listen(
    "tcp",
    fmt.Sprintf(":%v", s.config.Port),
  )
  // ...
}

After:

import (
  "os"
  "github.com/openziti/sdk-golang/ziti"
)

func (s *Server) ListenAndServe() {
  zitiCtx, _ := initZiti()
  listener, err := zitiCtx.Listen(
    os.Getenv("ZITI_SERVICE_NAME") + "-grpc")
  // ...
}

Every touched file gets a .pre-ziti.bak backup before writing. The go.mod update is handled automatically. The shared initZiti() helper is created once and referenced across all transformed files.

How Does the Provisioning Phase Work?

Phase 4 runs entirely through ziti-mcp-server v0.9.0, which exposes the full Ziti Management API to the agent as 201 tools. The provisioner queries the controller version to confirm network type (the demo ran against a self-hosted OpenZiti quickstart), then:

1. Creates podinfo-client and podinfo-server identities, writes enrollment JWTs to disk

2. Creates a Ziti service named per the ZITI_SERVICE_NAME env var, covering both HTTP and gRPC traffic

3. Wires a Dial policy for the client identity and a Bind policy for the server identity

4. Creates a Service Edge Router Policy for all routers

5. Emits ziti edge enroll commands for each JWT

The whole provisioning step runs in the same 5-minute window as the code transformation. The no-listening-ports model applies immediately after enrollment; the service binds to the Ziti fabric, not to a TCP port.

Both self-hosted OpenZiti and NetFoundry-managed networks are supported.

Which Model Should You Use?

Model choice has a material impact on agentic workflows. Here's what the podinfo run showed:

Claude Sonnet 4.6 reached the approve gate in 2 minutes 50 seconds and completed the full run in under 5 minutes. The code quality was high enough that both review gate warnings were substantive, not noise, and the transformations required no manual corrections.

One practical note: running this required a personal Claude Pro account to get an API key. NetFoundry's Team plan doesn't distribute API keys directly to engineers. If your org is in the same situation, a personal Pro account is the current workaround until that changes.

Why OpenCode for This Workflow?

The initial evaluation ran on OpenCode rather than Claude Code for one specific reason: provider-agnostic validation. One of the goals was confirming the SKILL works across models, Claude, GPT, Gemini, local models. OpenCode handles that in a single config line change. Claude Code is provider-locked by design.

The other factor is the explicit subagent architecture. Each phase in the SKILL runs as a named agent with declared tool permissions:

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#1e293b', 'primaryTextColor': '#e2e8f0', 'primaryBorderColor': '#3b82f6', 'lineColor': '#64748b', 'background': '#0f172a', 'mainBkg': '#1e293b', 'edgeLabelBackground': '#1e293b'}}}%%
flowchart TB
  classDef agent fill:#1e3a5f,stroke:#3b82f6,color:#93c5fd,stroke-width:2px
  classDef scope fill:#1e293b,stroke:#64748b,color:#94a3b8,stroke-width:1px
  classDef mcp fill:#14291f,stroke:#10b981,color:#6ee7b7,stroke-width:2px

  SKILL["📋 SKILL.md\nZitifier Orchestrator"]

  subgraph D ["zitifier-detect-go · READ ONLY"]
    DA["read files\nrun binary\nJSON output"]:::scope
  end

  subgraph P ["zitifier-transform (plan) · READ + DIFF"]
    PA["read files\ngenerate diffs\nawait approval"]:::scope
  end

  subgraph T ["zitifier-transform (apply) · READ + WRITE"]
    TA["read files\nwrite files\nbackup (.bak)"]:::scope
  end

  subgraph V ["zitifier-provision · MCP ONLY"]
    VA["ziti-mcp-server\nidentities\nservices/policies"]:::mcp
  end

  SKILL --> D --> P --> T --> V

Each agent is a .md file in .opencode/agents/, versioned and diffable. Tool scopes are declared per-agent, detect can't write, provision can't touch source files.

That audit trail matters for security-conscious teams. Every agent's capabilities are in a file you can read, diff, and review in a PR.

The next evaluation is Claude Code, the SKILL.md format is portable and the same 4-phase logic applies. That's the subject of a follow-up post.

Frequently Asked Questions About Auto-Zitification

Does this work with languages other than Go?

Go is the first supported language via zitifier-detect-go. A Node.js/TypeScript equivalent using the TypeScript Compiler API is in active development (zitifier-detect-node) and uses the same JSON output schema and confidence levels. Python and Java equivalents are on the roadmap but not yet scoped.

What happens if the detector flags something incorrectly?

The review gate in Phase 2 exists precisely for this. Every candidate appears in the diff with full context before any file is written. You can reject individual candidates at the approval step. The .pre-ziti.bak backups also mean you can revert any transformation after the fact without touching version control.

Does it work with NetFoundry-managed networks?

Yes. The provisioning phase uses ziti-mcp-server, which supports both self-hosted OpenZiti and NetFoundry-managed networks (V8 only). The --profile flag in ziti-mcp-server lets you pre-configure separate credentials for each environment.

Is the SKILL open source?

The component tools are open source: zitifier-detect-go is Apache-2.0 at github.com/openziti/zitifier-detect-go, and ziti-mcp-server is Apache-2.0 at github.com/openziti/ziti-mcp-server. The SKILL.md orchestration file and its sub-agents will be published alongside the component tools.

What's next on the roadmap?

Five items are scoped: zitifier-detect-node for TypeScript (testing now), a shared Ziti context helper to replace per-file initZiti() calls, transport composition to preserve OTel/Prometheus middleware, and scale validation against a large production Go codebase, Mattermost server at 500,000 lines is the current candidate.

Thanks

If you like OpenZiti, it's always very much appreciated if you take a moment to drop us a star. We see and appreciate every repository star.

File issues, contribute tooling improvements, or drop into the OpenZiti community Discourse if you run into anything unexpected.

The full demo runs at youtu.be/b-G_zvmzaks, all 4 phases including live provisioning against a Ziti network.

zitifier-detect-go: github.com/openziti/zitifier-detect-go

ziti-mcp-server: github.com/openziti/ziti-mcp-server

OpenCode: opencode.ai

What is OpenZiti? OpenZiti is an open-source, zero-trust networking platform that embeds security directly into application connectivity. It replaces traditional VPNs and network perimeters with identity-aware, policy-driven service access — no open ports, no implicit trust. ziti-mcp-server is built on top of it, which means the agent's access to your management plane is governed by the same primitives you use for everything else in your network. It's developed by NetFoundry,

TL;DR

• ziti-mcp-server is an MCP server exposing 201 Ziti Management API tools plus 8 session meta-tools, full coverage of identities, services, routers, policies, configs, and fabric operations

• Runs as a local process between your MCP client and your Ziti controller; credentials never leave your machine

• For quick evaluation, UPDB (username/password) is the fastest path; for production, create a Ziti identity, enroll it, and authenticate using the identity file, the agent becomes a real Ziti identity subject to your access policies

• Tested with OpenZiti v1.6+; requires Go 1.24+ if building from source

• One gotcha: after any configuration change, restart your MCP client, ziti-mcp-server does not hot-reload

What Is ziti-mcp-server?

ziti-mcp-server is a Model Context Protocol server that gives AI agents direct access to the Ziti Management API. It runs as a local process between your MCP client (Claude Desktop, Cursor, Windsurf) and your Ziti controller, exposing 209 tools: 201 wrapping the full REST management API and 8 for session lifecycle. Nothing is stubbed out or simplified — the tool surface is generated directly from the Ziti OpenAPI spec.

That 209-tool surface covers identities, services, edge routers, service policies, edge router policies, configurations, config types, authenticators, CAs, posture checks, and the fabric layer including terminators, circuits, and cluster operations.

%%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#1e293b', 'primaryTextColor': '#e2e8f0', 'primaryBorderColor': '#3b82f6', 'lineColor': '#64748b', 'background': '#0f172a', 'mainBkg': '#1e293b', 'edgeLabelBackground': '#1e293b'}}}%%
flowchart LR
  classDef client fill:#1e3a5f,stroke:#3b82f6,color:#93c5fd,stroke-width:2px
  classDef server fill:#1e293b,stroke:#0ea5e9,color:#7dd3fc,stroke-width:2px
  classDef ctrl fill:#1a1f2e,stroke:#6366f1,color:#a5b4fc,stroke-width:2px

  MCP["🤖 MCP Client\n(Claude Desktop · Cursor · Windsurf)"]:::client
  ZMS["⚙️ ziti-mcp-server\n(local process)"]:::server
  Ctrl[("🔑 Ziti Controller\nManagement API")]:::ctrl

  MCP -->|"tool calls via stdio"| ZMS
  ZMS -->|"REST + mTLS"| Ctrl

ziti-mcp-server runs locally and proxies MCP tool calls to the Ziti controller's management API. Credentials stay on your machine.

What Can You Actually Ask It?

Once connected, your MCP client can answer operational questions and execute management operations through natural language. The agent translates your request into one or more Ziti API calls and returns the result. Multi-step operations happen in a single conversational request, with no scripting required.

Some examples that work well:

• "Which identities have access to the payments-api service?", runs listServiceIdentities and formats the result

• "Show me a network summary: total identities, services, and edge routers", combines listIdentities, listServices, and listEdgeRouters with count aggregation

• "Create a new identity called ci-runner and return the enrollment JWT", runs createIdentity then createEnrollment

• "List all edge routers and flag any that haven't connected in the last 24 hours", uses listEdgeRouters with status filtering

• "What version is the controller running?", calls getVersion

The agent handles multi-step operations naturally. Asking "create an identity, generate an enrollment JWT, and give me the one-time token" produces three sequential API calls without you scripting any of it.

How Do You Get Running in 5 Minutes?

Pre-built binaries are available for macOS (Intel and ARM), Linux (amd64/arm64), and Windows. On macOS Apple Silicon:

curl -sL https://github.com/openziti/ziti-mcp-server/releases/latest/download/ziti-mcp-server_darwin_arm64.tar.gz | tar xz
sudo mv ziti-mcp-server /usr/local/bin/

Or build from source with Go 1.24+:

go install github.com/openziti/ziti-mcp-server/cmd/ziti-mcp-server@latest

Register with Claude Desktop, this writes the server entry into Claude's config file, nothing more:

ziti-mcp-server install

For Cursor or Windsurf:

ziti-mcp-server install --client cursor
ziti-mcp-server install --client windsurf

Now restart your MCP client. This is the step people miss. Config changes don't hot-reload, the client needs a full restart before it picks up the new server registration.

Once Claude Desktop is back up, you'll see the Ziti tools available. Authenticate at runtime using the loginUpdb tool, pass your controller host, username, and password when Claude asks. That's enough to start exploring.

What Authentication Mode Should You Use?

The right answer depends on your use case. ziti-mcp-server supports four modes:

For anything beyond kicking the tires, the identity file approach is the right model. Create a Ziti identity for the agent, enroll it, and configure ziti-mcp-server to authenticate using the resulting identity.json. Pre-configure it so no credentials are passed at runtime:

# Create the identity in your Ziti network
ziti edge create identity mcp-agent --role-attributes mcp-agents -o mcp-agent.jwt

# Enroll it to produce the identity file
ziti edge enroll mcp-agent.jwt -o mcp-agent.json

# Pre-configure ziti-mcp-server to use it
ziti-mcp-server init \
  --auth-mode identity \
  --ziti-controller-host your-controller.example.com \
  --identity-file ./mcp-agent.json \
  --profile prod

The reason this matters: the agent now has a real Ziti identity. You can scope its access using role attributes and service policies exactly as you would for any other identity in your network. Want the agent to be read-only against production but fully operational against staging? That's a policy decision, not a server configuration. The no listening ports model OpenZiti applies to application traffic applies equally well to management plane access, and as of OpenZiti v1.8, the controller APIs can themselves be OpenZiti services, making this pattern even tighter.

How Do You Restrict What the Agent Can Do?

Two mechanisms: the --read-only flag and --tools glob filtering.

--read-only strips all mutating tools at startup. The agent can query and inspect but cannot create, update, delete, or enroll anything:

ziti-mcp-server run --read-only

--tools lets you filter by tool name patterns:

# Identity operations only
ziti-mcp-server run --tools '*Identity*'

# All list and get operations
ziti-mcp-server run --tools 'list*,get*'

# Combine: read-only identity tools only
ziti-mcp-server run --tools '*Identity*' --read-only

Multi-profile support lets you configure separate credentials and restrictions per network. A --profile prod can run read-only while --profile staging has full access. Switch between them at runtime using the selectNetwork tool without restarting the server.

Frequently Asked Questions About ziti-mcp-server

Does the LLM see my credentials?

No. Credentials are stored in ~/.config/ziti-mcp-server/config.json with 0600 permissions, readable only by your user. The LLM sees tool names and results, not the underlying auth tokens or identity file contents. When using UPDB at runtime, you pass credentials through the tool call, but they're handled by the local process and never included in the response the model receives.

Which MCP clients does it support?

Claude Desktop, Cursor, and Windsurf have first-class install support. For any other MCP-compatible client, add the server manually to the client's config:

{
  "mcpServers": {
    "ziti": {
      "command": "/usr/local/bin/ziti-mcp-server",
      "args": ["run"]
    }
  }
}

Then restart the client.

What OpenZiti version does it require?

ziti-mcp-server has been tested with OpenZiti v1.6 and later. The tool surface is generated from the Ziti OpenAPI spec, so older controllers may not support all 201 API tools, earlier API versions will return errors on unsupported operations rather than silently failing.

Is it production-ready?

The server is stable and the tool coverage is complete, but treat it as you would any agent with management plane access: scope its identity tightly, run --read-only against production where possible, and use role-attribute-based service policies to limit what the identity can reach. The security model is sound precisely because it uses OpenZiti's own access control primitives rather than inventing new ones.

Can I manage multiple Ziti networks from one session?

Yes. Configure separate profiles with ziti-mcp-server init --profile <name> for each network. Use the listNetworks tool to see all configured profiles and their connection status, and selectNetwork to switch the active profile at runtime.

ziti-mcp-server is available now at github.com/openziti/ziti-mcp-server under Apache 2.0.

File issues, contribute tooling improvements, or drop into the OpenZiti community Discourse if you run into anything unexpected.

If you like OpenZiti, it's always very much appreciated if you take a moment to drop us a star. We see and appreciate every repository star.

When we shipped v1.0 last year, it was about proving that zrok was production-ready... a solid foundation with a redesigned web console, a new zrok Agent, and all the infrastructure that a serious release needs. v2.0 is a different kind of milestone. This release is about rethinking the conceptual model where it was weakest and building a much more powerful foundation for what comes next.

In case you're not sure what zrok is, take a look at the post introducing zrok. zrok is all about making powerful, secure, peer-to-peer sharing work in a simple way... network resources, files, applications, all with a single command. zrok is open source, self-hostable, and built on top of OpenZiti. It's developed by NetFoundry, and there's a free hosted instance available at zrok.io.

For those of you who prefer video, I did a zrok Office Hours previewing the v2.0 changes a while back that is still very relevant:

https://www.youtube.com/watch?v=iJfWj7umdFI

Let's dig in...

The One Main Issue: Namespaces and Names

The headline change in v2.0 is that reserved sharing is gone. The zrok reserve, zrok release, and zrok share reserved commands have been removed entirely, replaced by a new system of namespaces and names.

This is a major conceptual shift, and I think it's the right one. Here's why.

In the old model, share tokens did double duty as public identifiers. Your share token was the name that the outside world used to reach your share. That seems fine until you realize that share tokens are tied to shares, and shares are tied to environments, which means they're tied to a specific host system and a specific account.

This created real operational problems. Want to move a named share to a different machine? You couldn't do it without disrupting the public name that people were using to reach it. If you stop and think about it, this is a pretty significant limitation for anything beyond casual use.

In v2.0, names are decoupled from shares. You create names, and then you attach those names to shares. If you need to move the sharing backend to a different environment, you just detach the name from the old share and attach it to the new one. The public-facing identity stays stable. Your users never notice.

Namespaces provide the organizational layer above names, giving you logical grouping and scoping. The combination is dramatically more flexible than what reserved sharing could ever offer.

For the full details on how namespaces and names work, see the zrok v2 Migration Guide. But here's the practical upshot: anything you could do with reserved sharing, you can do with namespaces and names... and a lot of things you couldn't do before are now straightforward.

Try It, Then Keep It

One of my favorite new features is zrok2 modify name. It solves a workflow problem that has always bugged me.

Previously, you had to decide up front whether a share was going to be ephemeral or reserved. If you shared something ephemerally and then realized you wanted to keep it around, you had to tear it down and recreate it as a reserved share. With zrok2 modify name -r you can just promote an ephemeral name to a reserved name on the fly. It persists indefinitely, or until you decide otherwise. Conversely, zrok2 modify name -r=false will schedule a reserved name for release when its associated share terminates.

Ephemeral by default, persistent by choice. That's the workflow I always envisioned.

Private Share Allocation

Now that reserved sharing has been replaced with namespaces, and all shares are effectively ephemeral (while reserved names are not)... where does this leave private sharing?

Private sharing is still one of zrok's super powers. Private shares still use share tokens as identifiers in the same way they always have. But in v1.0, the zrok reserve command was doing double-duty for both public share names and private share tokens. So how does v2.0 handle this?

In v2.0 you can use zrok2 create share and zrok2 delete share to pre-allocate and manage shares for private sharing. This is a streamlined replacement for v1.0's zrok reserve and zrok release commands. The zrok2 share private command includes a --share-token flag that lets you start sharing using an existing pre-allocated share, in the same way that zrok share reserved worked in v1.0. You can also use --share-token to give your private shares human-readable vanity names that are easy to remember and share with collaborators.

Side-by-Side: Running v1 and v2 Together

We made a deliberate decision with v2.0: no forced migration. You can run zrok v1 and zrok v2 on the same system simultaneously, with zero interference between them.

The binary is now called zrok2 instead of zrok. The environment directory is ~/.zrok2 instead of ~/.zrok. Environment variables use the ZROK2_ prefix instead of ZROK_. Linux packages are zrok2 and zrok2-agent, the systemd service is zrok2-agent.service, and configuration lives in /etc/zrok2.

This means you can install v2 alongside your existing v1 setup, experiment with the new namespace model, and migrate at your own pace. Your existing v1 environment is completely untouched. When you run zrok2 enable, it creates a fresh environment in ~/.zrok2... nothing in ~/.zrok is affected.

When you're comfortable and ready to make the switch, just stop using the zrok binary and switch to zrok2. That's it.

Web Console Improvements

The web console received a significant overhaul for v2.0. The node graph layout has been drastically improved... the visual network navigator is cleaner and more intuitive. Under the hood, the internals have been cleaned up and refactored.

There are some nice new interaction features too. A new focus mode lets you press F on the keyboard to focus the graph on just the nodes reachable from your current selection, cutting through the noise when you're working with larger configurations. The detail panel can now be toggled on and off, giving you more screen real estate when you need it.

Overall the web console received a solid dose of attention and cleanup.

The New dynamicProxy Frontend

For self-hosters, v2.0 introduces zrok2 access dynamicProxy, a new frontend implementation designed to work with the namespace/names model.

The previous publicProxy frontend worked by parsing the Host header and extracting a share token. That approach was tightly coupled to the old model where share tokens were the public identifiers. The new dynamicProxy receives mapping updates directly from the zrok controller, allowing it to support any kind of mapped name, not just share tokens extracted from headers.

zrok2 access public remains available for legacy-style setups, so existing self-hosted deployments aren't disrupted. See the zrok dynamicProxy Guide for details on setting up the new frontend.

Agent Improvements

The zrok Agent continues to mature. In v2.0, it includes significantly improved error handling for subordinate processes. When shares or accesses encounter errors, whether during agent reloading or during active runtime, they're now retried using an exponential backoff approach rather than failing hard. Errored processes get transient err_XXXX tokens, which you can use to manage and release them through the normal agent interface. This makes the agent much more resilient in production environments where transient failures are a fact of life.

The agent has also been updated for the v2 naming model. Any share with a reserved name will be automatically restarted by the agent, giving you the persistence behavior you'd expect without manual intervention.

CLI Quality of Life

v2.0 includes a set of new commands that make working with your zrok account much more pleasant.

New zrok2 list commands for names, namespaces, environments, shares, and accesses let you query everything in your account. They support filtering on activity, accesses, shares, descriptions, host, IP address, and other criteria. Human-readable tabular output by default, with --json when you need machine-readable output.

zrok2 overview now defaults to a human-readable format that lays out your account details clearly. The classic JSON output is still available with --json.

zrok2 delete environment lets you clean up environments other than your currently enabled one. Pair it with zrok2 list environments --idle to find and remove stale environments.

Under the Hood

For those who care about the internals... v2.0 includes a complete overhaul of the core OpenZiti automation logic. The legacy controller/zrokEdgeSdk package has been replaced with a much more streamlined controller/automation package. If you've ever had to read through the controller code, this is a significant clarity improvement.

All logging has been migrated from pfxlog/logrus to df/dl and log/slog. Use DL_USE_JSON=true for JSON output or DL_USE_COLOR for colorized output.

The root package path has moved to github.com/openziti/zrok/v2 to follow Go's v2+ package naming conventions.

Get Started

To try zrok 2.0, grab a zrok2 binary from the releases page and run zrok2 enable to create a new v2 environment. Your existing v1 setup will remain completely untouched.

For the full details on migrating from v1 to v2, see the zrok v2 Migration Guide.

Thank You

Thank you for supporting zrok. We're really excited about where the v2.0 roadmap takes us... namespaces and names open up a lot of possibilities that we're already exploring for future releases.

If you like zrok, it's always very much appreciated if you take a moment to drop a star onto the zrok repository on GitHub. We literally see and appreciate every repository star. If zrok is an important part of your workflow, maybe consider getting a subscription on zrok.io? Those subscriptions are an incredibly helpful signal that we're on the right track.

If there is anything we can do to improve zrok or if you've run into anything we can help with, please reach out to us at our OpenZiti Discourse forum. There is a zrok category, and we're standing by ready to help.

Identity management is central to modern network security. This post shows how to use short‑lived X.509 client certificates from SPIRE — the reference runtime for SPIFFE (Secure Production Identity Framework For Everyone) — as OpenZiti identities. OpenZiti is an open‑source zero‑trust networking platform that represents workloads or devices with verifiable identities. Treating SPIRE as a CA lets you issue ephemeral, non‑interactive identities—ideal for workloads such as Kubernetes pods—without exposing services to public networks.

How it works

From OpenZiti’s perspective, SPIRE is another CA that you control. SPIRE issues a client certificate for Workload1. If the issuer is not a root CA, this must be presented with its supporting intermediate issuers. Workload1 has a standard OpenZiti identity configuration, such as the following example, which the deployer must craft, referencing the client cert from SPIRE, the private key, and the trust bundle from the OpenZiti controller.

OpenZiti identity configuration example: work1.json

{
  "ztAPI": "https://ziti.example.com:443",
  "id": {
    "cert": "file://work1.chain.pem",
    "key": "file://work1.key",
    "ca": "file://cas.pem"
  }
}

The client certificate in work1.chain.pem must have URI SAN spiffe://spire1.ziti.example.com/workloads/work1.

The OpenZiti Edge SDK can load this configuration file and connect to OpenZiti services. With this approach, there’s no need to enroll the OpenZiti identity. Each short-lived certificate from SPIRE can be used to obtain a session authorized to perform any operation granted to the OpenZiti identity for which its external ID maps to the SPIFFE ID.

Configuring OpenZiti to trust SPIRE

Let's walk through an example procedure for configuring OpenZiti to trust a SPIRE CA, define the OpenZiti identity roles for Workload1's SPIFFE ID, and connect to an OpenZiti service with the client certificate from SPIRE.

Step 1: Obtain the trust bundle from the OpenZiti Controller

Each workload’s OpenZiti identity configuration file must reference this file in the id.ca property.

curl -skSf https://ziti.example.com:443/.well-known/est/cacerts \
| base64 -d \
| openssl pkcs7 -inform DER -outform PEM -print_certs \
| tee ./cas.pem

Step 2: Prove ownership and register the SPIRE CA with OpenZiti

You can use the ziti CLI if you have direct access to the SPIRE CA’s private key to verify it in a single step.

ziti edge create ca "spire1" ./spire/certs/root.cert \
    --location "SAN_URI" \
    --matcher "SCHEME" \
    --matcher-criteria "spiffe" \
    --parser "NONE" \
    --auth

ziti edge verify ca "spire1" \
    --cacert ./spire/certs/root.cert \
    --cakey ./spire/keys/root.key

Alternatively, request a client cert from SPIRE with the verification token as the CN part of the DN. Obtain the verification token (e.g., S6ZLJD7XL8) by listing the newly created CA.

ziti edge list cas

Step 3: Issue a client certificate with the verification token

Use the client certificate, with the verification token as the CN value of the subject DN, to prove ownership of the SPIRE CA.

ziti edge verify ca "spire1" \
    --cert ./S6ZLJD7XL8.cert

Now, the OpenZiti controller trusts the SPIRE CA to issue client certificates for OpenZiti identities.

Provisioning a workload identity

Create an OpenZiti identity with the desired role and the SPIFFE ID that SPIRE will assign to this workload. The role must match the OpenZiti policies that authorize the identity to use OpenZiti services and routers.

ziti edge create identity "work1" \
    --external-id spiffe://spire1.ziti.example.com/workloads/work1 \
    --role-attributes httpbin-clients

Finally, you only need the standard OpenZiti identity configuration JSON file described earlier, referencing the client cert from SPIRE with a matching SPIFFE ID in the URI SAN.

Any OpenZiti Edge SDK can load this standard identity configuration file and use OpenZiti services.

Using the identity with the zitify tool

Here's how it would look to use the zitify tool on Linux—a shell script that invokes the OpenZiti Python SDK to “wrap” another program with Ziti sauce via LD_PRELOAD. We'll use curl in this example. The OpenZiti service address (i.e., the “intercept”) in this case is httpbin.ziti.internal:80.

zitify -i work1.json curl --data ziti=works http://httpbin.ziti.internal/post

The response is JSON from Httpbin containing the request’s form data.

Using the identity with a tunneler

You can use an OpenZiti identity configuration file with OpenZiti tunnelers, like ziti-edge-tunnel, which load identities from a standard configuration file by placing the file in the identity directory before starting the tunnel. The tunneler must also be able to read the referenced cert, key, and CA files.

Using the identity with the CLI

Authentication works identically for a network administrator, so a SPIRE-authenticated workload can also operate on the OpenZiti network via the management API.

For example, you can provision an admin identity and use its configuration file with the ziti edge administrative CLI commands.

ziti edge create identity "admin2" \
    --external-id spiffe://spire1.ziti.example.com/workloads/admin2 \
    --admin

Create an identity configuration file referencing the client certificate issued by SPIRE.

OpenZiti identity configuration example: admin2.json

{
  "ztAPI": "https://ziti.example.com:443",
  "id": {
    "cert": "file://admin2.chain.pem",
    "key": "file://admin2.key",
    "ca": "file://cas.pem"
  }
}

The client certificate in admin2.chain.pem must have URI SAN spiffe://spire1.ziti.example.com/workloads/admin2.

Then, log in to the OpenZiti management API using the configuration file.

ziti edge login --file admin2.json

...and perform administrative tasks.

ziti edge list summary

Conclusion

By leveraging SPIRE as a trusted CA within OpenZiti, we can streamline the management of workload identities, especially for short-lived, non-interactive services. This integration enhances security and simplifies identity provisioning, making your zero-trust network more robust and easier to manage.

References

• Using External CAs with OpenZiti

The short version: the Ziti controller exposes a management WebSocket at /fabric/v1/ws-api that streams real-time events - usage metrics, session lifecycle, circuit events, etc. It speaks a binary "channel V2" protocol over the WebSocket, which is the same protocol the Ziti infrastructure components use to talk to each other. Once you know the wire format, it's pretty straightforward to connect, subscribe, and start processing events.

The Discourse thread above includes a working JavaScript implementation if that's your language of choice. The rest of this post focuses on Go, since that's what I used, but the concepts are the same regardless of language.

If you're using Go, you can skip the manual wire protocol handling entirely. The OpenZiti project provides the channel library, which handles all the binary framing, message routing, and connection lifecycle. The Ziti CLI's own stream events command uses these libraries - you build your subscription as JSON, send it as a channel message, and register a receive handler. The library takes care of everything else. The rest of this post covers the wire protocol directly, which is useful if you're not in Go or want to avoid the dependency.

Authenticating

First you need a session token. The event stream endpoint uses the same auth as the rest of the management API. You have a couple options:

Username/password:

POST https://<controller>/edge/management/v1/authenticate?method=password
Content-Type: application/json

{"username": "admin", "password": "your-password"}

Certificate-based:

POST https://<controller>/edge/management/v1/authenticate?method=cert

No body needed - the client certificate from the TLS handshake is the credential. You'd use the cert and key from a Ziti identity file for this.

Either way, the response gives you a token in data.token. Hold on to it.

If you're using Go, the Ziti SDK handles this for you:

auth := rest_util.NewAuthenticatorUpdb("admin", "password")
auth.RootCas = caPool
session, err := auth.Authenticate(controllerURL)
token := *session.Token

Opening the WebSocket

Connect to wss://<controller>/fabric/v1/ws-api with your session token in a zt-session header.

One thing that tripped me up: the controller's WebSocket handler uses http.Hijacker, which isn't available over HTTP/2. You need to force HTTP/1.1 for the WebSocket handshake. In Go:

tlsCfg := &tls.Config{
    RootCAs:    rootCAs,
    NextProtos: []string{"http/1.1"},
}
httpClient := &http.Client{
    Transport: &http.Transport{
        TLSClientConfig:  tlsCfg,
        ForceAttemptHTTP2: false,
    },
}

opts := &websocket.DialOptions{
    HTTPClient: httpClient,
    HTTPHeader: http.Header{
        "zt-session":      []string{token},
        "Accept-Encoding": []string{"identity"},
    },
    CompressionMode: websocket.CompressionDisabled,
}
conn, _, err := websocket.Dial(ctx, "wss://controller:1280/fabric/v1/ws-api", opts)

I'm using github.com/coder/websocket here, but any WebSocket library that lets you set headers and force HTTP/1.1 will work.

The Channel V2 Wire Protocol

This is where it gets interesting. The WebSocket doesn't carry plain JSON - each binary message is a "channel V2" message with this layout:

Offset  Size  Field
------  ----  -----
0       4     Magic bytes: 0x03 0x06 0x09 0x0C
4       4     Content type (uint32 LE)
8       4     Sequence number (uint32 LE)
12      4     Headers length (uint32 LE)
16      4     Body length (uint32 LE)
20      var   Headers
20+H    var   Body (JSON)

Headers are sequential key-value pairs: [4B key LE] [4B value_len LE] [value bytes].

The content types you care about:

And two header keys:

Building and parsing these messages is mechanical. Here's Go code for both:

var channelMagic = []byte{0x03, 0x06, 0x09, 0x0c}

func channelMarshal(contentType, sequence uint32, headers map[uint32][]byte, body []byte) []byte {
    var hdrs bytes.Buffer
    for k, v := range headers {
        binary.Write(&hdrs, binary.LittleEndian, k)
        binary.Write(&hdrs, binary.LittleEndian, uint32(len(v)))
        hdrs.Write(v)
    }
    hdrBytes := hdrs.Bytes()

    var buf bytes.Buffer
    buf.Write(channelMagic)
    binary.Write(&buf, binary.LittleEndian, contentType)
    binary.Write(&buf, binary.LittleEndian, sequence)
    binary.Write(&buf, binary.LittleEndian, uint32(len(hdrBytes)))
    binary.Write(&buf, binary.LittleEndian, uint32(len(body)))
    buf.Write(hdrBytes)
    buf.Write(body)
    return buf.Bytes()
}

func channelUnmarshal(data []byte) (contentType, sequence uint32, headers map[uint32][]byte, body []byte, err error) {
    if len(data) < 20 || !bytes.Equal(data[:4], channelMagic) {
        return 0, 0, nil, nil, fmt.Errorf("invalid channel message")
    }

    contentType = binary.LittleEndian.Uint32(data[4:8])
    sequence = binary.LittleEndian.Uint32(data[8:12])
    hdrLen := binary.LittleEndian.Uint32(data[12:16])
    bodyLen := binary.LittleEndian.Uint32(data[16:20])

    headers = make(map[uint32][]byte)
    hdrData := data[20 : 20+hdrLen]
    for i := 0; i < len(hdrData); {
        if i+8 > len(hdrData) { break }
        key := binary.LittleEndian.Uint32(hdrData[i : i+4])
        vlen := binary.LittleEndian.Uint32(hdrData[i+4 : i+8])
        headers[key] = hdrData[i+8 : i+8+vlen]
        i += 8 + int(vlen)
    }

    body = data[20+hdrLen : 20+hdrLen+bodyLen]
    return
}

Subscribing to Events

Send a StreamEventsRequest (content type 10040) with a JSON body listing what you want:

sub := map[string]any{
    "format": "json",
    "subscriptions": []map[string]any{
        {"type": "fabric.usage", "options": map[string]any{"version": 3}},
        {"type": "edge.sessions"},
    },
}
body, _ := json.Marshal(sub)
msg := channelMarshal(10040, 1, nil, body)
conn.Write(ctx, websocket.MessageBinary, msg)

Then read messages until you get a Result (content type 2) that replies to your sequence number. Check that header key 2 has a first byte of 1 for success.

Available event types include fabric.usage, edge.sessions, fabric.circuits, edge.routers, services, fabric.links, and fabric.routers. For usage events, request version 3 - it gives you a cleaner format with separate tags and usage maps.

Reading Events

From here it's a read loop. Filter for content type 10041 and parse the JSON body:

for {
    _, data, err := conn.Read(ctx)
    if err != nil {
        break // reconnect
    }
    ct, _, _, body, err := channelUnmarshal(data)
    if err != nil || ct != 10041 {
        continue
    }

    var event struct {
        Namespace string            `json:"namespace"`
        Tags      map[string]string `json:"tags"`
        Usage     map[string]uint64 `json:"usage"`
    }
    json.Unmarshal(body, &event)

    if event.Namespace == "fabric.usage" {
        fmt.Printf("service=%s client=%s tx=%d rx=%d\n",
            event.Tags["serviceId"], event.Tags["clientId"],
            event.Usage["ingress.tx"], event.Usage["ingress.rx"])
    }
}

Usage events include both client and host identity IDs in the tags (clientId and hostId), along with the serviceId. The usage map has directional counters: ingress.tx and egress.rx represent client upload, ingress.rx and egress.tx represent client download.

Things to Know

The controller doesn't buffer events. If your WebSocket disconnects, events that occur during the gap are lost. Build your client to reconnect with backoff and re-subscribe. Session tokens expire, so you'll want to re-authenticate on reconnect too.

Usage events arrive roughly every minute. The controller aggregates and emits them on an interval, so don't expect sub-second granularity.

HTTP/2 will silently break the handshake. I learned this the hard way, but the controller requires HTTP/1.1 for the WebSocket upgrade. Force it explicitly in your client.

For self-hosted controllers that use Ziti's internal CA for TLS, you can fetch the CA bundle from the controller's well-known endpoint at /edge/client/v1/.well-known/est/cacerts and add those CAs to your trust store alongside system CAs. The Go SDK has rest_util.GetControllerWellKnownCas() for this.

A note on NetFoundry-hosted controllers. If you're connecting to a NetFoundry controller with a DNS name ending in -p, you're talking to a controller configured with publicly signed TLS certificates. Standard system CA trust stores will work - no custom CA bundles needed. Note that some NetFoundry environments require connecting to the management API through Ziti itself (i.e., the management API isn't directly reachable on the internet). Check with your NetFoundry contact if you're not sure which applies to you, but if you can't access the Ziti management API from a NetFoundry-hosted controller, this is probably why.

That's pretty much it. The wire protocol looks intimidating at first, but it's simple once you've written the marshal/unmarshal functions. From there, it's just JSON.

If you have questions or run into issues, the OpenZiti Discourse is the best place to ask. And if you do something interesting with the event stream, I'd love to hear about it.

TL;DR: Starting with OpenZiti 1.8, controller APIs can bind as OpenZiti services. The same app-embedded zero-trust that secures your applications now secures the management plane itself.

Application-Embedded Zero-Trust Management

OpenZiti has been bringing zero trust to applications since 2019. Embed an OpenZiti SDK into your app, and it immediately gains significant security benefits—no need to open ports, no need to expose services to IP-based networks, no attack surface on the underlay. Your app becomes dark to the network, accessible only through authenticated identities.

You've always had options when securing the controller's management APIs. Split the APIs off that need to be more secure onto private IPs, use internal networks, and offload to the API through nearby tunnelers. These work great and remain solid approaches. But now there's a new option that takes it further with full application-embedded zero-trust for the controller itself.

What Changed in 1.8

Starting in 1.8, securing those critical APIs changes. Now, XWeb supports overlay-based bindPoints directly! The APIs that shouldn't be exposed to the underlay can be removed from the underlay entirely. Have a look at this bit of controller configuration:

web:
  - name: ziti-management
    bindPoints:
      - identity:
          file: "/path/to/identity.json"
          service: "ziti-controller-api"
    apis:
      - binding: edge-management
      - binding: fabric

Instead of your bindPoint using an interface and a port like you've been doing for years, now you can specify an OpenZiti identity and bind the Controller APIs directly as OpenZiti services! Now, just like you do with your other services, these critical APIs can be accessed through the OpenZiti overlay itself with no need to bind the services to a tunneler and offload back into private IP space. Naturally, we expect most users will make an identity from their own OpenZiti network; however, you're not limited to using the same overlay you're protecting. If you need or want to use a different OpenZiti overlay to protect the controller's APIs, that's fine. You only need to supply the proper configuration, and the controller will bind the services to the overlay you specify.

Why This Changes Everything

If you're reading this blog, you probably understand already why this is so important.

No attack surface. Your controller doesn't need internet-facing interfaces for sensitive APIs. It's not in DNS. It can't be scanned. Can't be found. Even if an attacker can make local network requests on that private network, those requests can never reach those sensitive APIs.

Manage from anywhere. Only authenticated and authorized identities get access. Period. No VPN. No bastion host. No "are you on the right network" nonsense.

True zero-trust. The network that provides identity-based segmentation for applications now uses identity-based segmentation for its own management plane.

The Bigger Picture

This isn't just a feature. It's philosophical consistency. We tell users: "Don't trust networks. Explicitly trust connections using strong identities. Authenticate and authorize before allowing inbound connections. Hide everything. Verify always."

Routers already authenticate and authorize before connecting. You should be able to do the same with your sensitive management plane APIs, and now with 1.8, you can.

The entire management plane can live behind zero-trust segmentation.

The ziti CLI Goes App-Embedded Too

With controller APIs now exposed as OpenZiti services, the ziti CLI is now able to connect through the overlay just like any other zero-trust application. Enable the ziti CLI to use an overlay network by providing the --network-identity parameter during login, then just use the ziti CLI normally:

ziti edge login mgmt.ziti \
  --network-identity /path/to/network-identity.json \
  --username admin \
  --password admin \

Now, the CLI becomes a zero-trust client. It doesn't need network routes to the controller, doesn't need to know the controller's IP address, and doesn't need firewall exceptions. It just needs a valid identity with the right permissions. Manage your network securely from anywhere. The overlay handles the rest.

No special underlay rules required. Just OpenZiti.

Getting Started

If you're running 1.8+ and want to secure your APIs using an identity, follow these steps:

1. Create an identity for your controller to bind API access with

2. Define a service for your controller APIs and authorize the identity created above

3. Split the sensitive APIs off into a separate web section

4. Update the web section with overlay bindPoints by adding an identity accordingly

5. Create and authorize an identity to use to access (dial) the management plane APIs once secured

The network is the security. Now for everything.

Share the Project

If you find this interesting, please consider starring us on GitHub. It really does help to support the project! And if you haven't seen it yet, check out zrok.io. It's totally free sharing platform built on OpenZiti! It uses the OpenZiti Go SDK since it's a ziti-native application. It's also all open source too!

Tell us how you're using OpenZiti on X twitter, reddit, or over at our Discourse. Or you prefer, check out our content on YouTube if that's more your speed. Regardless of how, we'd love to hear from you.

It’s always a big deal when a project releases “version 1.0”. Our version 1.0 release validates all of the good things about zrok while improving a few areas that needed it, and also brings several exciting new features.

The initial release of 1.0 includes major new features like:

• A new polished API console (web interface)

• A new “zrok Agent” designed to streamline the management of your zrok resources and provide a web interface making zrok accessible to users who aren’t command-line natives

• The “zrok Canary” a new suite of tests designed to focus on both performance and reliability for production zrok environment

We’re going to take a look at these highlights of zrok 1.0 in this blog post. For those of you who prefer videos, I recently did a zrok Office Hours covering this same information:

History

I keep a detailed journal of my work. I like to save screenshots documenting interesting points in the development of the projects that I work on. Sometimes it’s fun to go back and look at them. Here’s a little look into zrok screenshots over the years.

First, starting in 2022, here is a v0.2 screenshot. v0.2 predates zrok.io and the hosted service. This is the very first version of zrok that had a “console” and integrated metrics:

v0.2 also predates private sharing in zrok. This was just a proof-of-concept exercise to see what a secure public reverse proxy solution on top of OpenZiti might look like.

Then we developed v0.3, which added private sharing and introduced the concept of a “backend mode”:

The “visualizer” concept was first introduced in v0.3. This version of the visualizer is similar to what we ended up with in v0.4.

The main goal of the v0.4 series was to develop zrok into a platform that could support a zrok as a service (zrok.io). A lot of refinement and polish went into making v0.4 usable and useful for a wider range of users:

Every time there is a major refresh of the zrok user interface, I feel like the new version feels like a substantial leap forward.

This brings us to the first major feature of zrok v1.0, which is the updated API console.

The New API Console

The new zrok API console (the primary “web interface”) has received a major refresh for v1.0. In addition to new branding, the technology under the covers has been streamlined as well. Here are some screenshots so you can get the vibes:

These things are subjective, but I very much look at this new user interface and feel the same way about it that I’ve felt about previous iterations… it looks like a substantial improvement in polish and usability versus the previous version.

Under the covers the API console has been re-developed in Typescript using Vite. It’s still a React application, but the stack has been radically cleaned up and simplified. This new stack will give us a solid platform to extend and build on going forward.

The new API console retains the same visual approach to navigating your zrok resources as the previous release… but in addition, the new API console provides a “tabular” mode to easily filter and sort the contents of your account:

When sorted by activity, the tabular mode operates kind of like a “top” command for your zrok account.

For users with large, active accounts, the sorting and filtering features can help you to zero in on the specific resources you’re looking for. The selection state is maintained between the visualizer and tabular modes, allowing you to search and then quickly visualize.

The forms and dialogs have also received a lot of attention and polish:

We’re excited about the new streamlined look-and-feel, too.

zrok Agent

If you’re a zrok power user who has ever used more than a single zrok share or zrok access, you’ve surely run into the issue where you’ve needed to manage a number of zrok processes. Some of them might go in systemd. Some of them might sit in terminal windows. You might even nohup some of them into the background.

zrok v1.0 introduces a new “zrok Agent” to make this situation much simpler. The Agent is a kind of intelligent “process manager” for all of your zrok shares and accesses. It’s designed to work well for both end-user environments (for developers and end-users) and also for headless production environments.

When you’re not running the zrok Agent, the v1.0 command-line works the way it always has… If you run a zrok share, you’ll end up with a single process for that share, like this:

But when I start the new zrok Agent using the zrok agent start command, we get a different operating mode. With the zrok Agent running, if I do a zrok share, I get a different result:

Notice that the zrok share public command above returned immediately, displaying the share token and the URL for my new share. When I ran zrok share public with the Agent running, the Agent was detected and the process for my new share was managed by the Agent automatically.

With the zrok Agent you’ll have a single process to manage. You can put your zrok agent start command into systemd using the zrok-agent package for Linux. You can nohup it. You can just stash it away in a terminal window somewhere. You can run it as a Windows service. But once you’ve managed that single process, your whole zrok experience becomes simpler.

You might also notice the zrok agent status command in the terminal above. That command is used to display the shares and accesses being managed by the Agent.

There are additional commands under the zrok agent tree for managing shares and accesses:

And the zrok agent release share command can direct the agent to release the share.

The zrok Agent also includes a localhost web interface, which can be used to create and remove shares from the agent… yes, when you’re running the zrok Agent you can create new shares without using the command-line interface!

The zrok “Agent console” is an early preview and a work-in-progress. There are currently a limited subset of share types that can be created through the Agent console, but we expect to elaborate this into a first-class interface over the next handful of releases.

We expect the zrok Agent console to grow into a super useful desktop assistant for managing your local zrok shares and accesses. This will be especially helpful for non-CLI users.

Remote Agent Administration

We’re not quite there yet, but we’re very excited about one of the features we’re actively developing for the zrok Agent… opt-in “remote administration”. Imagine you have a headless system somewhere with an enabled zrok environment running a zrok Agent. Sure, you could ssh into that system to create and manage shares. But what if you could simply enroll your zrok Agent so that it can be managed from the central zrok API console… from anywhere?

This kind of remote administration would allow you to create and release shares and accesses on remote environments through the zrok API console. Need to get into a remote system without direct access? This could be a way to make that very simple. This also allows you to shut down shares and accesses when you’re not using them, increasing security.

Remote administration will be completely opt-in and will be the kind of capability you can enable only when you need it. If you do not enroll your Agent, then that Agent is an island unto itself that you can only access locally. You’ll be in control.

zrok Canary

We take the reliability and performance of zrok very seriously. So seriously that we’re developing a set of performance and reliability tooling into zrok that we collectively refer to as the “zrok Canary”.

For the initial 1.0 release, we’ve re-worked the old zrok test loop public infrastructure into a more polished set of tools underneath zrok test canary.

We expect to continue elaborating these tools into a wide suite of monitors and tests to properly validate all of the major operating conditions of a zrok instance. And we expect to be rolling these tools out across zrok.io as they’re available to give us deeper and better visibility into how zrok is operating.

Switching to 1.0

To switch to 1.0, simply obtain a zrok binary at v1.0.2 or later, and your existing environments will automatically be migrated to the new version, transparently.

The Medium-term Roadmap

There are a lot of exciting new features and capabilities lined up for the v1.0.x release series!

zrok for OpenZiti Networks

Have an existing OpenZiti network? Want to quickly and easily add zrok capabilities to your OpenZiti network on-demand? We’re taking a good look at how best to add and remove zrok as a capability of an existing OpenZiti network.

Remote Agent Administration

Enroll your zrok Agent for remote administration through the zrok API console. Un-enroll the instant you’re done. All communications between your zrok Agent and the zrok instance will be handled securely through the OpenZiti overlay.

A Complete zrok Agent Console

Do everything through the zrok Agent console that you can do through the command-line. Keep this interface on your desktop and never touch the zrok CLI again.

Reserved Namespace Improvements

Everyone wants the same names for their reserved shares: staging, testing, etc. We’ll be taking a good hard look at how to give users better, more specific namespaces allowing them to use the names that they want to use.

More zrok Drives; Making Drives More Useful

Magic wormholes. Drive-by upload boxes. Ephemeral downloads. We’ll be looking at ways to build on the proof-of-concept that is the current zrok Drives implementation, making it useful for all kinds of data-in-motion use cases.

API Console Timeline and Notifications

See how your zrok resources have changed over time. Answer questions like “when exactly did I remove that share?” We’ll also be looking at notifications and working on providing better mechanisms for communicating with users.

OpenZiti HA Control Plane Integration

OpenZiti has an HA control plane and we’re going to teach zrok how best to take advantage of that. zrok is one of our examples of best practices around “Ziti Native Applications” and we’re going to continue and expand that tradition to include HA.

Thank you!

Thank you for supporting zrok. We’re really looking forward to seeing where the 1.0 roadmap takes us.

If you like zrok, it’s always very much appreciated if you take a moment to drop a star onto the zrok repository on GitHub. We literally see and appreciate every repository star.

If zrok is an important part of your workflow, maybe you would consider getting a subscription on zrok.io? Those subscriptions are an incredibly helpful signal that we’re on the right track.

If there is anything we can do to improve zrok or if you’ve run into anything we can help with, please reach out to us at our OpenZiti Discourse forum. There is a zrok category, and we’re standing by ready to help.

As part of the larger open source OpenZiti project, zrok uses an OpenZiti overlay network for any secure communications. When you start an instance of zrok, zrok will create a secure zero trust connection to the OpenZiti overlay. This connection is also how zrok can provide truly end-to-end encrypted tunnels with the --tcpTunnel backend mode. Given its nature, it’s difficult to know what services zrok is providing, but we were able to run some statistics based on the terminating port and make some inferences on whether or not the port was exposed as a public or private share. Here’s the list that we came up with. I found some of these use cases particularly cool, and I discovered some use cases in the process I’d never actually thought of. We’re hoping this post gives you some fresh eyes with zrok and you discover some new possibilities for how powerful it can be for resource sharing!

#1. Local Development Servers

Ports: 80,8000,8080,3000,3001,5173

No surprise here! This is probably the original use case that zrok was built around, and zrok just makes it so easy to share your local environment and alleviates the need to stand up an entire shared environment just to be able to show off a local demo, collaborate on a pre-release, or share resources in a distributed developer environment. Whether you’re running apache, nginx, nodejs, or Vite, zrok allows you to easily share any of these technologies.

zrok share public 8080

#2. Minecraft Servers

Port: 25565

Still one of the best games of all time, and considering how easy it is to run your own server and add custom mods, zrok provides an easy way to share a private game server with your friends without having to open up your firewall ports to the world.

See this tutorial for full details on how to run your minecraft server over zrok!

#3. AWS Sagemaker Notebooks

Port: 7860

About a year ago we had a huge amount of buzz from the community that outlined how to integrate zrok with AWS Sagemaker notebooks. Pogs Cafe put together some great video content that shows zrok integrated with their launcher for ephemeral AI image generation. See zrok in action here, or check out their blog post.

#4. BlueBubbles

Port: 1234

BlueBubbles is a free, open-source app ecosystem that allows users to send and receive iMessages on non-Apple devices. BlueBubbles leverages zrok as one of its built-in proxy services so that you can run BlueBubbles server without the need for port-forwarding.

#5. IRC - Internet Relay Chat

Port: 6666

The original chat server, has been alive since 1988. zrok provides a way for you to easily put modern security around your chat server with end-to-end encryption, and private access tokens with the use of private TCP shares.

#6. SSH

Port: 22

Who needs a Bastion anyway? That’s so 2010! Just leave 22 closed to the world and put a zrok private TCP share on the host and call it a day. No open ports. Simply fire up a zrok share as a systemd service

#7 - Jupyter Notebooks / AI Notebooks

Port: 8188

This one was a little bit ambiguous because there are multiple AI Notebook tools that all rely on this same default port. We know that Jupyter Notebooks is one of the biggest uses in this bucket, and some cloud providers like Google Cloud AI and Hauwei Cloud AI also make use of these tools.

#8. Squid Proxy

Port: 3128

Squid proxy is an easy-to-use caching and proxy server that can be used to give greater control over outbound traffic whether you’re testing compatibility with a corporate proxy for your application, or you need funnel traffic through a remote server’s address, zrok gives you an easy way to set this up. Just create a share that terminates on your proxy service and set your HTTP_PROXY/HTTPS_PROXY variables in your environment, and all http-based traffic will now filter through your zrok proxy.

#9. FoundryVTT

Port: 30000

Probably my personal favorite use case for zrok. FoundryVTT is a virtual table top game server, and zrok provides a fantastic way to host a virtual game night from your own PC while still protecting your home network. Check out this YouTube tutorial here!

#10. NoMachine Remote Desktop

Port 4000

Remote desktop is incredibly handy, but it’s also something you would never want to expose as an open port on the internet. NoMachine is a free multi-platform remote desktop tool. With the use of private TCP shares, zrok enables you to maintain secure remote access from anywhere without ever exposing a port to the internet.

#??? - You Tell Us!!

Port: ???

Got a killer idea for zrok that we missed? Post it in the comments! We always love to hear new use cases for how people are using zrok so that we can make this technology better. And if you haven’t already, help us spread the word about zrok by giving us a star on github!

This is How I (en)Roll

On my Linux(Ubuntu) laptop I check that the kernel detected TPM hardware and created a device for it.

ziggy@hermes:~$ ls -l /dev/tpm*
crw-rw---- 1 tss root  10,   224 Apr 26 14:41 /dev/tpm0
crw-rw---- 1 tss tss  253, 65536 Apr 26 14:41 /dev/tpmrm0

Device /dev/tpmrm0 is there, so I can proceed.

Add my user to the tss group so that the TPM device can be accessed. You will probably need to restart your login shell for that to take effect, check your groups with id command.

ziggy@hermes:~$ sudo usermod -G tss -a ziggy

Install some useful software packages to interact with the TPM device

• libtpm2-pkcs11-tools - includes tmp2_ptool to initialize PKCS#11 token on TPM device

• libtpm2-pkcs11-1 - TPM PKCS#11 library

• opensc - includes pkcs11-tool for interacting with PKCS#11 token. This is not needed for OpenZiti enrollment but is useful for inspecting tokens

ziggy@hermes:~$ sudo apt install libtpm2-pkcs11-tools libtpm2-pkcs11-1 opensc

Now that I have access to the device and the needed software I can initialize the token and test PKCS#11 driver access to it

ziggy@hermes:~$ tpm2_ptool init
action: Created
id: 1

ziggy@hermes:~$ tpm2_ptool addtoken --pid 1 --sopin 1111 --userpin ziggy1 --label ziggy-tpm

ziggy@hermes:~/ziti$ pkcs11-tool --module /usr/lib/x86_64-linux-gnu/pkcs11/libtpm2_pkcs11.so --list-slots
WARNING: Getting tokens from fapi backend failed.
Available slots:
Slot 0 (0x1): ziggy-tpm
  token label        : ziggy-tpm
  token manufacturer : Infineon
  token model        : SLB9665
  token flags        : login required, rng, token initialized, PIN initialized
  hardware version   : 1.16
  firmware version   : 5.62
  serial num         : 0000000000000000
  pin min/max        : 0/128
Slot 1 (0x2): 
  token state:   uninitialized

Note: fapi backend warning can be ignored.

To enroll I need to get an enrollment token from my Ziti controller. Once I have that I can enroll. Let’s enroll with ziti-edge-tunnel

# enroll identity with ziti-edge-tunnel
ziggy@hermes:~$ ziti-edge-tunnel enroll -j ./ziggy.jwt -i ziggy.json -k 'pkcs11:///usr/lib/x86_64-linux-gnu/pkcs11/libtpm2_pkcs11.so?label=ziggy-key&pin=ziggy1'
(1676722)[        0.000]    INFO ziti-sdk:utils.c:198 ziti_log_set_level() set log level: root=3/INFO
(1676722)[        0.000]    INFO ziti-sdk:utils.c:167 ziti_log_init() Ziti C SDK version 1.5.0 @ga39db85(HEAD) starting at (2025-02-24T15:21:31.292)
(1676722)[        0.000]    INFO ziti-sdk:ziti_enroll.c:112 ziti_enroll() Ziti C SDK version 1.5.0 @ga39db85(HEAD) starting enrollment at (2025-02-24T15:21:31.294)
(1676722)[        0.000]    INFO ziti-sdk:ziti_enroll.c:221 start_enrollment() pkcs11 key not found. trying to generate

ZIti identity is stores in the JSON file -- ziggy.json in my case. Let take a look

{
  "ztAPI": "https://<my-controller-address>:443",
  "id": {
    "cert": "-----BEGIN CERTIFICATE-----....",
    "key": "pkcs11:///usr/lib/x86_64-linux-gnu/libtpm2_pkcs11.so?label=ziggy-key&pin=ziggy1",
    "ca": "-----BEGIN CERTIFICATE-----...."
   }
}

As you can see the .id.key is referencing the private key stored in my laptop's TPM hardware. I can inspect it using pkcs11-tool

ziggy@hermes:~$ pkcs11-tool --module /usr/lib/x86_64-linux-gnu/pkcs11/libtpm2_pkcs11.so --list-objects --pin ziggy1          1 ↵
Using slot 0 with a present token (0x1)
Public Key Object; EC  EC_POINT 384 bits
  EC_POINT:   04610442fa885e120e9c63227aa7c53aeb84a52400264f3452fcf4dc4dbf4ade9ca46e7e15c4d37cc7a317edd860887ccd5746eea70cdea1d106b36b59297ecb93d43c57f4e8aef7029fa55f52cd740292d8de92fbce35c7788d3cc00f528cac2d0e4d
  EC_PARAMS:  06052b81040022 (OID 1.3.132.0.34)
  label:      ziggy-key
  ID:         dad9d0f20cc1d9ec
  Usage:      encrypt, verify, wrap
  Access:     local
Private Key Object; EC
  label:      ziggy-key
  ID:         dad9d0f20cc1d9ec
  Usage:      decrypt, sign, unwrap
  Access:     sensitive, always sensitive, never extractable, local
  Allowed mechanisms: ECDSA,ECDSA-SHA1,ECDSA-SHA256,ECDSA-SHA384,ECDSA-SHA512

As you can see my TPM token ziggy-tpm now has two objects private, and public keys with same label and id. OpenZiti SDK will use that hardware key to provide identity.

Now I can start ziti-edge-tunnel with that identity:

ziggy@hermes:~$ sudo -E ziti-edge-tunnel run -i ziggy.json

Note: -E flag is required since I initialized TPM token as user ziggy and there are some support files created in the user’s $HOME directory.

TL;DR

You can now use your own domain when creating zrok shares! With a zrok custom domain you’ll be able to create shares like <token>.your.domain.com instead of <token>.share.zrok.io. You’ll need to bring your own domain and have a myzrok.io Pro plan to get started. Creating a zrok custom domain via myzrok will create a zrok frontend for you. This will allow you to create ephemeral shares or reserved shares using your domain name instead of zrok’s!

What is myzrok?

myzrok is the self-service billing portal for your zrok account. This is where you can upgrade your zrok to the different plans outlined on our pricing page to receive increased data, share, and environment limits based on your needs. This is also where you will now be able to create and manage custom domains if you have a Pro plan!

Why Custom Domains?

Up to this point, all zrok hosted shares have been created using the share.zrok.io domain. This is great when you want to get something hosted quickly and easily. However, there are situations where it would be really nice to host a share on your own domain to make it identifiably yours. This is where custom domains come into play.

How Does it Work?

You can set up your custom domain in just a few easy steps:

1. Bring your own custom domain name

2. Create DNS records for certificate validation and traffic routing

3. Wait for zrok to validate your records and finalize configuration

4. Start sharing!

You only need to do this once, after that you can create your shares like normal but utilize your domain instead of zrok’s!

Can’t wait to get started? Check out the step by step instructions in our guide here!

Custom Domains in Action

Let’s take a look at how custom domains make zrok even more useful. Let’s say that I own a company acme-corp and I’ve got an upcoming convention I would really like to demo my latest software at. Rather than deploying to an environment, I could run my demo locally and use zrok to create a reverse proxy, which would allow others to access it. This works great, but by sharing my demo via a zrok share command, I would end up with a share link like https://sjy8zgif4s8w.share.zrok.io/. While this is nice and simple, the link generated doesn’t really identify my company, making it harder to leave a lasting impact. This is where a zrok custom domain comes in!

To do this I first create a zrok custom domain demo.acme-corp.com using the acme-corp.com domain.

Once the custom domain is created, I simply create a few DNS records for acme-corp.com. Within a few minutes I’m able to use the demo.acme-corp.com domain in my zrok shares!

By creating a custom domain, I have also created a zrok frontend. This will create shares in the form of <token>.demo.acme-corp.com. This is much more identifiable to my company however, it still leaves me with the share token at the start of my share. I can improve this by creating a reserved share. This will reserve a unique identifier for me to use in place of my share token. Not only can I make my share even more branded towards my specific use case, but I can also start and stop it and still maintain the same link!

I’ll first set my frontend as the default for my environment, then I’ll create a reserved share for the resource I want to share. In this case it’s a demo app running on localhost:4000

zrok config set defaultFrontend demo-acme-corp_YS6RyLSod
zrok configuration updated


zrok reserve public --unique-name "summit" http://localhost:4000
[   0.388]    INFO main.(*reserveCommand).run: your reserved share token is 'summit'
[   0.388]    INFO main.(*reserveCommand).run: reserved frontend endpoint: https://summit.demo.acme-corp.com

zrok share reserved "summit"

I can then have people visit summit.demo.acme-corp.com to check my demo

Success!

This is a relatively simple example of how custom domains can be used to enhance your zrok experience. There are many times where I might want to share something a bit more permanent than a demo, such as a production app. This is where I could take advantage of zrok frontdoor alongside my custom domain. A zrok frontdoor would allow me to run zrok alongside my production app, enabling me to publicly expose access to it via a reserved share!

Wrapping Up

If you’re ready to dive in and create your own custom domain head over to myzrok.io, sign up for a Pro plan and navigate to the custom domains page ( icon). For a more in-depth look at how to setup a custom domain you can check out the guide here.

If you find this interesting and like what we’re doing, consider starring the project on GitHub! zrok is open source and built on OpenZiti, which is another great project to check out and star if you haven’t done so already!

If you want to show us how zrok or ziti have been improving your workflow or have feedback you’d like to share with the team we’d love to hear from you over at X, reddit, or at our Discourse!

This article walks through some of Go’s object oriented features and also discusses how some patterns common in other languages can be written in a way that’s closer to idiomatic Go. It also covers some features of Go that surprised and (in some cases) delighted me.

Is Go Object Oriented?

Go allows you to group data into structs, and to associate logic to those structs.

type Beans struct {
    Count int
}

func (beans *Beans) Spill(n int) {
    beans.Count = beans.Count - n
    fmt.Printf("spilled %d beans\n", n)
}

Since Go supports these simple features, it meets certain bare-minimum standard for object oriented coding.

However, as we’ll cover below, many features like inheritance, method overriding, and abstract types work differently or not at all.

Inheritance vs Composition

Even in languages with a strong object oriented bias, people often give the advice to favor composition over inheritance. Large complicated type hierarchies can lead to fragile designs and unexpected behavior.

Go is strongly biased towards composition. This, in my experience, leads to cleaner, simpler designs, that are easier to refactor and have less surprising behavior.

Note: Most of the patterns explored below aren’t exclusive to Go. Go just encourages them more than other languages may.

Fun with functions: Nil Receivers

One difference from languages like Java, is that a nil receiver is allowed and can be handled in Go code. This is the equivalent of being able to call a method on a null object in Java without a NullPointerException and have the method check if this is null.

type Beans struct {
    Count int
}

func (beans *Beans) Spill(n int) {
    if beans == nil {
        fmt.Printf("you've got no beans to spill!\n")
     } else {    
        beans.Count = beans.Count - n
        fmt.Printf("spilled %d beans\n", n)
    }
}

What do we want to do?

Rather than just look at language features, we’re going to look at what we’re trying to accomplish. Then we can see how to achieve our goals in Go, and how that might be different than a pure object-oriented approach.

Handle different types with a common set of methods

For polymorphic types, Go uses interfaces.

type Spillable interface {
    Spill(amount int)
}

type Beans struct {
    Count int
}

func (beans *Beans) Spill(n int) {
    beans.Count = beans.Count - n
    fmt.Printf("spilled %d beans\n", n)
}

type Tea struct {
    Volume int
}

func (t *Tea) Spill(n int) {
    t.Volume = t.Volume - n
    fmt.Printf("spilled %d ml of tea\n", n)
}

You may have noticed that the types don’t have to declare that they are Spillable.

Duck(-ish) Typing

Unlike many other languages, Go types do not have to explicitly state which interfaces they implement. If the type has the correct methods, it can be used as that interface type.

I’m not sure if it’s accurate to call this duck typing, since you still need to define interfaces. But from where I’m standing, it looks a lot like duck typing and sounds like duck typing, so I’m going to call it duck typing.

This focuses the attention on where the interface is used, not where it’s implemented. In cases where a type is used in multiple contexts, Go pushes you towards multiple interfaces, one for each context. In other languages, I’ve seen types end up with one massive interface, covering all the possible contexts where it might be needed.

Consider an HR application. It might have various entities

package employees

type Employee struct {
    // has name, email, address, salary, etc
}

package vendors

type Vendor struct {
   // has company name, email, address
}

You might have a couple of modules, one for sending out email notifications and another for payroll.

package emailer

type Contact interface {
   Email() string
   Name() string
}

func SendEmail(contact Contact, subj, body string) error { ... }

package payroll

type Payee interface {
    Name() string
    HomeAddress() string
    Salary() float32
}

func SendPaychecks(payee Payee) error { ... }

Note that the payroll and email packages don’t know anything about the employee and vendor packages.

Package Coherency

Go encourages keeping interfaces in the package where they are consumed. Let’s try adding an Email method to Employee.

func (emp* Employee) Email(string subj, string body) error {
    return email.Email(emp.EmailAddress, subj, body)
}

If the Contact interface was defined in the employee package, we’d end up with a circular package dependency error. If the emailer package didn’t have a Contact interface, and was able to take Employee instances directly, that would also cause a circular package dependency error.

I found circular package dependency errors to be very frustrating when I first started with Go. Now however, having gotten more comfortable with Go style interfaces, I’ve realized the following:

1. It’s OK to have smaller, highly specific interfaces per package. There may be some overlap, but each package will end up with exactly what’s needed. This makes the use clearer for developers using the package.

2. Having the interfaces in the package will also decouple the package from other packages, eliminating circular dependency issues.

I’m still occasionally annoyed by circular dependency errors, but overall feel that the limitation promotes package decoupling and coherency.

Requiring Type Interface Conformance

In most cases, if your type doesn’t match up with your interface, you’ll get a compile error. In some cases, where you may be doing runtime checks before converting, you might not notice that your type no longer fits the interface.

If you wish to guarantee at compile time that a specific type implements a given interface, you can do this as follows:

var _ Spillable = (*Tea)(nil)

Fun with functions: Function Vs Interface

Sometimes you may have an API that only requires a single method. Something like:

type MessageHandler interface {
    Handle(msg []byte) error
}

type MessageHandlers interface {
    AddHandler(msgType string, handler MessageHandler)
}

This could also be implemented using a function type.

type MessageHandler func (msg []byte) error

type MessageHandlers interface {
    AddHandler(msgType string, handler MessageHandler)
}

Which to use? For some library consumers a function may be more convenient. For others, an interface to implement would be better. Fortunately Go lets you handle both. In Go a function type can have methods defined on it.

This broke my brain slightly when I first encountered it, but it works well to allow both kinds of API.

type MessageHandler interface {
    Handle(msg []byte) error
}

type MessageHandlerF func (msg []byte) error

func (f MessageHandlerF) Handle(msg []byte) error {
    f(msg)
}

// example use
handlers.AddHandler(MessageHandlerF(func (msg []byte) error {
    return nil
}))

Extending Interfaces

Go interfaces can extend other interfaces. The classic examples are the types in the io package.

type Closer interface {
    Close() error
}

type Reader interface {
    Read(b []byte) (n, error)
}

type ReadCloser() {
   Closer
   Reader
}

Older versions of Go used to complain if you extended multiple interfaces that had overlapping method signatures. Recent versions of Go no longer have this limitation, as long as method signatures with the same name match exactly.

Anonymous Interfaces

There may be times when you want to invoke a method on a variable, but the method isn’t on the interface you’ve got. You can use an anonymous interface to check if the method exists, and if it does, invoke it.

I’m not sure there’s ever a strong reason to use an anonymous interface over declaring a named interface. I’ve used this feature when an interface was only used in a single method.

In this example we’re looking for a non-standard close method.

func CloseIt(v any) error {
    if closeable, ok := v.(interface{ CloseMe() error }); ok {
        return closeable.CloseMe()
    }
    return nil
}

We Want to Share Functionality Across Types

In object-oriented languages, extending a type (or sometimes multiple types) to create a new type is very common. Go does not allow structs to extend other structs in exactly the same way as most object oriented languages.

What it does offer is type composition that can look something like inheritance. We’ll look at how it works, and the potential pitfalls from this approach.

Go allows embedding both structs and interfaces.

type Knocker interface {
    Knock()
}

type Door struct {
    lock sync.Mutex
    knocker Knocker
}

The Door type has a lock and knocker. As they are, they are just struct members. Since they are private, they won’t be accessible outside the package. If you try to Lock the door or Knock at the door, the compiler will complain.

A door isn’t a lock or a knocker, it has a lock and and a knocker. However, it still makes sense to be able to Lock or Knock the door itself. We can do that as follows.

 type Knocker interface {
    Knock()
}

type Door struct {
    sync.Mutex
    Knocker
}

When we embed types in our struct this way, the methods and fields of the embedded types become accessible at the top-level of the struct. The embedded types can still be referenced individually using the type name.

door := &Door{
    Knocker : &BrassKnocker{},
}  

door.Mutex.Lock()
door.Unlock()
door.Knocker.Knock()
door.Knock()

Note that we could also embed a pointer type.

type Door struct {
    *sync.Mutex
    Knocker
}

Method Overriding vs Method Shadowing

Methods from embedded types can’t be overridden, in an OO sense, but they can be shadowed. Here’s an example to highlight the difference.

type A struct {}

func (a *A) String() string {
    return a.Name() 
}

func (a *A) Name() string {
   return "A"
}

type B struct {
    A
}

func (b *B) Name() string {
    return b.A.Name() + " or B"
}

func main () {
    a := &A{}
    b := &B{}
    fmt.Println(a.Name())    // output: "A"
    fmt.Println(a.String())  // output: "A"
    fmt.Println(b.Name())    // output: "A or B"
    fmt.Println(b.String())  // output: "A"      <-- Maybe unexpected
}

Because A doesn’t know that it’s embedded by B, it won’t call B's versions of methods that have been overridden. As long as you use type embedding with a composition mindset instead of an inheritance mindset, you should be OK.

I Want Abstract Types!

Abstract types can be a very useful way of providing some base functionality that relies on methods to be supplied by a sub-type. When I first started using Go, I was generally happy and productive with the language. Two things I missed from Java were generics and abstract data types. Generics have since come to Go in version 1.18, and I’ve since learned patterns better suited to Go to replace abstract data types.

Here’s a java example of an abstract data type:

public abstract class AbstractDataStore<T> {

    public abstract T LoadEntity(Row r);

    public List<T> ListEntities(Database db, String query) {
         Cursor c = db.ExecuteQuery(query);
         List<T> result = new ArrayList<T>(); 
         while (c.HasNext()) {
             Row row = c.Next();
             T entity = this.LoadEntity(row);
             result.Add(entity);
         }
         return result;
    }
}

We’ve got a data store that’s got some generic functionality to iterate over database results and build an entity list. Each concrete implementation would manage the details specific to that entity type.

The Wrong Way

The first time I tried modeling something like this in Go, I came up with a pattern similar to abstract types, but significantly worse because the underlying abstractions weren’t there to support it.

The idea was that if you defined the full API as an interface, the abstract type could keep a reference to the concrete type, and call it as necessary.

type DataStore[T any] interface {
    LoadEntity(r *Row) T
    ListEntities(db Database, query string) []T
}

type BaseStore[T any] struct {
    impl DataStore[T]
}

func (store *BaseStore[T]) ListEntities(db Database, query string) []T {
    cursor = db.ExecuteQuery(query)
    var result []T
    for cursor.HasNext() {
        row := cursor.Next()
        entity := store.impl.LoadEntity(r)
        result = append(result, entity)
    }
    return result
}

While this does work, it has a downside. If you treat it as an actual abstract type and try to override any of the base methods in the embedding type, they will only be shadowed. If you then ever forget to call the concrete version on the embedded impl, you’ll get the default version. This kind of bug can be easy to miss and hard to track down.

Subjectively, it also doesn’t feel right. It feels like we’re working against the language, rather than with it.

A Better Way

The solution above is not that far off from something that feels more in line with Go’s ethos.

What is it we’re trying to accomplish? We’ve got a core set of functionality that needs to delegate some logic. There’s another pattern that fits this description:

Strategies

Instead of trying to implement abstract types, we can use strategies. Let’s look at what a strategy oriented version of the above code would look like:

type EntityLoader[T any] interface {
    LoadEntity(r *Row) T
}

type Store[T any] struct {
    entityLoader EntityLoader[T]
}

func (store *Store[T]) ListEntities(db Database, query string) []T {
    cursor = db.ExecuteQuery(query)
    var result []T
    for cursor.HasNext() {
        row := cursor.Next()
        entity := store.entityLoader.LoadEntity(r)
        result = append(result, entity)
    }
    return result
}

It looks very similar, but the intent is different. This is a composition-oriented approach. Rather than extending a BaseStore, then overriding pieces, we compose the various strategies we might need.

This approach also lends itself to sharing functionality in a clean way. You might have three or four different strategies. Some of them might have default implementations that are rarely replaced. Some might have only a small set of implementations that are used across many instances. Some might be different for every instance.

Rather than trying to shape these as overrides in a convoluted type hierarchy, each instance can mix and match as needed.

Conclusion

Every language has some inbuilt preferences. If you’re coming from a heavily object-oriented language, it can take time to adapt how you design your data types in Go. However, I think Go pushes you towards code that is less fragile and cleaner. If I were to write Java code again, it would be positively influenced by the habits I’ve picked up from Go.

About OpenZiti

OpenZiti is an open-source platform for providing secure and reliable access to network applications. It does this using strong, certificate-based identities, end-to-end encryption, mesh networking, policy-based access control, and app-embedded SDKs. If you find this interesting, please consider starring us on GitHub. It helps to support the project! And if you haven't seen it yet, check out zrok.io. It's a free sharing platform built on OpenZiti! It uses the OpenZiti Go SDK since it's a ziti-native application. It's also all open source too!

Tell us how you're using OpenZiti on XTwitter, Reddit, or over at our Discourse. Or you can check out our content on YouTube if that's more your speed. Regardless of how, we'd love to hear from you.

A question that sometimes gets asked is: What types of companies, self-hosters, or general use cases, will gain the greatest benefit from using OpenZiti BrowZer?

To help answer the “who benefits” question, much like I did in a previous article, this article will frame the answer in the context of a web application named Portainer (a web-accessible container management platform) when it needs to be internet-facing.

NOTE: Upcoming articles will describe how similar techniques employing BrowZer can be used to protect and secure other popular web applications. So be sure to subscribe to this blog to be notified when those articles are published.

Why is Portainer interesting?

Portainer can be used to manage Docker, Kubernetes, and other container environments. It is primarily a web-based management interface, designed to be accessed via a browser. The web-based approach provides convenient access and centralized management, allowing users to interact with their container infrastructure …from anywhere.

Companies benefiting from Portainer include teams with limited in-house container management expertise, those with rapid deployment requirements, those having resource constraints/constrained IT budgets, or teams with members of varying skills, technical and non-technical. It also benefits those with a security and compliance focus.

Hobbyists and IT enthusiasts can also benefit from Portainer to manage containerized services for personal projects, home automation, media servers, and more.

Need to check on something in your docker fleet… while you are standing in line at the coffee shop? Just open a browser on your phone, and surf to your Portainer instance. Easy. Convenient. Powerful.

However…as the old saying goes: Don’t run with scissors.

Portainer is an exceptionally privileged piece of software, and it has near root-level access to your Container infrastructure, so you had better make sure your Portainer instance is safe from malicious actors.

How to secure Portainer (without BrowZer)

Getting Portainer up and running is very straightforward using their standard deployment scripts.

When I decided to kick the tires, I used the following code in a docker compose file:

  portainer:
    image: portainer/portainer-ce:2.21.0
    restart: unless-stopped
    ports:
      - "8000:8000"
      - "8080:9000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./portainer_data:/data

But when you've decided to allow access to Portainer from the public internet, things require more work.

Various writeups have been published by the friendly folks on the Portainer team to provide advice on how to secure Portainer. While not wrong, the instructions are (in my opinion) inflexible and antiquated.

They mention port forwarding the Portainer UI ports (http:9000/https:9443) from a public IP to Portainer.

This implies you must open some ports to the internet. Having ports open to the internet is something we on the OpenZiti project, as well as all zero-trust advocates in general, will advise against.

They strongly recommend network ACLs on your firewall, so you only allow access from known trusted IP addresses.

This is painful and tedious to set up correctly. It is technically out of reach for many. And, it is also incredibly brittle and inflexible (recall my coffee shop reference — this use case would be precluded in this scenario).

They even question if you want to expose it to the internet directly at all, and instead, suggest you set up a VPN.

VPN deployment and configuration is a non-starter for many because of complexity, and inflexibility.

They assert that Portainer's internal authentication system should never be used when presenting Portainer to the internet, but instead you should configure an authentication system to use a suitably secure external mechanism, such as "LDAP" or "OAuth" (the latter of which supports 2FA/MFA).

Again, this is getting to be a heavy lift for many teams.

Understandably, they suggest you force/enable the HTTPS only option, and upload your own x509 certificates.

Many would have given up before this :(

How to secure Portainer (with BrowZer)

A better topology is one where Portainer resides in a VPC, and the host has NO ports open to the internet. In this kind of deployment, Portainer will be completely invisible to the internet.

I hear you thinking: How do I access Portainer over the web if it’s invisible?

The answer is via a zero-trust overlay network that requires authentication before you (or anyone) can connect to it.

How to Easily Deploy a Zero-Trust Overlay Network

The component that implements the zero-trust overlay network solution is OpenZiti.

OpenZiti is a free, open-source zero-trust networking platform that makes services invisible to unauthorized users. Add zero trust to existing applications with tunnelers, or embed SDKs directly for the strongest posture - either way, every connection is authenticated, authorized, and encrypted end to end.

BrowZer

The next component involved in the solution is BrowZer.

The Z in this component's name (within the word normally spelled "browser") is not a typo. It is a purposeful indication that this solution, unique in today's technology offerings for securing browser-based applications, is built as part of the OpenZiti project.

BrowZer enables you and your organization, enterprises, and self-hosting enthusiasts alike, in the cloud or at home, to operate private-to-the-internet web applications while still easily providing secure access for your authorized internet-based remote users.

I previously published a lengthy article that introduced the concept of browZer. I recommend giving it a read.

There is a vast amount of documentation, as well as quick starts for BrowZer, on the OpenZiti site.

There is also some related ZitiTV content:

https://www.youtube.com/watch?v=ZPkOQbVEnW0&t=816s

NOTE: If you like what you read about OpenZiti, or what you saw in the above ZitiTV episode, but prefer not to self-host it yourself, we also offer a zero-trust networking platform. If that interests you, reach out to us for more discussion.

Here is a diagram that describes at a high level how browZer operates:

Example of Accessing Portainer over BrowZer

Here is what a user would experience using browZer to access a private-to-the-internet instance of Portainer:

Here's what happens above:

• Brave web browser hits the public URL representing the protected instance of Portainer. This is where the BrowZer Bootstrapper runs.

• The BrowZer runtime is loaded into Brave by the BrowZer Bootstrapper

• Brave is redirected by BrowZer runtime to Auth0, then federated to Google (with 2FA) for authentication

• Using the OIDC auth token received from Auth0, the BrowZer runtime then (transparently) authenticates onto the Ziti overlay network and then bootstraps the necessary x509 certs into the Brave tab, and the BrowZer runtime then loads the Portainer web app over the mTLS-based zero-trust overlay network from the Portainer server which is invisible to the internet.

• User is presented with Portainer login screen, and user authenticates using Portainer’s internal authentication system (no need to integrate or setup an external "LDAP" or "OAuth" system)

• User clicks on BrowZer button, looks at various BrowZer settings, HTTP throughput chart, BrowZer changelog, BrowZer feedback form, etc

• User is presented with Portainer GUI welcome screen showing that 2 Containers are running (one is Portainer, the other is an instance of the BrowZer Bootstrapper for a staging environment)

• User clicks around, looks at logs for a running container, also removes an old Docker image, and then logs out

The BrowZer Difference

In the above section, we discussed how difficult it can be to properly secure Portainer using “traditional techniques”.

BrowZer uses more modern, game-changing techniques that can not only properly secure your Portainer, but do it more easily, more flexibly, and more thoroughly.

With BrowZer there is:

• no need to expose Portainer ports to the internet or mess around with port forwarding.

• no need to fuss with network ACLs on your firewall, or only allow access from statically configured IP addresses.

• no need to install VPN software on clients.

• remote access from ANY device that has a browser.

• remote access from ANY location on the internet.

• no need to alter Portainer AT ALL (use it off the shelf).

• retained ability to use Portainer’s built-in auth system.

• 2FA/MFA for free (use whatever IdP you want)

• not only is there HTTPS/TLS, but also automatic mTLS, and even end-to-end XChaCha20-Poly1305 encryption before data hits the mTLS wire.

Wrap up

Do you host a web app (like Portainer) and want to be invisible to malicious intruders?

Do you want your users to have easy access from anywhere with no additional software on their client devices?

Do you want to do all this without making any modifications to the web app?

If so, then we hope you'll reach out for a conversation about BrowZer.

OpenZiti has a powerful and flexible authentication system. It allows an OpenZiti overlay network operator to decide what authentication mechanisms any given identity must satisfy before being authenticated to the overlay network. OpenZiti is a zero trust overlay network; authentication is only half of the equation. Once authenticated, identities still require authorization before accessing services secured by the overlay following the zero trust pillar of "least privilege".

Certificate-based Authentication

Perhaps the most common type of authentication, OpenZiti supports authentication to the the OpenZiti overlay network using certificates. It might not be obvious, but a zssh user using an enrolled an OpenZiti identity to authenticate to the overlay network is already implicitly using multi-factor authentication. The first authentication factor is OpenZiti itself. OpenZiti requires connections to be both authenticated and authorized before being allowed to connect to the target service. Once authenticated and authorized, zssh can connect to sshd and attempt to authenticate. Just by using zssh, users are protected with two factors of authentication but zssh (and OpenZiti) offers other factors of authentication as well.

When enrolling an identity, the output of the enrollment flow will be an OpenZiti identity file containing a certificate, key, and CA bundle. This identity can then be used to authenticate connections to the target OpenZiti overlay network. If you are interested in learning how this process works, you can read about it in Andrew's five-part series about bootstrapping trust. Using zssh with an identity file and certificate-based authentication looks something like this (examples are taken directly from the GitHub repo):

zssh \
  -i "${private_key}" \
  -s "${service_name}" \
  -c "${client_identity}.json" \
  "${user_id}@${server_identity}"

In this example, zssh is accepting the ssh key to use to authenticate to sshd (-i), the OpenZiti service to dial (-s), the OpenZiti identity file (-c). Somewhat hidden within this example is the OpenZiti target identity that is binding the service. All zssh connections are made with the expectation that the target identity is provided to the zssh command where one would normal 'host' would be. This allows the OpenZiti overlay network operator to have a single service usable by any identity looking to provide ssh access.

OIDC-based Authentication Only

The zssh executable was recently enhanced to support OIDC-only-based authentication. Using external JWT signers, you can configure an OpenZIti overlay network to trust JWTs from configured IdPs as authentication tokens. This allows the operator to create authentication policies allowing for external, OIDC-based integrations. This is interesting because it allows you to authenticate to the OpenZiti overlay network without needing to enroll the client ahead of time. Instead, zssh users complete an Authorization Code Flow with PKCE. In this scenario, the user will see the familiar flow of a browser window popping up and asking the user to authenticate to an identity provider configured to be trusted by OpenZiti. When the flow completes, the zssh binary will have a JWT that can be used to authenticate to the OpenZiti controller.

zssh \
  -i "${private_key}" \
  -s "${service_name}" \
  -o \
  -a "${oidc_issuer}" \
  -n openziti-client \
  --oidcOnly \
  --controllerUrl https://localhost:1280 \
  "${user_id}@${server_identity}"

In this example, zssh is given the same -i and -s parameters as the certificate-based example above as well as the userid@identity but there are a few others supplied. The -o flag is passed to indicate zssh should obtain a JWT from the specified (-a) OIDC provider. In this example, Keycloak is used as a federated IdP to federate authentication to GitHub. To authenticate to Keycloak, a client id (-n) will need to be provided. This Keycloak client minimally needs to be configured with URLs it's allowed to redirect to after successful authentication. This example uses the --oidcOnly flag, indicating no OpenZiti certificate will be used for authentication. An identity will need to exist in the controller and a matching auth-policy to allow the identity to authenticate. Finally, as no OpenZiti identity is used in this example, zssh must be told where to send the authentication request with the --controllerUrl flag.

Certificate-based + OIDC-based Authentication

OpenZiti can also be configured to support OIDC as a secondary form of authentication. Accordingly, the zssh binary can be configured to use certificate-based authentication as a primary authentication source and an OIDC-based flow for secondary authentication. In the scenario, connecting to the OpenZiti overlay itself would require multiple forms of authentication. These mechanisms are a great way to prove there's both a human and a device. The device provides the certificate, while the human interacts with Keycloak/GitHub's OIDC to verify a human is indeed in the loop.

zssh \
  -i "${private_key}" \
  -s "${service_name}" \
  -o \
  -a "${oidc_issuer}" \
  -n openziti-client \
  -c "${client_identity}.json" \
  "${user_id}@${server_identity}"

In this example, zssh is given the all the same parameters as the OIDC-only example but because it specifies an OpenZiti identity with -c, the --controllerUrl and --oidcOnly flags are not necessary.

TOTP Instead of OIDC

Also added in this release was support for OpenZiti's TOTP-based authentication. Users can now be required to enter their TOTP code before making a connection, allowing for multi-factor authentication to the OpenZiti overlay without using an IdP. Or, if using the OIDC-only based flow with an IdP that doesn't support TOTP (for reasons), OpenZiti's TOTP can be used as a secondary form of authentication to the overlay. For TOTP example usage have a look at the readme on the repository. There, you’ll find the commands shown in the example gif below.

Download zssh/zscp

If you are interested in trying out zssh and it's partner zscp, you can download the latest releases from GitHub at https://github.com/openziti-test-kitchen/zssh/releases/latest, right next to the source code for the project.

Share the Project

If you find this interesting, please consider starring the projects on GitHub. It really does help to support the project! And if you haven't seen it yet, check out https://zrok.io. It's a totally free sharing platform built on OpenZiti and uses the OpenIti SDK! It uses the OpenZiti Go SDK since it's a ziti-native application. It's also all open source too!

Tell us how you're using OpenZiti on X twitter, reddit, or over at our Discourse. Or, if you prefer, check out our content on YouTube if that's more your speed. Regardless of how, we'd love to hear from you.

The OpenZiti project provides SDKs that developers can use to create secure connections. The zssh client demonstrates that adopting an OpenZiti SDK into an application is no harder than developing any application that uses traditional IP-based, underlay connectivity.

Secondly, though uncommon, there are still vulnerabilities found in ssh. Just this year (2024) RegreSSHion was discovered and was given a staggering CVSS score of 9.8. Scores greater than 9 are generally deemed a "patch this as soon as possible" type of CVE. Yes, today's ssh clients are incredibly robust, but if it's easy to remove a substantial portion of attackers from attacking a service by using a zero trust overlay network like OpenZiti, why wouldn't you?

Using *ssh.Client

The go ecosystem provides extended packages from the golang.org/x/* modules. One of these modules is golang.org/x/crypto. Within this module, there is an ssh package that provides everything needed to make a functional ssh client. In there is ssh.Client, the main thing you'll interact with. This struct, along with ssh.NewClientConn and the Golang standard library, provides all the functionality needed to create a simple ssh client.

Shown below is all the code needed to make a very contrived ssh example. Any errors are ignored, and all values are hard-coded or expected as arguments to the program to keep the example small. In total, there are fewer than 30 total lines. Hopefully, the example is straightforward enough to understand.

package main

import (
    "golang.org/x/crypto/ssh"
    "os"
)

func main() {
    key, _ := os.ReadFile(os.Args[1])
    signer, _ := ssh.ParsePrivateKey(key)
    config := &ssh.ClientConfig{
        User:            "ubuntu",
        Auth:            []ssh.AuthMethod{ssh.PublicKeys(signer)},
        HostKeyCallback: ssh.InsecureIgnoreHostKey(),
    }
    sshClient, _ := ssh.Dial("tcp", os.Args[2], config)
    defer sshClient.Close()
    session, _ := sshClient.NewSession()
    defer session.Close()
    session.RequestPty("xterm", 80, 40, ssh.TerminalModes{})
    session.Stdout = os.Stdout
    session.Stderr = os.Stderr
    session.Stdin = os.Stdin
    session.Shell()
    session.Wait()
}

Try it out! That's really all there is to it. You'll be able to ssh to any machine that uses key-based authentication. Although it's not a robust example, it demonstrates the overall idea and shows off how amazing the go ecosystem can be. Note that ssh.InsecureIgnoreHostKey is used for the HostKeyCallback, to keep the example short. See zssh's implementation if interested

Layering in Zero Trust Connectivity

The Golang standard library is well thought out. The abstractions in place make it amazing for building applications embedding zero trust. Adapting an application that uses normal IP-based connectivity (like the ssh example shown above that uses "tcp") to use an OpenZiti SDK is generally straightforward. From the example above, a single line needs to be changed: the line that creates the ssh.Client. This line:

    sshClient, _ := ssh.Dial("tcp", host, config)

Creating the sshClient needs to be adapted away from using IP-based underlay networking. Instead of "tcp" and "remote-machine-name:22", it needs to use a zero trust connection provided by the OpenZiti Golang SDK. Below is a simplified function that uses an OpenZiti identity file to create an OpenZiti context and dial an OpenZiti service, creating a Golang net.Conn that can be used to create an ssh.Client. Again, the example omits error handling and robustness for simplicity's sake, and looks like this:

func obtainZitiConn() net.Conn {
    cfg, _ := ziti.NewConfigFromFile(os.Args[3])
    ctx, _ := ziti.NewContext(cfg)
    dialOptions := &ziti.DialOptions{
        Identity:       host,
    }
    c, _ := ctx.DialWithOptions("zsshSvc", dialOptions)
    return c
}

With the IP-based underlay net.Conn connection replaced with a zero trust connection, an ssh.Client can be created by replacing the call to ssh.Dial, and instead using a call to ssh.NewClientConn combined with a call to ssh.NewClient. With the ssh.Dial line adapted, it looks like this:

    //adapted sshClient, _ := ssh.Dial("tcp", host, config)
    c, chans, reqs, _ := ssh.NewClientConn(obtainZitiConn(), "", config)
    sshClient := ssh.NewClient(c, chans, reqs)

Everything else in the example remains identical; these are the only lines that need to change! The full source for zssh is available on GitHub at https://github.com/openziti-test-kitchen/zssh. You'll find the examples shown above in that repo as individual, compilable go files available in the example folder.

If you're interested in zssh, the OpenZiti project and zero trust in general, check out the next article. It focuses on using zssh and using OpenZiti's OIDC-based authentication mechanisms and uses Keycloak for federated authentication to GitHub or Google.

Share the Project

If you find this interesting, please consider starring the projects on GitHub. It really does help to support the project! And if you haven't seen it yet, check out https://zrok.io. It's a totally free sharing platform built on OpenZiti and uses the OpenZiti Golang SDK and is also all open source!

Tell us how you're using OpenZiti on X twitter, reddit, or over at our Discourse. Or, if you prefer, check out our content on YouTube if that's more your speed. Regardless of how, we'd love to hear from you.

If you're involved with software creation or deployment, you likely use Docker. If so, the following saga probably rings some bells with you:

A common, painful scenario

You become aware that something's not working in your Docker fleet. Maybe a service is down? You ssh to the the host where the containers run, and do a docker compose ps. Yep, it's that buggy microservice that has crashed.

No problem, I'll restart it: docker compose restart. Okay now let's try again. Hmm... The issue is still there. docker compose ps again. Sigh... the service must have just crashed immediately after starting.

I probably would have known what was happening had I been reading the log stream, but there is a lot of clutter from other services. I could get the logs for just that one service via docker compose logs --follow myservice but that dies every time the service dies so I'd need to run that command every time I restart the service.

I could alternatively run docker compose up myservice and in that terminal window if the service is down I could just up it again, but now I've got one service hogging a terminal window even after I no longer care about its logs.

I guess when I want to reclaim the terminal real estate I can ctrl+P,Q, but... wait, that's not working for some reason. Should I use ctrl+C instead? I can't remember if that closes the foreground process or kills the actual service...

OK. You get the picture. Ugh... What a headache!

Of course, it makes no sense to manage a large scale Docker deployment by connecting to each container individually or restarting processes (an orchestrator like K8S will serve that large scale use case much better). But if you've got a few containers running in a self-hosted (home?) network, the above story probably sounds familiar.

A Better way

Memorizing docker commands is hard. Memorizing alias's isn't much easier. Keeping track of your containers across multiple terminal windows is untenable.

But hang on. What if you had all the information you needed in a single magical terminal window where every common docker command was one keypress away?

Picture this:

If that got your attention, Now, imagine if you could access the magical terminal window by simply opening a web browser, on any laptop or mobile device, and then navigating to a URL representing the remote host where your containers run. Everything is right there in the browser tab.

Bye-bye ssh.

Intrigued?

As nice as that is, It gets even better. Now, imagine the URL was internet-accessible, but only visible to you, invisible to others, thus protecting your containers from malicious actors.

This is not a fever dream. In this article, I'll show you how all this is possible today.

I'll discuss the following components that collectively implement the solution:

• Isaiah

• OpenZiti

• BrowZer

The Isaiah Service

One component involved in the solution is Isaiah. Isaiah is a relatively new open-source project (it first appeared in early 2024). It is a self-hostable service that enables you to manage all your Docker resources on a remote server. It is an attempt at recreating the lazydocker command-line application while making everything available as a web application.

The screencap in the above "better way" section shows Isaiah in action.

Isaiah Deployment

You run Isaiah on the host where your containers execute -- the host you would previously ssh to in the 'painful scenario' described at the top of this article.

(NOTE: Ensure that Docker 23.0.0+ is installed on your host before proceeding)

There are several ways to deploy Isaiah, but perhaps the easiest is via docker compose. Here is a compose.yml file you can use:

services:
  isaiah:
    image: mosswill/isaiah:latest
    restart: unless-stopped
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      SERVER_PORT: "80"
      AUTHENTICATION_SECRET: "some-very-long-and-mysterious-secret"

Then simply run docker compose up -d

(NOTE: if you already have a compose file on your host, you can simply add the above isaiah section to the existing services section of your compose file)

As you can see, this compose file will have Isaiah listening on HTTP, without TLS, on your LAN or perhaps even on the open internet. This is certainly sub-optimal from a security standpoint, but read on and I'll describe how to lock things down.

If the AUTHENTICATION_SECRET env var is configured, it tells Isaiah to require visitors to enter the secret when they arrive, like this:

But even with the AUTHENTICATION_SECRET in place, and HTTP visitors hitting your host being prompted as shown above...this is still not as secure as we need it to be.

A better topology is one where your Docker host, and Isaiah, reside in a VPC, and the host has NO ports open to the internet. In this kind of deployment, the host will be completely invisible to the internet. The best (most secure) way to access the host is via a zero-trust overlay network.

How to Easily Deploy a Zero-Trust Overlay Network

The next component involved in the solution is OpenZiti. OpenZiti is a free, open-source zero-trust networking platform that makes services invisible to unauthorized users. Add zero trust to existing applications with tunnelers, or embed SDKs directly for the strongest posture - either way, every connection is authenticated, authorized, and encrypted end to end.

There is a vast amount of documentation, as well as quick starts, on the OpenZiti site, so I will not replicate it here.

If you like what you read about OpenZiti, but you prefer not to self-host it yourself, we also offer a zero trust networking platform. If that interests you, reach out to us for more discussion.

BrowZer

The next component involved in the solution is BrowZer.

The Z in this component's name (within the word normally spelled "browser") is not a typo. It is a purposeful indication that this solution, unique in today's technology offerings for securing browser-based applications, is built as part of the OpenZiti project.

BrowZer enables you and your organization, enterprises and self-hosting enthusiasts alike, in the cloud or at home, to operate private-to-the-internet web applications while still easily providing secure access for your authorized internet-based remote users.

I previously published a lengthy article that introduced the concept of browZer. I recommend giving it a read.

Forward Proxy Authentication / Trusted SSO

With OpenZiti and BrowZer now in your mind, I want to return to Isaiah for a moment.

Isaiah has a feature known as Forward Proxy Authentication. This feature enables you to log in to an authentication portal, and then connect to Isaiah without having to type your AUTHENTICATION_SECRET every time.

In this mode, you protect Isaiah using your authentication portal rather than a cleartext / hashed password.

To use this mechanism, configure Isaiah using the following variables:

• Set FORWARD_PROXY_AUTHENTICATION_ENABLED to true.

• Set FORWARD_PROXY_AUTHENTICATION_HEADER_KEY to the name of the forwarded authentication header your auth proxy sends to Isaiah.

• Set FORWARD_PROXY_AUTHENTICATION_HEADER_VALUE to the value of the header that Isaiah should expect (or use * if all values are accepted).

By the way, by default, Isaiah is configured to work with Authelia out of the box. So if you are using the Authelia IdP as your auth proxy, you can just set FORWARD_PROXY_AUTHENTICATION_ENABLED to true and be done with it (no need to configure the other variables).

Trusted SSO Between BrowZer and Isaiah

Much like Authelia, BrowZer also works with Isaiah's Forward Proxy Authentication out of the box.

As you read in the browZer article linked above, browZer requires you to authenticate with an IdP before connecting to your zero-trust overlay network. A common setup is to use Auth0 as the browZer IdP and have Auth0 federate to Google. You can read about how to do this in our browZer IdP documentation.

Once you have set up your Ziti overlay network, and have set up browZer to enable web access to Isaiah, you are then using browZer as your auth proxy.

Now you can configure Isaiah with:

• FORWARD_PROXY_AUTHENTICATION_ENABLED to true, and

• FORWARD_PROXY_AUTHENTICATION_HEADER_VALUE to the value of your Google email address.

For example:

services:
  isaiah:
    image: mosswill/isaiah:latest
    restart: unless-stopped
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      SERVER_PORT: "80"
      FORWARD_PROXY_AUTHENTICATION_ENABLED: "true"
      FORWARD_PROXY_AUTHENTICATION_HEADER_VALUE: "you@gmail.com"

Here is what it looks like to use browZer to access a private-to-the-internet instance of Isaiah:

Here's what happens above:

• Brave web browser hits the URL representing the protected instance of Isaiah

• Brave is redirected by BrowZer to Auth0, then federated to Google (with 2FA) for authentication

• BrowZer bootstraps the necessary OpenZiti software into Brave tab, and the OpenZiti software loads the Isaiah web app over the zero-trust overlay network

• Isaiah sees that BrowZer has done the proper SSO by passing necessary info from Google to Isaiah, so no login prompt is rendered by Isaiah

• User is presented with Isaiah GUI welcome screen showing that "2 Containers" are running (one is Isaiah, the other is an instance of the BrowZer Bootstrapper for a staging environment)

• User clicks around, looks at logs for a running container, also removes an old Docker image

Wrap up

Do you host a web app (like Isaiah) and want to be invisible to malicious intruders?

Do you want your users to have easy access from anywhere with no additional software on their client devices?

Do you want to do all this without making any modifications to the web app?

If so, then we hope you'll reach out for a conversation about BrowZer.

Over the last weeks, the team has worked together to figure out the best compromise to continue offering our users the best free sharing solution possible, while also offering a reasonable level of protection against abuse. We've batted around a number of different ideas.

The solution we've settled on is to introduce an "interstitial page" for free-tier accounts using public sharing. The interstitial page will be shown the first time an internet user visits a zrok public share (with a timer resetting every week). The page is there to make sure that the user visiting your share is very clear that the web resource they're visiting is shared through zrok, and very likely not a financial institution (or at least that they should be careful with personal information).

The interstitial page is designed to only be displayed to interactive clients (web browsers presenting with a User-Agent header starting with Mozilla/5.0). Other clients like curl, or HTTP clients from various programming langues will bypass the interstitial page.

zrok users who upgrade to any paid tier (see pricing) will not have an interstitial page presented on their shares. The interstitial pages feature does not impact private zrok sharing in any way. When you zrok access private a share and create a private frontend, that frontend does not present interstitial pages. In those cases, we're pretty confident the user understands what they're accessing.

We're also actively working on bringing some additional capabilities to the zrok paid offerings, including "bring your own domain" support, which will allow users to create public shares using their own domain name. So, we're kind of hoping you might want to consider one of those options outside of removing the interstitial page.

If you're a self-hoster, you'll have extensive options to customize the interstitial page configuration. Each public frontend can enable or disable interstitial support, and interstitial pages can be disabled on a per-account basis.

I recently did a short zrok Office Hours video about the interstitial pages feature:

We appreciate every zrok user. Thank you for your support and attention. If you appreciate what we're doing with zrok, it always means a lot when you can drop a star on the zrok repo on GitHub.

If you have any questions or concerns, feel free to reach out on the OpenZiti discourse using the zrok topic.

curl -sSLf https://get.openziti.io/install.bash | sudo bash -s zrok-share

Edit /opt/openziti/etc/zrok/zrok-share.env.

# env
ZROK_ENABLE_TOKEN="w9pEuX3Gb750"  # account token from the console
ZROK_ENVIRONMENT_NAME="my zrok vpn service on hostname"

# share
ZROK_UNIQUE_NAME="mysecretsharetoken"
ZROK_BACKEND_MODE="vpn"
ZROK_TARGET="172.21.71.1/24"  # allocate ip in some private subnet
ZROK_FRONTEND_MODE="reserved-private"

You can control access to your VPN by keeping ZROK_UNIQUE_NAME a secret (the share token) or adding the following to restrict zrok accounts by email. Only your account can use the share in closed mode if you don't add grants.

ZROK_PERMISSION_MODE="closed"
ZROK_ACCESS_GRANTS="alice@example.com bob@example.org"

Grant kernel capability NET_ADMIN to the service.

sudo sed -Ei 's/.*AmbientCapabilities=CAP_NET_ADMIN/AmbientCapabilities=CAP_NET_ADMIN/' /etc/systemd/system/zrok-share.service.d/override.conf
sudo systemctl daemon-reload

Start the service now and auto-start after a reboot.

sudo systemctl enable --now zrok-share.service

Check the logs.

sudo journalctl -lfu zrok-share.service

The logs should look like this, confirming this device is allocated 172.21.71.1 on the VPN.

Aug 02 16:33:08 ubuntu zrok-share.bash[6243]: {"level":"info","ts":1722616388.2556849,"msg":"interface created tun0"}
Aug 02 16:33:08 ubuntu zrok-share.bash[6243]: {"level":"info","ts":1722616388.255752,"msg":"exec /sbin/ip [link set dev tun0 mtu 16384]"}
Aug 02 16:33:08 ubuntu zrok-share.bash[6243]: {"level":"info","ts":1722616388.2595484,"msg":"exec /sbin/ip [addr add 172.21.71.1/24 dev tun0]"}
Aug 02 16:33:08 ubuntu zrok-share.bash[6243]: {"level":"info","ts":1722616388.2627363,"msg":"exec /sbin/ip [-6 addr add fd00:7a72:6f6b::1/64 dev tun0]"}
Aug 02 16:33:08 ubuntu zrok-share.bash[6243]: {"level":"info","ts":1722616388.2659466,"msg":"exec /sbin/ip [link set dev tun0 up]"}
Aug 02 16:33:08 ubuntu zrok-share.bash[6243]: {"level":"info","ts":1722616388.2711568,"msg":"interface configured tun0"}

Join the VPN from Another Device

The zrok console looks like this for an account with two environments, one per VPN peer, and one VPN share.

To temporarily join the zrok VPN you only need to run this command.

sudo -E zrok access private mysecretsharetoken

You will see zrok's terminal user interface (TUI) telling you what is happening. This second device will be allocated 172.21.71.2 and subsequent devices will get unique IP allocations when they join. This is a network-layer VPN. You can send ICMP, TCP, UDP, etc.

Now, this second device is joined to the VPN, so it has a dashed line to the VPN share indicating a zrok "private access."

Share the Project

If you find this interesting, please give zrok a star on GitHub!

Let us know if you found a good use for this or have an improvement or question in mind on X twitter, in /r/openziti, or the Discourse forum. We upload and stream on YouTube too. We'd love to hear from you!

Why zrok?

zrok.io makes it easy to self-host web applications while obscuring your real public IP and protecting your private network. Your visitors will see a trusted certificate with the name *.share.zrok.io provided by zrok.io as a service when they visit your Jitsi Meet instance's public URL.

Jitsi Meet requires a trusted certificate for WebRTC, so zrok.io also spares you the chore of issuing, renewing, and configuring a TLS server certificate for Jitsi.

Orientation

You only need Docker to follow this tutorial. There's nothing else to install. You can use this guide with a free account from zrok.io or a self-hosted zrok instance.

Jitsi Meet's Docker Compose project is pre-configured to publish the container ports to the Docker Host's external interfaces. Typically, that is necessary so they are reachable by clients.

zrok works differently and does not need the ports to be published externally. Instead, zrok runs inside the Compose project on the meet.jitsi bridge network. zrok will proxy the traffic securely to the containers' internal ports, so we'll override the forwarded, published ports, exposing them only inside the Compose project.

The Steps

1. Follow Jitsi's Docker Quickstart and wait to run docker compose up.

2. In your terminal, change to the directory where you have created these Jitsi quickstart files: .env and docker-compose.yml.

3. Download the zrok public share compose example and save it as the filename docker-compose.zrok.yml in the same directory.

4. Add the following YAML as the filename docker-compose.override.yml in the same directory.

    services:
      web:
        ports: !override []
      jicofo:
        ports: !override []
      jvb:
        ports: !override []
      zrok-share:
        networks:
          meet.jitsi:
   

5. Think of a name for your self-hosted Jitsi Meet instance. You will use it in the next step to define the unique name of the zrok share which is part of the public URL. The name must be 4-32 lowercase letters or numbers.

6. Save the following variable assignments as the filename .env.zrok in the same directory.

    PUBLIC_URL="https://myjitsi.share.zrok.io"  # subdomain must match ZROK_UNIQUE_NAME
    ZROK_UNIQUE_NAME="myjitsi"                  # must match PUBLIC_URL subdomain
    ZROK_ENABLE_TOKEN="ix9XrvQt13Rf"            # zrok account token from console
    ZROK_ENVIRONMENT_NAME="jitsi-zrok-compose"  # name for the environment in the console graph
    ZROK_API_ENDPOINT="https://api.zrok.io"     # must be set to the zrok API you're using
    ZROK_TARGET="https://web:443"               # this is correct for the web container's internal port
    ZROK_INSECURE="--insecure"                  # let zrok skip cert verification for the internal web:443 target
   

7. Optionally, turn on OAuth for this Jitsi Meet instance with zrok. Add the following to the .env.zrok file (Docker public share guide has more info).

    ZROK_OAUTH_PROVIDER="google"  # google, github
    # space-separated list email patterns verified by the provider
    ZROK_OAUTH_EMAILS="alice.example@gmail.com *@acme.example.com"
   

8. Save the following script as the filename compose.bash in the same directory. This script configures the compose project and environment files.

   
    export COMPOSE_FILE="docker-compose.yml:docker-compose.zrok.yml:docker-compose.override.yml"
    export COMPOSE_ENV_FILES=".env,.env.zrok"
   
    docker compose  "${@}"
   

9. Ensure you have all the necessary files.

   1. docker-compose.yml

   2. docker-compose.zrok.yml

   3. docker-compose.override.yml

   4. .env

   5. .env.zrok

   6. compose.bash

10. Run Jitsi and zrok.

    bash ./compose.bash up
    

11. Open Jitsi in a web browser at the address from the PUBLIC_URL environment variable, e.g., https://myjitsi.share.zrok.io .

12. If you need to change the name, authentication, etc. you can delete the environment in the zrok web console and delete the Docker volumes like this to start over. It's also possible to make surgical changes if you don't want to start over. Ask for help in Discourse.

    bash ./compose.bash down --volumes
    

zrok frontdoor

This tutorial for Jitsi Meet is a great example of zrok frontdoor. zrok frontdoor brings many advantages for self-hosters and is always enabled when using zrok.io as a service with a production-ready service like this zrok public share in Docker. zrok.io users enjoy additional shielding for their Jitsi Meet public URL.

Relatedly

zrok is built with OpenZiti. Here's another post about running an Asterisk PBX without published ports, just as your Jitsi Meet instance has no open ports on the Docker Host's outward-facing interfaces.

Share the Project

If you find this interesting, please consider starring us on GitHub. It helps. Let us know if you found a good use for this or have an improvement or question in mind on X twitter, in /r/openziti, or the Discourse forum. We upload and stream on YouTube too. We'd love to hear from you!