Intro
Monitoring is very important in today's technology world. It allows system administrators to
detect early signs of failures and security breaches, and address them before they cause outages
improve resource utilization
reduce time spent on active monitoring and instead be alert-driven
Monitoring systems are usually deployed alongside the main system or software that is being monitored. What this means is that monitoring systems must have the same security requirements as any other production system. If your monitoring system is vulnerable a sophisticated attacker can launch a multi-prong attack on your whole infrastructure. Disabling or impairing monitoring can lead to a delayed discovery of incidents or missing them altogether.
This is a typical monitoring system at a very high level. The link from Agent to Collector is likely to go over the public Internet, which means that Collector endpoints are vulnerable to attacks by malevolent actors. These attacks can range from DDOS to targeting vulnerabilities in Collector software (e.g. Log4Shell). When the collector becomes compromised or impaired, your Analytics engine cannot trigger appropriate alerts. While these attack vectors can be mitigated with VPNs, firewalls with port-forwarding/ACL/IP-whitelists, etc., those solutions are non-trivial to implement and maintain.
OpenZiti is offering a solution that avoids deploying additional software (e.g VPN agents) or maintaining other infrastructure (e.g. firewalls). OpenZiti SDKs allow building zero trust networking right into your applications and leverage full mesh, zero trust overlay, policy controlled access without separate agents.
The rest of this article will walk through connecting modified popular agent and collector software over OpenZiti overlay network. It makes Collector inaccessible from the public Internet and makes it available only to Agents with proper permissions.
Implementation Overview
For this exercise, we picked Elastic Logstash as the collector and Elastic Beats tools as the agents. It was done for two reasons:
The Beats/LogStash combination is a very popular monitoring solution
Our own Ops team (at NetFoundry) is using them for monitoring data collection and is incorporating Zero Trust into our internal tools
Beats
Elastic Beats tools are written in Golang. We use OpenZiti Golang SDK to embed Zero Trust networking right into the executables.
To get zitified Beats binaries, clone our fork and build them:
$ git clone --branch=zitify https://github.com/openziti-test-kitchen/beats.git
$ cd beats
$ mkdir build
$ go build -o build/ ./filebeat
These binaries can be dropped into the existing beats install and configured to use OpenZiti overlay network.
Beats implementation details
The fork above just pulls the zitified version of the Elastic agent support library with the following replace
github.com/elastic/elastic-agent-libs => github.com/openziti-test-kitchen/elastic-agent-libs v0.0.0-20221118205208-c84fcc069fb2
In the modified library the only code change (aside from added dependency in go.mod) is this:
The Ziti dialer takes care of matching the target address to the service with a matching intercept.
LogStash
On the LogStash side, we created a zitified version of the LogStash Beats input plugin. LogStash is built with JRuby/Java. We use OpenZiti JVM SDK to embed Zero Trust networking into the Beats plugin.
To build and install zitified plugin (make sure you have all the required tools for LogStash plugin development):
$ git clone --branch=zitify https://github.com/openziti-test-kitchen/logstash-input-zitibeats.git
$ cd logstash-input-zitibeats
$ ./gradlew vendor
$ gem build ./logstash-input-zitibeats.spec
$ ${LS_HOME}/bin/logstash-plugin install ./logstash-input-zitibeats-6.4.1-java.gem
We implemented a Beats input endpoint server by using org.openziti:ziti-netty library.
ServerBootstrap server = new ServerBootstrap();
server.group(workGroup)
.channelFactory(new ZitiServerChannelFactory(ztx))
.childHandler(getBeatsInitializer());
Service service = ztx.getService(serviceName, 10000L);
Channel channel = server.bind(new ZitiAddress.Bind(service.getName())).sync().channel();
It fits nicely with the original plugin and reuses the rest of the original implementation.
Configuring OpenZiti
To tie it all together, the OpenZiti network has to be configured with appropriate identities, services, and policies.
The diagram below shows how ziti configuration is mapped/used in the LogStash/Beats deployments.
Running Zitified LogStash
Once our new input is configured LogStash can be started. This is a sample output you'd see in the LogStash log.
[2022-12-27T12:33:56,061][INFO ][org.openziti.impl.ZitiImpl][main] ZitiSDK version 0.23.18 @23cb6b8()
[2022-12-27T12:33:56,332][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-12-27T12:33:56,398][INFO ][org.openziti.logstash.beats.Server][main][a0be65cb7a03f4efc1785fd4746bd82f98469e6bef41caf7afeb4b8ac3763006] Starting server for service: beats.logstash
[2022-12-27T12:33:56,406][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-12-27T12:33:56,714][INFO ][org.openziti.api.Controller][main] controller[https://fec44561-7d3b-41e7-a26f-e9f851a9ec33.production.netfoundry.io/] version(v0.26.11/807dd591b1f5)
It shows that Ziti SDK is loaded and the plugin is waiting for connections on the configured service.
Running Zitified Beats agents
To connect zitified Beats agents over Ziti overlay network you need to do two things:
point output.logstash to the service intercept address
output.logstash:
hosts: ["beats.logstash.ziti:5044"]
reference enrolled agent identity file in ZITI_IDENTITIES environment variable
$ export ZITI_IDENTITIES=beatz.json
$ ./metricbeat -c metricbeat.yml
Conclusion
Using Zero Trust solutions is important not only in customer-facing production systems but in internal ones as well. Systems monitoring falls under that umbrella.
OpenZiti SDKs make it easy to embed Zero Trust networking right into your applications provided you can modify the source code. OpenZiti SDKs are designed to match target languages/frameworks styles of programming and in most cases can be embedded with just a few lines of code.
See all of them on GitHub.
Want to learn more or provide feedback? Head to our docs and discussion forum, and follow this very blog.
