Apache SkyWalking – Cloud Native

Blog: Meet Horizon UI · 7/16: The Log Explorer

Tue, 23 Jun 2026 00:00:00 +0000

This is the seventh post in the Meet Horizon UI series. Part 6 was one request’s spans; this one is the log lines around it. Horizon surfaces logs through two distinct tabs, because there are really two different questions: “what did this service log over the last half hour?” and “what is this pod printing to stdout right now?”

The Logs tab queries the logs SkyWalking has already collected and stored — indexed, filterable, correlated with traces.
The Pod Logs tab live-tails a Kubernetes pod’s container logs on demand — these aren’t stored logs at all: OAP reads them straight from the Kubernetes API server (the kubectl logs path), Horizon shows the window, and it’s discarded. Nothing is persisted, and SkyWalking’s log storage is never involved.

Which tabs a layer shows is up to its template: the Logs tab appears on layers that enable it (General, Mesh, Nginx, the Envoy AI Gateway, the mobile and mini-program layers); the Pod Logs tab appears only on the Kubernetes-aware layers (Kubernetes Service, Mesh, Mesh data plane). Browser JavaScript errors are a different thing again — not service logs but client-side error events the browser agent reports, with their own categories and their own source-map de-obfuscation (turning a minified app.min.js:1:… frame back into your original file:line). That’s a separate tab on the Browser layer, and its own post later in the series.

The stored log stream

Open a layer that has a Logs tab, pick a service in the header, and its stored log stream loads newest-first. Like the trace explorer, this tab owns its own time range — the global topbar picker is paused while you’re here, so auto-refresh can’t shift the window out from under an investigation. Pick a rolling preset (last 15 minutes through 24 hours, default 30) or a custom absolute window; queries run at second precision so the most recent lines are never rounded off the minute.

A conditions bar narrows the stream, and every filter is optional and AND-joined:

Instance — restrict to one service instance (labelled Sidecar on a sidecar layer).
Endpoint — type to search the service’s endpoints, click to pin, × to clear.
Trace ID — show only the lines correlated with one trace. This is also how a log lands when you arrive from a trace: the field pre-fills and the stream is already scoped.
Tags — a single key=value field with autocomplete; start a key to see suggestions, type = to switch to known values, Enter to commit. Committed tags ride along as removable chips.
Level — the Levels strip above the stream doubles as a filter: click error, warn, info, or debug to keep only that level, click again to clear.

There’s no log query language here — no LogQL box to learn. The conditions above are the whole surface, and edits refresh the stream as you make them; Run query is just the explicit “I’m done editing, refresh now” button that resets to the first page.

Reading the stream

The point of a log view isn’t to list lines, it’s to find the shape in them — so the stream comes with two pieces of orientation above it.

A density histogram plots log count over time, each bar stacked by level in the legend’s colors; hover a bar for that bucket’s time range and per-level counts. It’s drawn from the page currently on screen, so it shows the shape of what you’re looking at. And the Levels strip carries a running count per level — sampled across the window, not just the visible page, so the error/warn/info mix reflects the whole window you’re querying.

Each row then shows the timestamp, the level (the row is color-keyed to it), the service, an ↗ trace link when the line is trace-correlated, a JSON / YAML / TEXT format chip, and a one-line preview of the content. Horizon decides that chip by what the payload actually is: OAP labels a body JSON or plain text, and on top of that Horizon sniffs for JSON and YAML structure, so an unlabelled-but-structured line still gets the right treatment — JSON flattened to one line in the preview, YAML keeping its keys, plain text whitespace-collapsed.

Figure 1: A service’s stored log stream — a level histogram and level counts over the window, then the rows, each tagged JSON / YAML / TEXT and linked to its trace.

Into a single line

Click a row and the full payload opens in a popout: the complete content with format-aware pretty-printing — JSON and YAML laid out properly, plain text given the whole canvas instead of a cramped strip — plus a Copy button, the service / instance / endpoint / trace context, and a table of every tag on the line. When the line is trace-correlated, an ↗ trace button opens the related trace’s waterfall in an overlay without leaving the stream — and it passes the row’s timestamp along, so the trace is found even when it has aged into a colder storage tier. Escape or a backdrop click closes it.

Figure 2: One line in full — its payload pretty-printed by format, with its context and every tag laid out beside it.

Pod Logs: tailing what’s printing right now

The Pod Logs tab answers the other question, and it’s a fundamentally different source: not SkyWalking’s stored logs, but the pod’s container output read live from the Kubernetes API server through OAP — the exact thing kubectl logs -f reads. There’s no stored history to page through; each refresh pulls the trailing window, shows it, and throws it away.

Starting a tail is a few picks: choose a Pod (one service instance, pinned), a Container (Horizon lists the pod’s containers and selects the first), a look-back Window (last 30s, 1m, 5m, 15m, or 30m — how far back each poll reaches), and a poll Interval (2s, 5s, 10s, or 30s — how often it re-fetches). Press Start and the window streams into a read-only viewer that keeps the newest line in view and re-polls until you Pause; a header strip shows the container, the line count, a live indicator, and how long ago it last updated. Two Include / Exclude filter rows narrow what you see — each chip is a full-line regular expression (.*error.*) evaluated by OAP, so they match the whole line rather than a substring, and they stack.

One thing to know going in: on-demand pod logs are disabled by default on OAP, because container output can carry secrets. When the feature is off — or when the pod you picked has been rolled or scaled away — OAP returns a reason, and Horizon shows it in a banner rather than an empty pane, so you can tell “turn this on” apart from “that pod is gone.”

Figure 3: A live tail of one pod’s container — windowed, interval-polled, regex-filterable, never persisted.

Where to go next

Both tabs — the stored queries, the tag and container autocomplete, and the live tail — are gated by a single logs:read permission, so granting “can read logs” is one switch. For the field reference — every condition, the histogram, the Pod Logs windows and filters — see the Logs docs.

Next up: Browser & RUM monitoring — the browser agent’s own error stream, and de-obfuscating a minified stack with source maps.

Blog: Meet Horizon UI · 5/16: The 3D Infrastructure Map

Mon, 22 Jun 2026 00:00:00 +0000

This is the fifth post in the Meet Horizon UI series. Part 3 drew the map between services and Part 4 drew it inside one service. This post zooms all the way out: a single WebGL view of your entire deployment at once — every SkyWalking layer’s services rendered as cubes, stacked in 3D, with live traffic, alarms, and the calls between them. It’s the “stand back and look at everything” companion to the per-layer dashboards.

It’s also genuinely interactive, so rather than describe it cold, here it is — the real map running on the demo’s sample data. Drag to rotate, scroll to zoom, click a cube:

Interactive · sample data Open the 3D map

One scene for the whole estate

The 3D map is a standalone, full-screen view at /3d/map, opened from the 3D Infra pill in the topbar. It deliberately drops the rest of the console — no sidebar, no topbar, no global time picker — so the scene gets the whole viewport. The SkyWalking mark sits bottom-left; the × top-right returns you to Horizon. Everything in it is read live from the same OAP the rest of Horizon talks to: the service roster, each layer’s topology, the per-service traffic, and the active alarms.

Figure 1: The whole deployment in one scene — services as cubes, layers as colored zones, roles as stacked tiers.

Tiers are the spine

A tier is a horizontal plane that groups related layers by their role in the system. Tiers read top-to-bottom the way a request flows — from the apps a user touches down to the platform everything runs on. Horizon ships four bundled tiers:

Apps (top) — application surfaces and the dependencies as the app sees them: General agent services, Browser/RUM, mobile, and the Virtual* targets (database, cache, MQ, gateway, GenAI).
Middleware — the data and messaging services, gateways, and self-observability: MySQL, PostgreSQL, Redis, Kafka, RocketMQ, APISIX, Nginx, the SkyWalking SO11Y components, and cloud-managed data services.
Service Mesh — the mesh that fronts the apps: Istio control/data plane, Cilium, the Envoy AI Gateway.
Infra (bottom) — the platform the rest runs on: Kubernetes, hosts, VMs.

Every layer OAP reports lands on exactly one tier. A layer Horizon hasn’t classified yet — a brand-new OAP layer, say — falls to the failover tier (Middleware by default) with an “unclassified” mark, so it shows up and an operator can re-assign it rather than silently dropping off the map. The panel on the right mirrors the stack: click a tier row to fly the camera to it, use the eye toggle to show or hide a whole tier (or a single layer) at once, and read off how many of its services are currently visible.

Reading the map: cubes, zones, traffic

Each cube is one service. Cubes are grouped into their layer’s zone on the tier — a translucent rectangle painted in the layer’s brand color and stamped with the project’s logo (Istio’s sail, the Kubernetes helm, a database cylinder, a queue) so you can pick a zone out from any camera angle. Layers that ship a topology (General, Service Mesh, Kubernetes Service, Cilium) lay their cubes out by call dependency — upstream callers on one side, downstream services on the other, the 3D analogue of the 2D service map. Layers without a topology pack their cubes into a tidy grid.

A small traffic pill under a cube shows that service’s live headline throughput — requests per minute for app and mesh services, queries or operations per second for data services, each with its own unit. The pills appear on cubes close enough to read and fade away as you zoom out to keep the scene clean, then return as you come back in; a selected cube always shows its number.

Alarms, and Beacon mode for incidents

When a service has a currently-firing alarm (Horizon polls the last 20 minutes and counts only service-scoped, still-firing ones), its cube pulses red — a beacon you can spot from clear across the scene, while the alarm feed refreshes on its own.

On a busy map, one more red cube among hundreds can still be hard to find — so there’s Beacon mode. Toggle it from the toolbar and every healthy cube dims to a dark wireframe ghost, leaving only the alarming services lit and glowing. The shape of the deployment stays legible, but the services that are actually on fire are the only thing with color. It turns the bird’s-eye view into an incident triage surface in one click.

Figure 2: Beacon mode dims everything healthy to a ghost, so the firing services are the only thing you see.

The lines between things

The map draws the call graph, not just the nodes:

In-layer calls — light cyan tubes between two services in the same layer, with packets animating along them. This is each layer’s internal call graph, always on.
Cross-layer calls — soft amber arrows between services in different layers on the same tier (a Browser app calling a Frontend, a Frontend calling a Virtual Database), pointing from caller to callee.
Hierarchy links — and here’s the one that makes the 3D layout earn its keep. Select a cube and thicker gray tubes connect the different faces of the same logical service across tiers — the service as its agent sees it, as the mesh sees it, as a Kubernetes service. These represent identity, not traffic, so they stay hidden until you select a cube and then show just that cube’s relatives, climbing the stack from tier to tier. It’s the Smartscape idea from Part 3, drawn in the dimension it was always meant for.

Figure 3: Select a cube and its identity links climb the tiers — one service, seen by its agent, the mesh, and Kubernetes.

Moving around

Drag to rotate, scroll to zoom, and arrow keys or WASD pan the view (hold Shift for a bigger step); a top-left toolbar offers the same gestures as buttons for trackpads. There’s one deliberate rule worth knowing: clicks inside the 3D scene never move the camera — they only select. Click a cube and it highlights, a detail card appears beside it (the service’s name, its tier and layer, and an Open dashboard button that jumps into that service’s layer dashboard in a new tab), and its hierarchy links light up. The camera-move surface is the side panel and the toolbar — click a layer row to glide the camera to its zone. Keeping those two jobs separate is what makes selecting a small cube feel reliable instead of having the cube slide out from under your cursor.

How it builds

A whole deployment is too much to fetch in one request, so the map loads in stages, and a slim timeline strip along the bottom shows the progress live: Services (the roster and their layers) → Templates (which layers carry a topology) → Topologies (each topology-bearing layer’s call graph) → Hierarchy (the cross-tier identity links) → Layout (placing the cubes) → Metrics (the per-service traffic, fetched in batches so the cubes light up progressively). Click any step for a drawer with its detail, or hit refresh to re-run the whole sequence.

Two touches make refreshes cheap. The hierarchy step is incremental — only services that are new since the last run get probed, the rest reused from cache, so a steady deployment costs nothing there. And the scene is re-keyed on a per-layer structure hash, so an unchanged refresh keeps your camera exactly where it was; only a real roster or edge change rebuilds the layout. Under the hood it’s Three.js with a thin Vue wrapper, every geometry and material shared across cubes of the same kind — the kind of detail that keeps a few hundred services rendering smoothly in a browser tab.

Configured, not coded

None of the above is a hard-coded “3D screen.” What the map shows is driven by a single configuration an administrator edits in a structured form editor at /admin/3d-map — tiers, layers, colors, and metrics through form controls, not raw JSON. From it you can:

Filter layers with one global regex — anything it excludes drops off the map entirely.
Arrange tiers — rename them, reorder them top-to-bottom, and pin each layer to a tier (with a nominated failover tier so nothing falls off silently).
Group layers — cluster several related layers (the SkyWalking self-observability components, say) into one labelled block, each member keeping its own color.
Color each layer and choose its traffic metric — the MQE expression, a label, and a unit, seeded by default from that layer’s dashboard template so most layers show a sensible number out of the box.

Horizon ships a bundled default, so the map is useful immediately; your edits live as a local draft until you Check diff & push them to OAP — the same draft → preview → publish model behind every dashboard, and the same Export/Import for backup or moving a configuration between deployments. The map itself is a read-only observe surface that runs against your current OAP; publishing the config that shapes it is part of the config-driven customization story a later post in this series covers end to end.

Figure 4: The map is configuration, not code — tiers, colors, and per-layer traffic metrics edited as a form, then published to OAP.

Where to go next

The 3D map is the bird’s-eye summary; the 2D per-layer pages stay the authoritative service maps. Viewing it needs only read access (infra-3d:read, held by the built-in viewer role and up); shaping it needs the same write permission as the dashboards. For the field reference — tiers, the config shape, the loading stages — see the 3D Infrastructure Map docs.

Next up: the Trace Explorer — from the bird’s-eye view of the whole deployment back down to a single request, drawn three different ways.

Blog: Meet Horizon UI · 6/16: The Trace Explorer

Mon, 22 Jun 2026 00:00:00 +0000

This is the sixth post in the Meet Horizon UI series. The last few were maps — topology between services, the deployment inside one, the 3D view of the whole estate. Those answer “what does my system look like.” This post is about the opposite move: from an aggregate down to one request — its spans, its timing, the exact hop that went slow. That’s the Traces tab.

Built for triage, not tailing

The Traces tab is a distributed-trace explorer that lives inside a layer: pick a service, set your conditions, and read a single trace’s span timeline. And it behaves differently from the rest of the console on purpose — because traces are triage data, not a live feed.

It owns its own time range and conditions. It does not follow the global topbar time picker, and it does not auto-refresh. You stage what you’re looking for and press Run query; nothing is fetched until you do. Before the first run the list simply says “Pick your conditions, then click Run query.” When you’re chasing one bad trace from twenty minutes ago, the last thing you want is the page sliding forward under you every few seconds — so it doesn’t.

Conditions, not a query language

The whole filter surface is structured form controls — selects, number ranges, and tag chips — staged in a toolbar that only takes effect on Run query:

Instance and Endpoint narrow within the service (the endpoint dropdown lists the service’s own endpoints).
Status — ALL / SUCCESS / ERROR.
Order — Newest (by start time) or Slowest (by duration).
Limit — how many rows to pull (30 by default); the BFF caps the page size server-side so a client can’t ask OAP for the world.
Time range — a rolling preset (last 15 minutes through 24 hours) or a custom absolute window, evaluated at second precision so a trace that just finished still falls inside it instead of being rounded off the minute.
Trace ID — paste one to look it up directly.
Duration range — a min–max in milliseconds.
Tag — free-form span tags as key=value (e.g. http.status_code=500), added with Enter as removable chips and AND-joined; keys and values get typeahead from the backend.

One thing this is not: a query language. There’s no TraceQL box anywhere in Horizon — the structured conditions above are the entire surface. (TraceQL is a separate path: SkyWalking’s backend can serve traces to Grafana over TraceQL, which is its own story. Horizon’s explorer is forms, not a DSL.)

A distribution you can box-select

When the results come back, the toolbar is joined by a Distribution chart: one dot per trace, plotted with start time on the X axis and duration as its height — slower traces sit higher. Dots are colored by status — errors in red, successes in the accent color — so a cluster of red high up is exactly the “slow and failing” corner you came to find.

The chart is also a filter. Click a dot to pick it, or drag a box across a band of them, and the result list narrows to just that selection — the header switches to an “N picked” count with a Reset. This is a client-side filter over what’s already loaded; it never issues a new query. Dragging a band across the slow-and-erroring corner and reading only those rows is the fastest way to go from “200 traces” to “these six.”

Figure 1: Stage conditions, run the query, then box-select a band of the distribution to whittle the list down to the traces worth opening.

Each result row shows the trace’s root endpoint, an OK/ERR flag, the duration, and a bar sized against the slowest trace in the set. What a row represents depends on the storage backend, and Horizon detects it for you: a banner reads either “Full traces are returned inline” (the backend returns whole traces with their spans, so a click opens immediately) or “Each row is a trace segment — click one to fetch its full trace.” You never configure this; the banner just tells you which you’re looking at.

Three ways to read one trace

Click a row and the trace opens with a three-way view toggle — Default, Tree, and Statistics — over the same spans:

Default is the span waterfall: one indented row per span, each carrying a service-colored bar positioned and sized by the span’s start offset and duration on a shared timeline, a span-kind glyph, the component’s icon (the same icon set the topology map uses), the endpoint or peer name, and the span’s own duration. Errored spans are highlighted, and a flag marks any span carrying attached events. Crucially, the waterfall stitches spans across segments using their parent references, so a request that crossed five services renders as one connected timeline rather than five disjoint ones.
Tree draws those same spans as a zoomable, pannable node graph — root on the left, callees flowing right — for when you care about the shape of the call tree more than the exact timing.
Statistics rolls the spans up by name: a sortable table of count and total / average / maximum duration per operation, so “which span am I spending all my time in, across every occurrence in this trace” is one sort away.

Figure 2: The waterfall (Default) — one connected timeline across every service the request touched.

Figure 3: The Tree view — the same spans as the call tree’s shape, zoom and pan to explore.

Inside a span

Click any span and a detail panel opens beside it. Meta lays out the essentials — service, instance, endpoint, kind (entry / exit / local / producer / consumer), component, peer, layer, start time, duration, and the error flag. Below it, when they apply:

Cross-trace refs — when a span’s parent lives in a different trace (an async hop, a message consumed later), the reference is listed with the parent’s trace id, segment, and span — and the trace id is a link that swaps you straight into that other trace.
Tags, Logs (timestamped per-span entries), and Attached Events (named events with start/end times and summary key/values).

The detail header reports the trace’s start, total duration, span count, and how many distinct services it touched; from there you can copy the trace id or a shareable link. Opening a shared ?traceId=… URL lands you directly on that trace in an overlay — which is what makes a trace something you can paste into an incident channel and have a teammate land on the exact same view.

Native and Zipkin, side by side

Not every layer’s traces come from SkyWalking’s own agents. A layer template carries a traces.source setting — native, zipkin, or both — and Horizon routes accordingly. Agent-instrumented layers (like General Service) use the native explorer above; service-mesh and Kubernetes-flavored layers, where spans arrive as Zipkin/OpenTelemetry data, use the Zipkin explorer; and a layer set to both simply gets two sidebar tabs, since native and Zipkin spans have genuinely different shapes and conditions.

The Zipkin tab queries an upstream Zipkin store through OAP’s Zipkin query API (the same compatibility surface OAP exposes for any Zipkin client — not a GraphQL or TraceQL path). Because Zipkin organizes data by its own service universe (the serviceName on each span, which can drift from SkyWalking’s service list), the tab carries its own service controls instead of binding to the page’s service picker — with Zipkin-native conditions like Remote service, Span name, and an Annotations query (error or key=value). The two stores fail independently: if Zipkin is unreachable, the native traces are unaffected, and vice versa.

Figure 4: A layer set to Zipkin gets its own tab and its own service universe, querying the upstream Zipkin store through OAP.

Open a Zipkin span and its detail keeps that Zipkin shape — the CLIENT / SERVER kind, the local and remote endpoints, and the raw Zipkin/OpenTelemetry tags (the istio.* set, http.status_code, the sidecar node_id) exactly as Zipkin recorded them, with no translation into SkyWalking’s span model.

Figure 5: A Zipkin span keeps its native fields — the kind, the endpoints, and the raw istio.* / HTTP tags.

From a slow row to the trace behind it

There’s a second way into a trace that doesn’t go through the explorer at all. A trace overlay is mounted once for the whole app, and several things open it by trace id: a ?traceId= link, a cross-trace ref, a log line, and — as Part 2 showed — the jump-to-trace icon on a record widget’s slow-statement row. Because it resolves by id rather than by layer, it works even when the trace belongs to a different layer than the one you’re looking at: a Virtual Database, Cache, or MQ service has no traces tab of its own, yet its slow statement still lands you on the originating trace. And when the jump carries a timestamp (a log row knows when its line was written), the lookup widens around that moment — so a trace that has already aged into cold storage still resolves instead of quietly missing.

Where to go next

The Traces tab is one request in full detail; the dashboards and maps from the earlier posts are where you notice something’s wrong in the first place. For the field reference — every condition, the native-vs-Zipkin split, the span detail panel — see the Traces docs.

Next up: the Log Explorer — the same triage instincts, applied to log streams instead of spans.

Blog: Meet Horizon UI · 1/16: SkyWalking's New Observability Console

Sun, 21 Jun 2026 00:00:00 +0000

Apache SkyWalking Horizon UI is the next-generation web console for SkyWalking. It talks to the same OAP backend SkyWalking already runs — the same GraphQL query protocol, the same admin REST surface, the same MQE language, the same Layer concept — so you can point it at a running OAP and log in without changing a thing on the backend. What changes is everything in front of that contract.

This is the first post in a series. Across it we will walk through the dashboards and the metric query language, the topology views and a WebGL 3D map of your whole deployment, the trace and log explorers, profiling, the operations surface, access control, and the config-driven customization that ties it all together. This post sets the stage: what Horizon is, the one idea the whole UI is built around, and how to put it in front of your OAP today.

Horizon is built around four verbs. You observe — topology, traces, logs, all five flavors of profiling, read-only alarms, and per-layer dashboards. Then you operate what you observe, govern who is allowed to touch it, and customize the whole console without writing UI code. Observe, operate, govern, customize: that is the arc of this series. We start where every session starts — the sidebar.

Open Horizon and the left sidebar is not a hand-built menu — it is a live reflection of what your OAP is actually reporting. Horizon asks OAP which layers exist and which of them have services, and renders exactly those, refreshing on a 60-second cadence. A layer starts reporting, it shows up; it goes quiet, it falls away. The menu can’t drift from reality because the menu is reality, polled.

Figure 1: Horizon’s home — the live estate on the left, the cross-layer Services overview on the right.

A few things are happening in that sidebar:

Live service counts. The Layers heading shows how many layers currently have services — 13 with services in Figure 1 — and each layer’s own count surfaces as you open it. Those counts come from a single server-side catalog the whole UI shares, refreshed once a minute, so the sidebar, the alarm layer-tagger, and the landing pages never disagree by a stale poll.
Grouped, not a flat list. Layers cluster under their group — Virtual Targets, Istio, Kubernetes, MQ — beneath the Overviews and Alarms entries up top. Operate and admin areas (Cluster Status, Alerting rules, DSL Management, Users, Roles & permissions) appear lower down, and only for the roles allowed to open them — access control is woven into the menu itself, not bolted on after.
Nothing silently hidden. Every layer OAP reports now appears — including ones with no bundled template, which fall back to a plain Service page. A hard-coded “hidden layers” list used to quietly drop layers like BanyanDB; that’s gone. Hiding a layer is now an explicit choice in horizon.yaml via layers.excluded (which ships defaulting to FAAS and VIRTUAL_GATEWAY — clear the list to surface everything).

Click into any layer and Horizon opens its first available tab, following a consistent spine across every layer:

service → instance → endpoint → topology → trace → logs → profiling

The slot names follow the layer. The General Service layer in Figure 2 labels its endpoint slot API, adds an API dependency view, and fans Profiling into the engines it actually has — Trace, eBPF, pprof (Go), and Async. Tabs a layer can’t support are simply turned off in its template, so you never land on an empty page. Pick a service and its dashboard fills the canvas to the right — a header strip of KPIs (RPM, Apdex, error rate, each with its own sparkline) over a widget grid scoped to exactly that entity.

Figure 2: Expand a layer and it fans out into its full workflow — the tab spine on the left, the selected service’s dashboard on the right.

Never a blank page

A console that follows live data has to handle the moments when there isn’t any — a fresh install, a partially configured deployment, an OAP that just restarted. Horizon treats those as first-class states instead of dead ends.

Open the app at / and Horizon cascades to a real destination: the first available public overview dashboard, or failing that the first layer with services, and only if neither exists, the empty landing. When it does land on the empty page, it tells you which problem you have in plain language — “No data is flowing yet” (nothing is reporting) versus “No dashboard configured yet” (services exist, but no overview is set up) — and points you at your operations team rather than dropping you on a blank grid. As soon as a service reports or an operator publishes a dashboard, the next 60-second refresh replaces the empty page with the real one.

Figure 3: The empty landing names the actual situation — here, services are reporting but no overview is configured — instead of a blank dashboard.

The same instinct shows up when OAP itself blips. If the backend goes briefly unreachable, Horizon keeps the last known sidebar shape on screen and raises an “OAP unreachable” banner, with service counts marked unknown until it recovers — so a short outage never looks like your configuration vanished.

When data is flowing, that landing is the war-room you already saw in Figure 1 — a cross-layer overview with per-kind service tiles (General services, Virtual databases, caches, MQs, GenAI), the live topology, and the active-alarm rail. Per-layer landings rank the true top-N services by your chosen column (no more capping at an arbitrary first 25 before ranking) and tell you “top N of M” so the trim is never silent.

And because long layer names and deep namespaces happen, the shell gets out of the way: drag the divider to resize the sidebar (double-click to reset), or fold it to a thin icon rail to give the canvas every horizontal pixel — and the width is remembered per browser.

Figure 4: Drag the divider to widen the sidebar — long names stop truncating, and the width is remembered per browser.

Figure 5: Fold the sidebar to a thin icon rail when you want the canvas to have every pixel.

One new tier makes the rest possible

Until now, SkyWalking’s web UI talked straight to OAP from the browser. Horizon introduces one small piece of infrastructure in between: a Backend-for-Frontend (BFF), a Fastify service on Node.js that serves the UI and proxies every call to OAP.

The browser talks only to the BFF; the BFF owns auth, RBAC, audit, capability probing, and server-side i18n / widget gating, then proxies to OAP’s query host (:12800) and admin host (:17128).

That tier is why the later posts in this series exist at all. Authentication, role-based access control, and the audit trail are enforced on the server, where a forged request can’t get past them. The BFF probes OAP’s GraphQL schema once at startup and degrades gracefully when a capability is missing — which is exactly how Horizon supports two OAP generations from one build (more on that below). It caches the service catalog once a minute so the whole UI shares one view of the estate. We’ll come back to each of these in the posts on operations, security, and customization; for now the thing to know is that there’s a server here now, and it’s doing real work.

A 3D look at the whole thing

One surface is worth previewing up front, because it captures the “stand back and look at everything at once” idea better than any screenshot can: the 3D Infrastructure Map. Every layer’s services become cubes, stacked onto tiers that read top-to-bottom the way a request flows, with live traffic, alarms, and call relationships drawn between them. Drag to orbit it:

Interactive · sample data Open the 3D map

A dedicated post later in the series takes the 3D map apart properly — the tiers, the alarm beacons, “Beacon mode” that ghosts everything healthy so only what’s firing glows, and the structured editor that configures it. For now, it’s a fair picture of the ambition: your whole deployment, in one view, alive.

What’s in this series

Fifteen posts follow this one, each a standalone tour of one corner of Horizon — read them in any order; each links back here for the lay of the land. They fall into four arcs.

See your data

Dashboards & MQE — widgets that query only what’s relevant to the entity in front of you, value formatting humans can read, synced crosshairs, and multi-entity compare.
Topology & service dependency — one topology engine that repaints for every layer, the de-noising filter, and the multi-hop API-dependency graph.
The Deployment tab & BanyanDB self-observability — a view that looks inside a single clustered service, and SkyWalking finally watching its own database the way it watches everything else.
The 3D Infrastructure Map — the full treatment of the view above.
Trace explorer — lasso the slow traces on a duration scatter, then read each one three ways.
Log explorer — a Loki-style stream with facets, top patterns, and structured payloads.
Browser & RUM monitoring — front-end error logs, and de-obfuscating a minified stack back to its original source line.
Five profilers, one flame graph — trace, async-profiler, eBPF, Go pprof, and network profiling, unified behind one workflow.

Operate it

Alarms & incident triage — incident-centric active alarms that ship the chart that fired them.
Runtime rules, live debugging & inspect — hot-reloadable rules and a live debugger that steps OAL (traces), MAL (metrics), and LAL (logs) against live samples.
Platform & cluster introspection — read the OAP cluster’s health, resolved config, and data retention from the UI.

Govern & secure it

Access control & security — server-enforced RBAC, LDAP/AD, an audit trail, and break-glass — the capabilities that make the UI enterprise-deployable.

Make it yours, and adopt it

Customization: config-driven layer templates — draft-to-publish, and adding a whole new monitored layer with zero UI code.
Localization — every dashboard in eight languages, translated by clicking widgets in a live preview.
Getting started & migration — install, the OAP version matrix, and swapping an existing UI for Horizon.

Try it against your OAP today

Horizon runs against the OAP you already have — and on today’s OAP 10.x nearly all of it works. Every dashboard, the topology, traces (native and Zipkin), logs, alarms, and all five profilers render off OAP’s query host (:12800), and Horizon’s access control, audit, and themes run in the BFF, independent of the OAP version. What waits for OAP 11.0 — the admin host (:17128), releasing soon — is the operate layer: runtime-rule (DSL) management, the Live Debugger, Metrics Inspect, the alarm-rule editor, the Cluster Status admin pane, and publishing template edits back to OAP. Horizon detects each admin module by its presence and simply hides the pages 10.x can’t serve, so the full observability console runs on 10.x today and the operate tooling lights up the moment you move to 11.0.

Point Horizon at your existing cluster and bring it up — no backend changes, the same OAP your deployment already talks to:

docker run -d --name horizon \
  -p 8081:8081 \
  -v "$PWD/horizon.yaml:/app/horizon.yaml:ro" \
  -v horizon-state:/data \
  ghcr.io/apache/skywalking-horizon-ui:<version>

A minimal horizon.yaml is just where OAP lives and one local user to log in as:

oap:
  queryUrl: http://<oap-host>:12800
  adminUrl: http://<oap-host>:17128
auth:
  backend: local
  local:
    users:
      - username: admin
        passwordHash: "$argon2id$v=19$..."   # generated, never plaintext
        roles: [admin]

Open http://<host>:8081/, log in, and the first stop is Cluster Status to confirm Horizon and OAP are talking. From there, the sidebar fills in with your estate.

For the full setup path — binary tarball, Kubernetes, LDAP, TLS, and the production checklist — see the Horizon UI documentation, which covers setup, compatibility, access control, customization, components, and operations from its left-side menu.

Other notable points

Drop-in against your existing OAP. Horizon is a greenfield rewrite that keeps every backend contract — the same GraphQL query protocol, admin REST surface, MQE language, and Layer concept — so you point it at a running cluster with no backend change. On today’s OAP 10.x the whole observability console works (dashboards, topology, traces including Zipkin, logs, alarms, profiling) along with Horizon’s BFF-side access control, audit, and themes; only the operate tooling — runtime rules, Live Debugger, Inspect, the Cluster Status admin pane, and publishing template edits — waits on OAP’s admin host (:17128), which ships with OAP 11.0 — releasing soon.
Dark-first and dense. A 12-column grid built for incident scanning — more signal above the fold, less whitespace.
Built on a modern stack. Vue 3 + TypeScript on Vite, Pinia, Apache ECharts, D3, and Monaco on the front end; Fastify on Node.js for the BFF.
It’s Apache-licensed and community-built. Horizon UI lives at apache/skywalking-horizon-ui. Try it against your cluster, and tell us what’s missing — issues and pull requests are welcome.

Next up in the series: the dashboards — and why a widget can decide, on the server, that it shouldn’t even run its query for the entity you’re looking at.

Blog: Meet Horizon UI · 2/16: Dashboards That Adapt — MQE, Smart Widgets, and Numbers Humans Can Read

Sun, 21 Jun 2026 00:00:00 +0000

This is the second post in the Apache SkyWalking Horizon UI series. The first one introduced the console and its layered navigation; this one is about the surface you spend most of your day on — the dashboards.

Every dashboard in Horizon is the same machine: a grid of widgets, each one an MQE expression the BFF resolves against OAP. What makes the surface worth a post of its own is what that machine does around the numbers — it hides the widgets that don’t apply to the entity you’re looking at (and skips their queries entirely), it renders coded values and raw byte counts as things a human can read, it ties every chart on the page to one cursor, and it drops you from a slow-SQL row straight into the trace that produced it. Let’s walk through it.

A layer dashboard is a dense 12-column grid (120px rows, gaps backfilled so a wall of tiles has no holes, collapsing to one column under ~1100px). Each tile is one of five widget types, and the type follows the shape of its MQE expression:

card — the expression collapses to a single scalar (latest(...), avg(...), service_sla/100). One big number.
line — a time series; one line per expression, optional dual y-axis for mixed units (throughput on the left, latency on the right).
top — a ranked list from top_n(endpoint_cpm, 20, des), with a small tab switcher to flip the ranking between Traffic / Slow / Successful Rate.
record — record-shaped output like slow database statements or slow cache commands: rows of text + value.
table — a labeled latest(...) metric, one row per label combination (pod phase per service, node condition, replicas per deployment).

You don’t pick the chart; you write the MQE and the right widget renders it. And it’s the same grid system at every altitude — the dashboards.<scope> map on a layer template carries a different widget set for the service, instance, and endpoint pages, so drilling down swaps the whole dashboard to the right scope. (All of these run on the BFF tier introduced in part 1 — the browser never talks to OAP directly.)

A dashboard that adapts to the entity in front of you

Here’s the feature that changes how a dashboard feels. A widget can carry a visibleWhen gate, and when the gate doesn’t hold, the widget doesn’t render — and, crucially, its query never runs.

There are two kinds of gate:

MQE metric — show the widget only when an expression has value (op: exists), or when its value crosses a threshold (gt / lt). Point a widget at its own metric and it self-gates: the JVM widgets carry "visibleWhen": { "kind": "mqe", "expression": "instance_jvm_cpu", "op": "exists" }, so they appear on a Java instance and vanish on a Go one.
Entity attribute — on the Instance scope, gate on the selected instance’s attributes (language eq JAVA, or an attribute simply being present).

Because the gate is evaluated server-side, a non-JVM instance doesn’t just hide the JVM tiles — the BFF never sends their queries to OAP at all. Open the same Instance dashboard for a JVM service and a non-JVM one and you’re looking at one template adapting itself, not two hand-built pages:

Figure 1: On a JVM instance the JVM widgets render — their visibleWhen gate holds.

Figure 2: The same dashboard on a Go instance — the JVM widgets aren’t there, and their queries never ran. One template, adapting to the entity.

Numbers humans can read

Raw metrics are not always readable metrics. Horizon’s widgets format three cases that used to make operators do math in their heads:

enum — a value→label map turns a coded gauge into words: a 1/0 success metric renders OK / Failed instead of the bare number. The labels are translatable per locale.
duration — a metric in seconds renders as a human time-ago: 5m 20s ago, compact to 5m / 2h on an axis.
SI suffixes — large magnitudes on chart axes and tooltips read as 45.1k, 1.34M, 2.5G rather than 4.51e4, with the axis tick and its hovered value sharing one notation.

Figure 3: Dense byte and count series get compact SI suffixes, axis and tooltip in step.

Figure 4: The enum and duration formats in action on BanyanDB’s lifecycle cards — OK instead of 1, “5m 20s ago” instead of a seconds count.

Read the whole grid as one timeline

Every line chart on a page shares one hover cursor. Point at minute 32 on the throughput chart and minute 32 lights up on the latency chart, the error-rate chart, and every sparkline tile. The contract is enforced at the chart-wrapper level — no widget can opt out — so the page reads as a single coordinated view of one moment, not a dozen independent charts. The multi-series tooltip is a fixed, aligned table that shows each series’ title (never the raw MQE), with values in one right-aligned column.

Figure 5: One cursor moves across every line chart on the page, so you read the same instant everywhere at once.

From a slow row to its trace, in one click

record widgets — Slow Statements, Slow Commands, Slow Database Statements — are lists of sampled records, and each row that carries a trace id gets a jump-to-trace icon at its head that opens the originating trace’s waterfall. It resolves the trace by id, not by layer, which matters: the Slow Statements on a Virtual Database / Cache / MQ service belong to the caller on another layer, and a virtual-target layer has no traces tab of its own — yet the jump still lands. The statement text itself is click-to-copy.

Figure 6: From a slow statement to the trace that ran it — resolved by trace id, so it works even on a virtual layer with no traces tab of its own.

Pin and compare entities

Sometimes one entity isn’t enough. Horizon lets you lock several services, instances, or endpoints — even ones from different services — and compare them in place. Pin entities from the picker or the instance/endpoint list; the one you’re viewing is always part of the cohort (tagged CURRENT) and still drives the header, and each pin adds its own hue. Every widget then compares inline — line widgets overlay one series per entity, cards show a row each, top and record widgets get per-entity tabs, tables gain an Entity column. A persistent comparison bar holds the cohort no matter how the underlying list paginates or which entity you’re currently viewing, and each entity loads as its own request, so one slow one never blanks the others.

Figure 7: Lock entities — even across services — and every line widget overlays them hue-by-hue; the comparison bar holds the cohort while the CURRENT entity still drives the header.

The time picker moves the whole dashboard

The topbar time range drives everything on the page — the header KPI strip, the widget body, and (on BanyanDB’s tiered hot/warm/cold storage) the Cold pill flow end-to-end. Earlier the landing and topology routes were pinned to the last 60 minutes, so picking “12 days ago” quietly kept showing recent numbers; now the picker is honored everywhere, and when an upstream control changes, each dependent tile visibly resets and shows a “Reading data…” hint rather than leaving a stale value under a spinner.

Where to go next

It’s worth stressing that this is one system. The same five widgets, the same MQE, the same gating render every layer’s dashboard — the JVM panels above, BanyanDB’s lifecycle cards, the percentile latency on a mesh service, and purpose-built panels for things like an Envoy AI Gateway (token throughput, time-to-first-token) or a GenAI virtual layer (per-model estimated cost). What changes from layer to layer is the MQE, not the machinery.

Everything above is the reading experience. Each widget’s MQE, its visibleWhen gate, its format, and the per-scope grids are all editable from the Layer dashboards admin — but that authoring story (draft → preview → publish, with the inline-and-expand MQE editor) is its own post later in the series. For the field-level reference, see the docs on dashboard widgets and charts.

Next up: topology and service dependency — the same data Horizon charts here, drawn as a map you can walk.

Blog: Meet Horizon UI · 3/16: Topology & Service Dependency

Sun, 21 Jun 2026 00:00:00 +0000

This is the third post in the Meet Horizon UI series. Part 2 was about reading your services as numbers on a dashboard; this one is about reading them as a map — who calls whom, how hard, and how that same logical service looks across every layer it reports through.

The call data behind a SkyWalking topology is one thing; the views Horizon draws from it are several. There’s the per-layer service map, a drill-down into the instances behind a single call, an endpoint-level dependency graph, and a cross-layer overlay that ties a service’s faces together. They’re all the same engine, pointed at different questions. (The Deployment tab — the map of one clustered service’s own instances — and the WebGL 3D map are big enough to get their own posts next.)

One topology engine, repainted per layer

Open any layer’s Topology tab and you get a left-to-right hierarchical service map: User (when present) seeds the left, and each service sits in a column by its call depth — within a column, nodes keep the order the graph walk reached them, so the dominant chain reads top-down. Each service is a hexagonal node, and everything it shows is driven by the layer’s config — nothing is hardcoded:

the hexagon’s border carries the node’s ring metric as an SLA-style health band (green → red);
the component icon sits inside the hex — the same icon set the trace waterfall uses, so a PostgreSQL node looks like PostgreSQL, a Kafka node like Kafka;
the node’s headline number — its center metric — prints just above the hex, with its unit;
the service name prints below it, and a secondary metric (latency, by default) sits beneath the name;
each edge carries the call’s throughput as an RPM chip (the server-side metric, falling back to the client side).

Here’s the part worth stressing: every one of those is just the General layer’s bundled default. The node’s center / ring / secondary metrics and the edge metrics each live in the Layer dashboards admin → Topology scope as an MQE expression with a unit and a role — so you can point any slot at a different metric, and the same engine paints a different map for a different layer, or for this one your way. (The choices travel with the layer template’s export/import, like everything else.)

Figure 1: One template-driven topology engine — health-banded hex nodes, RPM-chipped edges, real component icons; every metric on it is configured per layer.

Cut the noise

Real topologies are noisy — a dense map fills with conjectured peers OAP couldn’t fully resolve (a bare rcmd:80, an un-instrumented address). Horizon’s Filter control turns those off without hiding your real dependencies. It derives one facet automatically — by layer — and presents it exactly as the sidebar does: each row carries the layer’s own icon and localized name (Virtual Database, Java Agent, …), plus an Others bucket for nodes OAP couldn’t classify and a standalone User toggle. Uncheck Others and the uninstrumented clutter — and its dangling edges — disappears, while your databases, caches and queues (separated by their own VIRTUAL_* rows) stay on the map. Filtering is client-side and the rows re-derive on every refresh, so it never goes stale.

Figure 2: De-noise in one click — drop the unresolved “Others” peers and keep your real dependencies.

When a layer’s services fall into OAP service groups, the map’s service-focus selector groups by them too, and clicking a group header batch-selects or clears every service in that group — so you can focus a whole team’s slice of a busy map at once.

Figure 3: Focus a whole service group at once from the map’s selector.

Drill from a call into its instances

A service-to-service edge is an aggregate — behind it are real instances talking to real instances. Click a call on the map and choose Instance map →, and Horizon draws exactly that: the client service’s instances in the left column, the server service’s in the right, with the instance-level calls between them, animated client→server. It reuses everything from the service map — the health-ring nodes, the per-call client/server metric sidebar, a node popover with Open instance dashboard — and labels the columns in the layer’s own vocabulary (Pods on Kubernetes, Sidecars on the data plane). The two service pickers are relationship-aware: the server list is the chosen client’s callees, the client list is the chosen server’s callers, each re-deriving as you change the other.

Figure 4: Drill from an aggregate call into the instance-to-instance traffic behind it.

Walk the request chain, endpoint by endpoint

Service topology answers “which services call this one.” The API dependency tab answers the sharper question — “which endpoints call this endpoint, and which does it call.” Pick an endpoint and it lays out in columns by direction: callers on the left, the focus endpoint in the centre, callees on the right, with the same SLA-coloured node border, the RPM and latency on every edge, and the heaviest edge labeled. A selected node shows a single + handle that pulls in its own callers and callees in one click, so you walk the chain one hop at a time instead of drowning in the whole graph; drag nodes apart, and the drill-out links (Open endpoint, Service →) open in a new tab so you keep the graph you’re exploring.

Figure 5: Walk the endpoint call chain a hop at a time, latency on every edge.

One service, every layer it reports through

A single logical service often reports through several layers at once — a General agent, a Service Mesh sidecar, the mesh data plane, a Kubernetes pod. SkyWalking has modeled that cross-layer hierarchy since OAP 10; what Horizon adds is making it one click from anywhere on the map. Select a node and Horizon lazily probes its hierarchy — if the service has cross-layer peers, a small chevron-stack chip clips to the node.

Figure 6: The chevron-stack chip on a selected node — lazily probed on selection, it shows only when the service has cross-layer peers.

Click the chip and the topology dims under a Smartscape overlay: the focused node re-renders bright in place, and its peers fan out vertically by OAP’s layer order — request-near layers above, infra-near below. From there a two-step click opens any peer in its own layer, pre-selected. (Auto-refresh pauses while the overlay is open so nothing shifts under you.)

Figure 7: One service, every layer it reports through — the cross-layer hierarchy as a one-click overlay on the map.

Where to go next

Every metric, threshold, and edge weight on these maps lives in the layer template’s topology block — which means you tune them the same config-driven way you tune dashboards, the subject of a later post in this series. For the field reference, see the layer-template topology docs.

Next up: the Deployment tab and BanyanDB self-observability — where the same map technique turns inward to show how one clustered service’s own instances are deployed and talk to each other.

Blog: Meet Horizon UI · 4/16: The Deployment Tab & BanyanDB Self-Observability

Sun, 21 Jun 2026 00:00:00 +0000

This is the fourth post in the Meet Horizon UI series. Part 3 drew the map between services. This one turns the same map inward — onto the instances inside one clustered service — and uses it for something SkyWalking has never shown well before: its own storage engine, BanyanDB, modeled as the cluster it actually is.

The Deployment tab: a map of one service’s own instances

The service map answers “who calls this service.” The new per-layer Deployment tab answers a different question: “how is this one service deployed, and how do its own instances talk to each other?” Pick a service and the tab draws its instances as nodes with the instance-to-instance calls between them — the same pan/zoom canvas, health-ring nodes, animated edge flow and per-call metric sidebar you know from the service map, but scoped to a single service’s internals.

Three things make it more than a flat node cloud:

Instances render as hexagons that bundle into pods. A pod’s main container is a full hex with its sibling containers attached as smaller hexes around its edge — so a main process and its sidecars read as one unit, and a cross-pod sidecar link connects the exact small hex it belongs to.
Pods cluster into labelled boxes by a rule you choose — a single instance attribute (role), several attributes combined (e.g. node_role + node_type), or a name regex — so a fleet of mixed-role nodes reads as one box per role instead of a cloud.
The layout is tiered. Each cluster box lays its pods out by call depth — sources on the left, what they call to the right — so an upstream→downstream chain reads left-to-right; drag any pod and its box re-flows to keep everything enclosed.

Edges are keyed by the (source-role → target-role) pair, so each kind of link shows its own metrics rather than one flat set, prints its headline number inline on the edge, and lists in full in a Flows sub-tab — one aligned table per role-pair. It’s off by default and, like the service map, entirely configured from the Layer dashboards admin → Deployment scope.

BanyanDB, watched like everything else

That machinery exists for a reason. SkyWalking’s native database, BanyanDB, is a clustered, role- and tier-aware system — and until now SkyWalking couldn’t really observe it as one. The new BanyanDB layer (under Self-Observability), pairing with OAP backend SWIP-15, models the whole deployment from metrics scraped through BanyanDB’s FODC proxy:

the cluster is one Cluster (a service),
each container is one Container (an instance, carrying its container_name role and node_type tier as attributes),
and each storage Group is an endpoint.

So the same Service / Instance / Endpoint spine every other layer uses now means Cluster / Container / Group for BanyanDB — and the Deployment tab on top of it draws the cluster itself.

Figure 1: SkyWalking watching its own database — the BanyanDB cluster drawn by role and tier, with liaison→data and lifecycle→data edges between the pods.

Cluster, Container, Group

Each scope is a purpose-built dashboard:

The Cluster dashboard is the war-room for the whole database: write / query / error-rate KPIs, CPU / memory / disk capacity, throughput and error trends, and a Containers by Role table.
The Container dashboard adapts to the selected container’s role. Every container shows CPU / memory / Go-runtime resources; a liaison adds ingestion, query, gRPC errors and the tier-2 publish pipeline and write-queue depth; a data node adds storage totals, merge / compaction, the inverted index, the subscribe queue and retention; a lifecycle sidecar shows migration cycles and last-run time / status. The role-specific panels are gated on the container’s role attribute, so you only ever see what applies to the node in front of you.
The Group dashboard splits per data-model — measure, stream, trace, property — and because a BanyanDB group stores exactly one catalog, only the matching model’s panels render: a measure group shows write-rate / query-latency / merge panels, a property group its index-write / term-search / series panels, and so on.

Figure 2: The Cluster scope — the whole database at a glance, with a roll-call of containers by role.

The role-gating is easiest to see by opening the same Container dashboard on two different roles:

Figure 3: A data/liaison Container — ingestion, query, storage and compaction panels on top of the shared resource panels.

Figure 4: The same dashboard on the lifecycle node — just the migration panels. Same template, gated by role.

And the Group scope gives each storage catalog its own page:

Figure 5: The Group scope — one storage catalog at a time, its panels gated to the group's data model.

Edges that know their role pair, and a Flows table

On the Deployment tab, the call edges between containers carry role-pair-specific metrics off the SWIP-15 instance-relation families: a liaison → data edge shows write / query / part-sync throughput and p99; a liaison → liaison edge shows write-forward and control; a lifecycle → data edge shows tier-migration volume / rate / p99. Each edge prints up to three of its pair’s metrics inline, the selected-edge panel keeps the full client-vs-server breakdown, and the Flows sub-tab lays every edge out as one aligned table per role-pair.

Figure 6: Flows — the same role-pair edges as a sortable table, one block per pair.

(Two preconditions. The edges and the role-specific panels assume a real clustered BanyanDB — a single-process standalone instance shows only the shared resource and Go-runtime panels, with the rest lighting up as the cluster’s roles report. And the container-to-container edges in particular need the OAP build to expose the SERVICE_INSTANCE_RELATION scope; until it does, the Deployment tab still draws the full inventory — just without the edges between pods.)

Configured, not coded

None of the above is a hand-built “BanyanDB screen.” The clustering rules, the per-role node metrics, and the role-pair edge metrics are all a self-contained block on the layer template, edited from the Layer dashboards admin → Deployment scope and carried with the template’s export/import — the same config-driven model behind every other layer, which a later post covers end to end.

Next up: the 3D Infrastructure Map — where this same deployment, and every other layer, lifts off the page into a WebGL view of your whole estate.

Blog: Monitoring Envoy AI Gateway with Apache SkyWalking

Thu, 02 Apr 2026 00:00:00 +0000

LLM traffic is becoming a first-class citizen in production infrastructure. Teams are calling OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Gemini — often multiple providers at once. But most organizations have no unified visibility into this traffic:

Token costs spiral without knowing which teams, models, or providers drive the spend. A single misconfigured prompt template can burn through thousands of dollars before anyone notices.
Provider outages cause cascading failures. When OpenAI has a bad hour, your application goes down with it — and you have no failover visibility to understand what happened or switch providers automatically.
No unified metrics across heterogeneous LLM calls. Latency, Time to First Token (TTFT), Time Per Output Token (TPOT), token usage, error rates — each provider reports these differently, if at all. There is no single dashboard to compare them.

This is the same observability gap that microservices faced a decade ago. The solution then was service meshes and API gateways with built-in telemetry. For AI workloads, the answer is an AI gateway.

Why an AI Gateway

Envoy AI Gateway is an open-source AI gateway built on top of Envoy Proxy and Envoy Gateway. It is not a standalone SaaS product or a Python proxy — it is infrastructure-grade software built on the same Envoy that already handles traffic for a large portion of cloud-native deployments.

Key capabilities:

Multi-provider routing — supports 16+ AI providers (OpenAI, Anthropic, AWS Bedrock, Azure OpenAI, Google Gemini, Mistral, Cohere, DeepSeek, and more) behind a unified API.
Token-based rate limiting — rate limit by token consumption, not just request count.
Provider fallback — automatic failover when a provider is down or slow.
Model virtualization — abstract model names so applications are decoupled from specific providers.
Two-tier architecture — a reference architecture with a centralized entry gateway (Tier 1) for auth and global routing, and per-cluster gateways (Tier 2) for inference optimization.
CNCF ecosystem native — runs on Kubernetes, composes with existing Envoy filters, WASM plugins, and standard Kubernetes Gateway API resources.

Because Envoy AI Gateway natively emits GenAI metrics and access logs via OTLP following OpenTelemetry GenAI Semantic Conventions, it plugs directly into any OpenTelemetry-compatible backend.

Starting from SkyWalking 10.4.0, the OAP server natively receives and analyzes Envoy AI Gateway’s OTLP metrics and access logs — no OpenTelemetry Collector needed in between.

Data Flow

The AI Gateway pushes telemetry directly to SkyWalking via OTLP gRPC:

Application sends LLM API requests through the Envoy AI Gateway.
Envoy AI Gateway routes requests to AI providers (or local models like Ollama) and records GenAI metrics (token usage, latency, TTFT, TPOT) and access logs.
The gateway pushes metrics and logs via OTLP gRPC directly to SkyWalking OAP on port 11800.
SkyWalking OAP parses metrics with MAL rules and access logs with LAL rules, then stores everything in BanyanDB.

No OpenTelemetry Collector is needed. SkyWalking OAP’s built-in OTLP receiver handles everything.

Try It Locally

This demo uses Ollama as a local LLM backend so you can try everything without an API key. The Envoy AI Gateway CLI (aigw) provides a standalone mode that runs outside Kubernetes — perfect for local testing.

Prerequisites

Docker and Docker Compose
Ollama installed on your host

Step 1: Start Ollama

Start Ollama on all interfaces so Docker containers can reach it:

OLLAMA_HOST=0.0.0.0 ollama serve

Pull a small model for testing:

ollama pull llama3.2:1b

Step 2: Start the Stack

Create a docker-compose.yaml:

services:
  banyandb:
    image: apache/skywalking-banyandb:0.10.0
    container_name: banyandb
    ports:
      - "17912:17912"
    command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:17913/api/healthz || exit 1"]
      interval: 5s
      timeout: 3s
      retries: 10

  oap:
    image: apache/skywalking-oap-server:10.4.0
    container_name: oap
    depends_on:
      banyandb:
        condition: service_healthy
    ports:
      - "11800:11800"
      - "12800:12800"
    environment:
      SW_STORAGE: banyandb
      SW_STORAGE_BANYANDB_TARGETS: banyandb:17912
    healthcheck:
      test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/12800' || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 30
      start_period: 60s

  ui:
    image: apache/skywalking-ui:10.4.0
    container_name: ui
    depends_on:
      oap:
        condition: service_healthy
    ports:
      - "8080:8080"
    environment:
      SW_OAP_ADDRESS: http://oap:12800

  aigw:
    image: envoyproxy/ai-gateway-cli:latest
    container_name: aigw
    depends_on:
      oap:
        condition: service_healthy
    environment:
      - OPENAI_BASE_URL=http://host.docker.internal:11434/v1
      - OPENAI_API_KEY=unused
      - OTEL_SERVICE_NAME=my-ai-gateway
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://oap:11800
      - OTEL_EXPORTER_OTLP_PROTOCOL=grpc
      - OTEL_METRICS_EXPORTER=otlp
      - OTEL_LOGS_EXPORTER=otlp
      - OTEL_METRIC_EXPORT_INTERVAL=5000
      - OTEL_RESOURCE_ATTRIBUTES=job_name=envoy-ai-gateway,service.instance.id=aigw-1,service.layer=ENVOY_AI_GATEWAY
    ports:
      - "1975:1975"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    command: ["run"]

Start everything:

docker compose up -d

Wait for all services to become healthy (BanyanDB starts first, then OAP, then UI and AI Gateway):

docker compose ps

The key OTLP configuration on the aigw service:

Env Var	Value	Purpose
`OTEL_SERVICE_NAME`	`my-ai-gateway`	Service name in SkyWalking
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://oap:11800`	SkyWalking OAP gRPC endpoint
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`	OTLP transport
`OTEL_METRICS_EXPORTER`	`otlp`	Enable metrics push
`OTEL_LOGS_EXPORTER`	`otlp`	Enable access log push

The OTEL_RESOURCE_ATTRIBUTES must include:

job_name=envoy-ai-gateway — routing tag for MAL/LAL rules
service.instance.id=<id> — instance identity
service.layer=ENVOY_AI_GATEWAY — routes logs to AI Gateway LAL rules

The MAL and LAL rules are enabled by default in SkyWalking OAP. No OAP-side configuration is needed.

Step 3: Run the Demo App

Create a simple Python application that sends requests through the AI Gateway (app.py). It mixes normal requests, streaming requests (for TTFT/TPOT metrics), and error requests (non-existent model → HTTP 404, always captured by the LAL sampling policy):

import time, random, requests

GATEWAY = "http://localhost:1975"
HEADERS = {"Authorization": "Bearer unused", "Content-Type": "application/json"}

questions = [
    "What is Apache SkyWalking? Answer in one sentence.",
    "What is Envoy Proxy used for? Answer in one sentence.",
    "What are the benefits of an AI gateway? Answer in two sentences.",
    "Explain observability in three sentences.",
]

def chat(model, question, stream=False):
    resp = requests.post(
        f"{GATEWAY}/v1/chat/completions",
        json={"model": model, "messages": [{"role": "user", "content": question}], "stream": stream},
        headers=HEADERS, timeout=60, stream=stream,
    )
    if stream:
        chunks = []
        for line in resp.iter_lines():
            if line:
                chunks.append(line.decode())
        return resp.status_code, f"[streamed {len(chunks)} chunks]"
    return resp.status_code, resp.json()

while True:
    r = random.random()
    if r < 0.2:
        # Error request: non-existent model triggers 404
        status, body = chat("non-existent-model", "hello")
        print(f"[error] model=non-existent-model status={status}")
    elif r < 0.5:
        # Streaming request — generates TTFT and TPOT metrics
        q = random.choice(questions)
        status, info = chat("llama3.2:1b", q, stream=True)
        print(f"[stream] status={status} {info}")
    else:
        # Normal non-streaming request
        q = random.choice(questions)
        status, body = chat("llama3.2:1b", q)
        answer = body.get("choices", [{}])[0].get("message", {}).get("content", "")[:80]
        tokens = body.get("usage", {})
        print(f"[ok] status={status} tokens={tokens} answer={answer}...")
    time.sleep(random.randint(20, 30))

Run it:

pip install requests
python app.py

The application talks to the AI Gateway on port 1975, which routes to Ollama. Each request generates GenAI metrics (token usage, latency, TTFT, TPOT) and access logs that the gateway pushes to SkyWalking via OTLP.

The error requests (non-existent model → HTTP 404) are always captured by the access log sampling policy, so you will see them in the SkyWalking log view.

Step 4: View in SkyWalking UI

Open http://localhost:8080 and select the GenAI > Envoy AI Gateway menu.

The service list shows my-ai-gateway with CPM, latency, and token rates at a glance:

Click into the service to see the full dashboard — Request CPM, Latency (average + percentiles), Input/Output Token Rates, TTFT, and TPOT:

The Providers tab breaks down metrics by AI provider:

The Models tab shows per-model metrics including TTFT and TPOT (streaming only). Note the unknown model entries — these are the error requests with non-existent models:

The Log tab shows access logs. The sampling policy drops normal successful responses but always captures errors (HTTP 404) and high-token requests:

Cleanup

docker compose down

Deploying on Kubernetes

For production deployments, Envoy AI Gateway runs as a full Kubernetes controller with Envoy Gateway as the control plane. See the Envoy AI Gateway getting started guide for Kubernetes installation.

The OTLP configuration is the same — set the OTEL_* environment variables on the AI Gateway’s external processor to point at SkyWalking OAP’s gRPC port (11800). See the SkyWalking Envoy AI Gateway Monitoring documentation for details.

GenAI Observability Without an AI Gateway

Not every deployment uses an AI gateway. If your applications call LLM providers directly, SkyWalking 10.4.0 also provides GenAI observability through the Virtual GenAI layer.

This works with any SkyWalking-instrumented, OpenTelemetry-instrumented, or Zipkin-instrumented application. When traces carry gen_ai.* tags (following OpenTelemetry GenAI Semantic Conventions), SkyWalking derives per-provider and per-model metrics from the client side: latency, token usage, success rate, and estimated cost.

For Java applications, the SkyWalking Java Agent (9.7+) includes a Spring AI plugin that automatically instruments calls to 13+ providers (OpenAI, Anthropic, AWS Bedrock, Google GenAI, DeepSeek, Mistral, etc.) with the correct gen_ai.* span tags — no code changes needed.

This is a different use case from the Envoy AI Gateway monitoring covered above:

Envoy AI Gateway layer: infrastructure-level observability — what the gateway sees across all traffic. Best for platform teams managing centralized AI routing.
Virtual GenAI layer: application-level observability — what each instrumented app sees for its own LLM calls. Best for teams without a centralized gateway, or for per-application cost tracking.

References

Envoy AI Gateway — project site and documentation
Envoy AI Gateway CLI — standalone mode for local development
SkyWalking Envoy AI Gateway Monitoring — OAP setup doc
SkyWalking Virtual GenAI — client-side GenAI observability
OpenTelemetry GenAI Semantic Conventions — the metric/attribute standard both projects follow

Zh: 用 Apache SkyWalking 监控 Envoy AI Gateway

Thu, 02 Apr 2026 00:00:00 +0000

问题：LLM 流量缺乏统一观测

LLM 流量正在成为生产基础设施中不可忽视的一部分。团队同时在调用 OpenAI、Anthropic、AWS Bedrock、Azure OpenAI、Google Gemini——往往还不止一个提供商。但大多数组织对这些流量缺乏统一的可见性：

Token 费用失控，却不知道哪个团队、哪个模型、哪个提供商在烧钱。一个配置不当的 prompt 模板就可能在无人察觉的情况下烧掉几千美元。
提供商故障引发连锁反应。 OpenAI 出问题的那一小时，你的应用也跟着挂——而你既没有故障切换的可见性，也无法自动切换提供商。
缺乏统一指标。 延迟、首 Token 耗时（TTFT）、每 Token 输出耗时（TPOT）、Token 用量、错误率——每个提供商的报告方式都不一样，有些甚至不提供。没有一个统一的面板能做对比。

这和十年前微服务面临的可观测性困境如出一辙。当时的解法是服务网格和内置遥测的 API 网关。对 AI 工作负载来说，答案就是 AI 网关。

为什么选择 AI 网关

Envoy AI Gateway 是一个开源 AI 网关，构建在 Envoy Proxy 和 Envoy Gateway 之上。底层就是云原生世界里已经广泛部署的 Envoy，天然具备基础设施级的稳定性和性能。

核心能力：

多提供商路由 —— 支持 16+ AI 提供商（OpenAI、Anthropic、AWS Bedrock、Azure OpenAI、Google Gemini、Mistral、Cohere、DeepSeek 等），统一 API 接入。
基于 Token 的限流 —— 按 Token 消耗限流，而不只是按请求数。
提供商故障切换 —— 某个提供商宕机或响应慢时自动切换。
模型虚拟化 —— 抽象模型名称，让应用与具体提供商解耦。
两层架构 —— 参考架构包含一个集中入口网关（Tier 1）负责认证和全局路由，以及每集群网关（Tier 2）负责推理优化。
CNCF 生态原生 —— 运行在 Kubernetes 上，兼容现有的 Envoy Filter、WASM 插件和标准 Kubernetes Gateway API 资源。

Envoy AI Gateway 原生支持通过 OTLP 发送 GenAI 指标和访问日志，遵循 OpenTelemetry GenAI 语义约定，可以直接接入任何兼容 OpenTelemetry 的后端。

从 SkyWalking 10.4.0 开始，OAP 原生接收和分析 Envoy AI Gateway 的 OTLP 指标和访问日志——中间不需要部署 OpenTelemetry Collector。

数据流

AI Gateway 通过 OTLP gRPC 直接将遥测数据推送到 SkyWalking：

应用通过 Envoy AI Gateway 发送 LLM API 请求。
Envoy AI Gateway 将请求路由到 AI 提供商（或 Ollama 这样的本地模型），同时记录 GenAI 指标（Token 用量、延迟、TTFT、TPOT）和访问日志。
网关通过 OTLP gRPC 直接将指标和日志推送到 SkyWalking OAP 的 11800 端口。
SkyWalking OAP 用 MAL 规则解析指标、用 LAL 规则解析访问日志，然后统一存储到 BanyanDB。

不需要 OpenTelemetry Collector。SkyWalking OAP 内置的 OTLP 接收器可以直接处理所有数据。

本地体验

这个 Demo 使用 Ollama 作为本地 LLM 后端，不需要任何 API Key 就能跑起来。Envoy AI Gateway CLI（aigw）提供独立运行模式，不依赖 Kubernetes，非常适合本地测试。

前置条件

Docker 和 Docker Compose
主机上已安装 Ollama

第一步：启动 Ollama

让 Ollama 监听所有网络接口，以便 Docker 容器能访问到：

OLLAMA_HOST=0.0.0.0 ollama serve

拉取一个小模型用于测试：

ollama pull llama3.2:1b

第二步：启动服务栈

创建 docker-compose.yaml：

services:
  banyandb:
    image: apache/skywalking-banyandb:0.10.0
    container_name: banyandb
    ports:
      - "17912:17912"
    command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:17913/api/healthz || exit 1"]
      interval: 5s
      timeout: 3s
      retries: 10

  oap:
    image: apache/skywalking-oap-server:10.4.0
    container_name: oap
    depends_on:
      banyandb:
        condition: service_healthy
    ports:
      - "11800:11800"
      - "12800:12800"
    environment:
      SW_STORAGE: banyandb
      SW_STORAGE_BANYANDB_TARGETS: banyandb:17912
    healthcheck:
      test: ["CMD-SHELL", "bash -c 'echo > /dev/tcp/localhost/12800' || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 30
      start_period: 60s

  ui:
    image: apache/skywalking-ui:10.4.0
    container_name: ui
    depends_on:
      oap:
        condition: service_healthy
    ports:
      - "8080:8080"
    environment:
      SW_OAP_ADDRESS: http://oap:12800

  aigw:
    image: envoyproxy/ai-gateway-cli:latest
    container_name: aigw
    depends_on:
      oap:
        condition: service_healthy
    environment:
      - OPENAI_BASE_URL=http://host.docker.internal:11434/v1
      - OPENAI_API_KEY=unused
      - OTEL_SERVICE_NAME=my-ai-gateway
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://oap:11800
      - OTEL_EXPORTER_OTLP_PROTOCOL=grpc
      - OTEL_METRICS_EXPORTER=otlp
      - OTEL_LOGS_EXPORTER=otlp
      - OTEL_METRIC_EXPORT_INTERVAL=5000
      - OTEL_RESOURCE_ATTRIBUTES=job_name=envoy-ai-gateway,service.instance.id=aigw-1,service.layer=ENVOY_AI_GATEWAY
    ports:
      - "1975:1975"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    command: ["run"]

启动所有服务：

docker compose up -d

等待所有服务变为健康状态（BanyanDB 先启动，然后是 OAP，最后是 UI 和 AI Gateway）：

docker compose ps

aigw 服务的关键 OTLP 配置：

环境变量	值	用途
`OTEL_SERVICE_NAME`	`my-ai-gateway`	SkyWalking 中的服务名
`OTEL_EXPORTER_OTLP_ENDPOINT`	`http://oap:11800`	SkyWalking OAP gRPC 端点
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`	OTLP 传输协议
`OTEL_METRICS_EXPORTER`	`otlp`	启用指标推送
`OTEL_LOGS_EXPORTER`	`otlp`	启用访问日志推送

OTEL_RESOURCE_ATTRIBUTES 必须包含：

job_name=envoy-ai-gateway —— MAL/LAL 规则的路由标签
service.instance.id=<id> —— 实例标识
service.layer=ENVOY_AI_GATEWAY —— 将日志路由到 AI Gateway LAL 规则

MAL 和 LAL 规则在 SkyWalking OAP 中默认启用，不需要额外配置。

第三步：运行 Demo 应用

创建一个简单的 Python 应用，通过 AI Gateway 发送请求（app.py）。它混合了普通请求、流式请求（用于产生 TTFT/TPOT 指标）和错误请求（不存在的模型 → HTTP 404，始终会被 LAL 采样策略捕获）：

import time, random, requests

GATEWAY = "http://localhost:1975"
HEADERS = {"Authorization": "Bearer unused", "Content-Type": "application/json"}

questions = [
    "What is Apache SkyWalking? Answer in one sentence.",
    "What is Envoy Proxy used for? Answer in one sentence.",
    "What are the benefits of an AI gateway? Answer in two sentences.",
    "Explain observability in three sentences.",
]

def chat(model, question, stream=False):
    resp = requests.post(
        f"{GATEWAY}/v1/chat/completions",
        json={"model": model, "messages": [{"role": "user", "content": question}], "stream": stream},
        headers=HEADERS, timeout=60, stream=stream,
    )
    if stream:
        chunks = []
        for line in resp.iter_lines():
            if line:
                chunks.append(line.decode())
        return resp.status_code, f"[streamed {len(chunks)} chunks]"
    return resp.status_code, resp.json()

while True:
    r = random.random()
    if r < 0.2:
        # Error request: non-existent model triggers 404
        status, body = chat("non-existent-model", "hello")
        print(f"[error] model=non-existent-model status={status}")
    elif r < 0.5:
        # Streaming request — generates TTFT and TPOT metrics
        q = random.choice(questions)
        status, info = chat("llama3.2:1b", q, stream=True)
        print(f"[stream] status={status} {info}")
    else:
        # Normal non-streaming request
        q = random.choice(questions)
        status, body = chat("llama3.2:1b", q)
        answer = body.get("choices", [{}])[0].get("message", {}).get("content", "")[:80]
        tokens = body.get("usage", {})
        print(f"[ok] status={status} tokens={tokens} answer={answer}...")
    time.sleep(random.randint(20, 30))

运行：

pip install requests
python app.py

应用通过 1975 端口与 AI Gateway 通信，AI Gateway 再路由到 Ollama。每次请求都会产生 GenAI 指标（Token 用量、延迟、TTFT、TPOT）和访问日志，由网关通过 OTLP 推送到 SkyWalking。

错误请求（不存在的模型 → HTTP 404）始终会被访问日志采样策略捕获，所以在 SkyWalking 的日志视图中一定能看到。

第四步：在 SkyWalking UI 中查看

打开 http://localhost:8080，选择 GenAI > Envoy AI Gateway 菜单。

服务列表显示 my-ai-gateway，可以一览 CPM、延迟和 Token 速率：

点击进入服务详情，查看完整仪表盘——请求 CPM、延迟（平均值 + 百分位数）、输入/输出 Token 速率、TTFT 和 TPOT：

Providers 标签页按 AI 提供商维度展示指标：

Models 标签页展示每个模型的指标，包括 TTFT 和 TPOT（仅流式请求）。注意 unknown 模型条目——这些就是使用不存在模型的错误请求：

Log 标签页展示访问日志。采样策略会丢弃正常的成功响应，但始终保留错误（HTTP 404）和高 Token 消耗的请求：

清理

docker compose down

Kubernetes 生产部署

生产环境中，Envoy AI Gateway 作为完整的 Kubernetes 控制器运行，以 Envoy Gateway 作为控制面。详见 Envoy AI Gateway 入门指南。

OTLP 配置方式相同——在 AI Gateway 的 External Processor 上设置 OTEL_* 环境变量，指向 SkyWalking OAP 的 gRPC 端口（11800）。详见 SkyWalking Envoy AI Gateway 监控文档。

不用 AI 网关也能做 GenAI 可观测

并非所有场景都需要 AI 网关。如果你的应用直接调用 LLM 提供商，SkyWalking 10.4.0 也提供了基于 Virtual GenAI 层的 GenAI 可观测方案。

任何接入了 SkyWalking、OpenTelemetry 或 Zipkin 探针的应用都能使用这个功能。只要 Trace 中携带 gen_ai.* 标签（遵循 OpenTelemetry GenAI 语义约定），SkyWalking 就能从客户端视角推导出每提供商、每模型的指标：延迟、Token 用量、成功率和预估费用。

对于 Java 应用，SkyWalking Java Agent（9.7+）内置了 Spring AI 插件，自动为 13+ 提供商（OpenAI、Anthropic、AWS Bedrock、Google GenAI、DeepSeek、Mistral 等）的调用注入正确的 gen_ai.* Span 标签——不需要改代码。

这与上面介绍的 Envoy AI Gateway 监控是不同的使用场景：

Envoy AI Gateway 层：基础设施级可观测——网关视角，覆盖所有流量。适合负责集中 AI 路由的平台团队。
Virtual GenAI 层：应用级可观测——每个应用自己看到的 LLM 调用情况。适合没有集中网关的团队，或者需要按应用维度跟踪费用的场景。

参考资料

Envoy AI Gateway —— 项目官网和文档
Envoy AI Gateway CLI —— 本地开发用的独立运行模式
SkyWalking Envoy AI Gateway 监控 —— OAP 配置文档
SkyWalking Virtual GenAI —— 客户端侧 GenAI 可观测
OpenTelemetry GenAI 语义约定 —— 两个项目共同遵循的指标/属性标准

Zh: AI Coding 如何重塑软件架构师的工作方式

Sun, 15 Mar 2026 00:00:00 +0000

以 SkyWalking GraalVM Distro 为例，看 AI Coding 如何把一批探索性 PoC 打磨成一条可重复的迁移流水线。

这个项目给我最大的启发，不是 AI 能写多少代码，而是 AI Coding 改变了架构设计的试错成本。当一个想法可以很快做成 PoC、跑起来验证、不行就推翻重来时，架构师就更有机会逼近自己真正想要的设计，而不是过早停在“团队现在做得出来”的折中方案上。

这种变化在成熟开源系统里尤其重要。Apache SkyWalking OAP 长期以来一直是一个功能强大且经过生产验证的可观测性后端，但大型 Java 平台该有的问题它一个不少：运行时字节码生成、重反射初始化、classpath 扫描、基于 SPI 的模块装配，以及动态 DSL 执行——这些机制方便扩展，但做 GraalVM Native Image 时全是障碍。

SkyWalking GraalVM Distro 的出现，源于我们把这个挑战当成一个架构设计问题来处理，而不是一次性的移植工程。目标不仅是让 OAP 能以原生二进制运行，更是把 GraalVM 迁移本身做成一条可重复执行、能够持续跟上上游演进的自动化流水线。

如果你想看完整的技术设计、基准数据和上手方式，请阅读配套文章：SkyWalking GraalVM Distro：设计与基准测试。

从停滞的想法到可运行的系统

这件事其实很多年前就开始了。在这个仓库创建不久之后，yswdqz 曾花了数个月探索迁移方案。真正做下来才发现，这个项目远比 GraalVM 文档里列出的那些单点限制复杂得多，这项工作最终也因此搁置了很多年。

这段停滞很重要。缺少的并不是想法。成熟维护者通常从来不缺想法，真正稀缺的，是把这些想法真正做出来的时间、人力和精力。即使架构师已经看到了几条很有前景的路线，有限的开发资源也会迫使大家更早做出权衡：优先选择实现成本最低的方案，而不是那个更干净、更可复用、更经得起未来变化的方案。

这种情况非常普遍，并不特殊。在开源社区里，很多工作依赖志愿者或有限的企业赞助；在商业产品里，约束的形式不同，但本质仍然一样：路线图承诺、团队规模和交付压力都会让工程资源始终紧张。在这两种环境里，很多好想法被放弃，并不是因为它们错了，而是因为要把它们真正验证清楚、实现完整，成本太高。

还有一个同样重要的约束：架构师通常同时也是非常资深的工程师，而不是一个可以全职扑在实现细节上的人。问题在于个人编码精力有限、时间高度碎片化，同时还要在代码尚未出现之前，不断向其他资深工程师解释自己的设计意图。传统上，这种解释主要通过图、文档和沟通完成。它很慢、信息损失大，而且充满不确定性。我们都体验过“传话游戏”：哪怕是很简单的意思，也很容易被误解，而等误解真正暴露出来时，时间已经过去很多了。

到了 2025 年末，AI Coding 让”同时尝试多条路线”这件事终于变得现实。我们不必再因为实现能力稀缺而过早接受折中，而是可以在多个设计之间来回切换，用代码验证，快速淘汰弱方案，持续迭代，直到架构本身变得足够稳固、足够实用、足够高效。

这种设计自由度至关重要。GraalVM 文档对单个限制讲得很清楚，但成熟 OSS 平台遇到的是一整套彼此牵连的系统性问题。只修补一个动态机制远远不够。要让 native image 真正落地，我们必须把整类运行时行为改造成构建期产物和自动生成的元数据。

在这条路的早期历史中，还有一座非常具体的大山。那时上游 SkyWalking 仍然大量依赖 Groovy 来处理 LAL、MAL 和 Hierarchy 脚本。理论上，这只不过是另一个“不支持运行时动态行为”的例子；但在实践中，Groovy 是整条路径上最大的障碍。它不仅意味着脚本执行，还意味着一整套在 JVM 里极其便利、在 native image 里却极其不友好的动态模型。

为了跨过这道坎，我们围绕 AOT-first 模式重新设计了 OAP 的核心引擎。早期实验必须直接面对 Groovy 时代的运行时行为，并尝试不同的脚本编译方案来绕过去。最终方案走得更远：对齐上游编译器流水线，把动态生成前移到构建期，并引入自动化机制，让这条迁移路径在上游持续演进时依然保持可控。具体来说，就是把 OAL、MAL、LAL 和 Hierarchy 的生成过程变成构建期预编译器的输出，而不是继续保留为启动期的动态行为。

AI Coding 如何改写架构迭代

这次转变的关键，并不只是“写代码更快了”。AI 真正改变的，是想法、原型、验证和重设计之间来回迭代的速度。围绕同一个问题，我们可以很快做出几个可运行的 PoC，迅速淘汰不成立的方向，再把值得保留的抽象慢慢沉淀成一套连贯的迁移系统。

这并不会削弱人的架构价值，反而会放大它。哪些行为应该前移到构建期，哪些地方应该保留可配置性，哪里应该引入 same-FQCN 替换，如何让上游同步保持可控，以及哪些抽象值得不惜代价保留下来，这些判断仍然只能由人来做。不同的是，AI 的速度让我们终于有机会把这些更好的设计真正做出来，而不是过早退回到更简单、也更差的折中方案。

这才是软件架构师工作方式真正发生变化的地方。过去，架构师往往已经知道更干净的方向在哪里，但有限的工程产能会逼着那个愿景退回到一个更便宜的妥协方案。现在，架构师在某种意义上又重新变回了“能快速动手的人”：可以直接用代码把思路搭出来，把高层抽象落成接口，再用真实运行的实现去证明设计。

这不仅改变了实现，也改变了沟通方式。在开源里，我们常说：talk is cheap, show me the code。在 AI Coding 时代，“把代码拿出来”这件事变得容易多了。设计不再那么依赖一个缓慢的、自上而下的翻译过程：从想法到文档，再到解释，再到实现。代码可以更早出现，也可以更早跑起来。

这也让其他资深工程师受益。他们不必只靠图、会议或长篇解释来还原整个设计，而是可以直接审查抽象、阅读真实代码、运行它、质疑它，并在具体实现上一起打磨。这让架构协作更快、更清晰，也少了很多沟通误差。

也正因为如此，我总觉得今天很多 AI 讨论有点跑偏。很多项目确实很有趣、也很好玩，拿来体验当然没问题，但高级工程工作并不会因为“给代码库接了个 agent”就自然变好。真正重要的，不是哪个 demo 看起来最炫，而是哪些工程能力真的被放大了，同时软件开发本身的纪律有没有被保留下来。

对于架构师和资深工程师来说，这里真正重要的能力包括：

快速做对比式原型验证：不是只用 slides 和文档去论证某个想法，而是直接把多个方案做成可运行代码来比较。
大规模代码理解能力：能在大量模块之间快速阅读，同时保持对整个系统的全局认识。
系统性的重构能力：把基于反射、依赖运行时动态行为的路径，系统性地改造成适配 AOT 约束的设计。
搭建自动化的能力：当一个迁移步骤在每次上游同步时都必须重做一次，靠手工处理本身就很费时费力，而且越往后只会越累。AI 让我们真正有条件去投资生成器、清单、一致性检查和漂移检测，把重复的人力劳动变成可重复的自动化流程。
大范围审查能力：在很大的代码面上检查边界条件、兼容性约束，以及方案是否经得起反复执行。

这些能力也都体现在最终的设计结果里。same-FQCN 替换为 GraalVM 特定行为建立了清晰、受控的边界；反射元数据不再依赖手工维护的猜测清单，而是直接从构建产物中生成；各种清单机制和漂移检测，则把原本模糊的“上游同步风险”变成了显式的工程工作流。

对于初级工程师，我觉得这里的启发同样重要。AI 不会让架构设计、系统约束、接口设计、测试和可维护性这些基本功变得不重要。恰恰相反，这些能力只会变得更重要，因为它们决定了“被加速的实现”最终产出的是一个可持续演进的系统，还是只是更快地制造出更多代码。真正的杠杆来自工程判断力，而不是新鲜感。

Claude Code 和 Gemini AI 在整个过程中都扮演了工程加速器的角色。在 GraalVM Distro 这个项目里，它们具体帮我们做了几件事：

把迁移思路直接做成可运行代码：不是争论哪个方向可能行得通，而是把多个真实原型做出来、跑起来、比较掉，把不成立的方向淘汰掉。
重构重反射、重动态的代码路径：把不适合运行时的模式系统性替换成 AOT 友好的实现方式。
让上游同步真正可持续：每次 distro 从上游 SkyWalking 拉取变更后，元数据扫描、配置再生成和重新编译都必须再来一次。AI 帮助我们把这些过程做成流水线，使每次同步都变成一个可控、且大部分自动化的过程，而不是一次比一次更长的手工重复劳动。
在大范围内审查逻辑和边界情况：特别是在功能对等性比纯实现速度更重要的地方。

最终产出的，不只是一次大重写，而是一套可重复的系统：预编译器、manifest 驱动的加载、反射配置生成、替换边界，以及让上游迁移可审查、可自动化的漂移检测机制。

如果你想看这种开发方法背后的更广泛背景，可以读这篇文章：在成熟开源大型项目中实践 Agentic Vibe Coding：软件工程与工程控制论还在延续。这篇文章则是这个故事的下一步：不仅是在一个成熟代码库里增强功能，而是重新激活一项曾经停滞的工作，并把它真正做成可运行系统。

真正改变的到底是什么

这个项目最重要的结果，并不是一张 benchmark 表。基准数据当然属于 distro 本身，而且它们很重要，因为它们证明这套系统是真实可运行的。但对这篇文章来说，更深层的变化发生在方法论层面：AI Coding 改变了我们探索、验证和打磨架构方案的方式。

过去，架构往往更像一项以文档为主、后面拖着漫长而昂贵实现过程的活动。现在，我们可以更快地在想法、原型、比较和重设计之间切换。这让我们真正有机会去追求更高抽象层次的方案，保留更干净的边界，并建设那些让迁移过程可持续维护的自动化机制。

这项工作的技术证据，就是 SkyWalking GraalVM Distro 本身：它不仅是一个可运行的系统，更是一条由预编译器、自动生成的反射元数据、受控替换边界和漂移检查组成的迁移流水线。基准数据之所以重要，是因为它们证明这套系统在实践里是成立的；但从架构角度看，真正的结果是：这次迁移不再是一场一次性的移植，而是变成了一套可重复执行的系统工程。关于完整测试方法、原始数据和技术设计，请阅读配套文章：SkyWalking GraalVM Distro：设计与基准测试。

项目仓库位于 apache/skywalking-graalvm-distro。我们欢迎社区成员测试这个新发行版、提交 issue，并帮助它逐步走向生产可用。

对我来说，更深层的启发并不止于这个发行版。AI Coding 不会让架构变得不重要，反而会让架构更值得被认真追求。当实现速度提升到一定程度时，我们终于有机会在真实代码里验证更多想法，保留那些真正好的抽象，并把那些过去常常因为投入太大而半途妥协的系统真正做出来。

对于资深工程师来说，瓶颈正在从单纯的代码实现速度，转向品味、系统判断力，以及定义稳定边界的能力。对于初级工程师来说，真正该走的路不是追逐每一种看上去都很刺激的 AI 工作流，而是把基础能力练得更扎实，让加速真正产生复利：理解需求、阅读陌生系统、质疑假设，并识别出在系统快速变化时仍然必须保持正确的那些部分。AI Coding 降低了验证好设计的代价，但并没有降低工程判断本身的门槛。

Zh: SkyWalking GraalVM Distro：设计与基准测试

Sun, 15 Mar 2026 00:00:00 +0000

这篇文章会完整介绍我们如何把 Apache SkyWalking OAP 迁移到 GraalVM Native Image。目标不是做一次性移植，而是把这件事做成一套能持续跟上上游演进的流程。

如果你想看这项工作的更大背景，以及 AI Coding 如何让这个项目真正做得出来，请阅读：AI Coding 如何重塑软件架构师的工作方式。

为什么 GraalVM 在这里是刚需

GraalVM Native Image 可以把 Java 应用做 Ahead-of-Time（AOT）编译，生成独立可执行文件。对于像 SkyWalking OAP 这样的可观测性后端来说，这不是“锦上添花”的性能优化，而是明确的工程刚需。

可观测性平台必须是基础设施中最可靠的部分。它必须在自己要观测的那些故障发生时依然存活。在云原生环境里，工作负载会不断扩缩容、迁移和重启，负责观测一切的后端本身不能还是那个启动慢、空闲占用大、恢复缓慢的重型进程。

我们的基准测试结果让这个结论变得非常具体：

**启动时间：**约 5 ms 对比约 635 ms。在 Kubernetes 集群里，当 OAP Pod 被驱逐或重新调度时，635 ms 的差距意味着这段时间里的遥测数据可能会丢失。5 ms 的情况下，新 Pod 往往在大部分客户端还没感知到中断之前就已经重新开始接收数据了。
**空闲内存：**约 41 MiB 对比约 1.2 GiB。可观测性后端是 24/7 常驻运行的。在多租户或边缘部署场景里，基础 RSS 降了 97%，可以放进更小的节点，而不再必须占用一台专用机器。
**负载下内存：**在 20 RPS 下约 629 MiB 对比约 2.0 GiB。生产级负载下内存降了 70%，直接对应更少的节点、更低的云账单，以及在后端本身成为扩容瓶颈之前更多的余量。
**没有预热惩罚：**峰值吞吐可以更早发挥出来。JVM 的 JIT 编译器往往需要数分钟流量才能完成热点优化，在这段时间里，尾延迟更差，数据处理也会滞后。原生二进制没有同样的阶段。
**更小的攻击面：**不再需要完整 JDK 运行时，需要跟踪和修补的 CVE 也就少了很多。对于一个会接收整个集群所有服务数据的组件来说，这一点很重要。

这些都不是“小修小补”。它们直接改变了哪些部署形态开始变得可行：无服务器形态的可观测性后端、边车式采集模型、内存预算极其紧张的边缘节点。只有当后端足够轻、足够快时，这些方案才真正有落地空间。

挑战：一个成熟、动态特性很多的 Java 平台

SkyWalking OAP 身上有大型 Java 平台的所有典型问题：运行时字节码生成、重反射初始化、classpath 扫描、基于 SPI 的模块装配，以及动态 DSL 执行。这些机制方便扩展，但做 GraalVM native image 时全是障碍。

GraalVM 文档中列出的限制，只是问题的开始。在一个成熟的 OSS 平台里，这些限制会深深缠绕在多年积累下来的运行时设计决策中。常规的 GraalVM native image 很难处理运行时类生成、反射、动态发现和脚本执行，而这些在 SkyWalking OAP 中都不是零散存在的，它们本来就是系统设计的一部分。

在这个发行版的早期历史里，还有一座非常具体的大山。那时上游 SkyWalking 仍然高度依赖 Groovy 来处理 LAL、MAL 和 Hierarchy 脚本。理论上，它只是另一个“不支持运行时动态”的组件；但在实践里，Groovy 是整条路径上最大的障碍。它不仅仅是脚本执行问题，而是代表着一整套在 JVM 世界里极其便利、在 native image 世界里极其不友好的动态模型。

设计目标：让迁移这件事可以重复做

设计目标不是”把 native-image 跑通一次就完”，而是做出一套能反复用、能长期维护的迁移系统：

把运行时生成的产物前移到构建期。 OAL、MAL、LAL、Hierarchy 规则，以及 meter 相关的生成类，都在构建期完成编译并打包，而不是等到启动时才动态生成。
用确定性的加载机制替代动态发现。 classpath 扫描和运行时注册路径被转换为基于 manifest 的加载方式。
减少运行时反射，并在构建期生成 native 元数据。 反射配置不再依赖人工维护的猜测清单，而是根据真实 manifest 和扫描结果生成。
让上游同步边界保持清晰。 same-FQCN replacements 会被显式打包、列清单，并通过陈旧性检查守住边界。
让变化第一时间暴露出来。 一旦上游 provider、规则文件或被替换的源文件发生变化，测试就会失败，迫使我们做显式审查。

这才是最关键的架构转变。好的抽象和前瞻性，在 AI 时代并没有变得不重要，反而变得更重要了，因为它们决定了 AI 带来的速度，最终产出的是一个可维护的系统，还是一堆膨胀得更快的代码。

把运行时动态行为变成构建期产物

SkyWalking OAP 里有多个在 JVM 世界里很自然、但在 native image 里很棘手的动态子系统：

OAL 会在运行时生成类。
LAL、MAL 和 Hierarchy 在历史上与大量基于 Groovy 的运行时行为绑定在一起，这也是早期 distro 工作中最难处理的阻碍之一。
MAL、LAL 和 Hierarchy 规则依赖运行时编译行为。
基于 Guava 的 classpath 扫描会发现注解、dispatcher、decorator 和 meter function。
基于 SPI 的模块和 provider 发现依赖更动态的运行时环境。
YAML/config 初始化和框架集成依赖反射访问。

在 SkyWalking GraalVM Distro 里，这些问题不是靠零散补丁一个个修掉的，而是被统一收敛到一条构建期流水线里。

预编译器会在构建过程中运行 DSL 引擎、导出生成类、写入 manifest、序列化配置数据，并生成 native-image 元数据。这样一来，启动时只需要做类加载和注册，不再需要运行时代码生成。运行期之所以能变得更简单，是因为原本的复杂性被前移到了构建期。

这也是为什么这个项目不只是一次性能优化。我们的设计目标，是把复杂性前移到一个更容易验证、更容易自动化、也更便于反复执行的位置。

same-FQCN 替换：一条可控的边界

这个发行版里最实用的设计选择之一，就是使用 same-FQCN 替换类。我们没有依赖模糊的启动技巧，也没有依赖未文档化的加载顺序假设。相反，我们会重新打包 GraalVM 特定 jar，排除原本的上游类，再让替换类占据完全相同的 fully-qualified class name。

这对可维护性非常关键，因为它建立了一条非常清晰的边界：

上游类仍然定义行为契约；
GraalVM 侧的替换类提供兼容的实现策略；
打包过程则让这次替换变得显式可见。

例如，OAL 的加载过程从运行时编译变成了基于 manifest 的预编译类加载。类似的替换也处理了 MAL 和 LAL DSL 加载、模块装配、配置初始化，以及多个对反射敏感的路径。目标不是把一切都 fork 出去，而是只替换那些运行时模型从根本上不适合 native image 的部分。

随后，这条边界还会通过测试来守护：测试会对照与 replacement 对应的上游源文件做哈希。当上游改动了这些文件中的任何一个，构建就会失败，并明确告诉我们哪个 replacement 需要重新审查。这样一来，“如何跟上上游”就不再是一个充满焦虑的抽象问题，而变成一项明确、可落地的工程工作。

反射配置不是猜出来的，而是生成出来的

在很多 GraalVM 迁移项目里，reflect-config.json 最终会变成一个靠经验不断累积的工件。它会越来越大，越来越陈旧，最后没有人真正清楚它是不是完整，也不清楚每一项配置为什么存在。这种模式在一个持续演化的大型 OSS 平台里是无法扩展的。

在这个发行版里，反射元数据直接从构建产物和扫描结果中生成，包括：

OAL、MAL、LAL、Hierarchy 以及 meter 生成类的 manifest；
注解扫描得到的类；
Armeria HTTP handler；
GraphQL resolver 和 schema 映射类型；
被接受的 ModuleConfig 类。

这是一种健康得多的模式。我们不再依赖人去记住所有可能触发反射访问的路径，而是让系统根据真实迁移流水线推导出反射元数据。构建过程本身，成为了事实来源。

让上游同步变得现实可行

如果这个发行版只是一次性的工程冲刺，那它的意义会小很多。真正困难的事情，是在上游 SkyWalking 继续演进的同时，让它还能持续维护下去。

这也是为什么仓库里会有一整套显式的清单和漂移检测机制：

provider 清单，用来强制新上游 provider 被分类；
规则文件清单，用来强制新 DSL 输入被显式确认；
预编译 YAML 输入的 SHA watcher；
带 GraalVM 特定 replacement 的上游源文件 SHA watcher。

好的抽象不仅仅是代码结构优雅，更在于你是否选择了一种能在未来变化面前继续成立的迁移设计。

基准测试结果

我们在一台 Apple M3 Max（macOS、Docker Desktop、10 CPUs / 62.7 GB）上，对标准 JVM OAP 和 GraalVM Distro 做了对比测试，两者都连接到 BanyanDB。

启动测试（Docker Compose，无流量，3 次取中位数）

指标	JVM OAP	GraalVM OAP	差异
冷启动时间	635 ms	5 ms	约快 127 倍
热启动时间	630 ms	5 ms	约快 126 倍
空闲 RSS	约 1.2 GiB	约 41 MiB	约降低 97%

启动时间的测量方式，是从 OAP 第一条应用日志时间戳开始，到出现 listening on 11800 日志（即 gRPC 服务 ready）为止。

持续负载下（Kind + Istio 1.25.2 + Bookinfo，约 20 RPS，2 个 OAP 副本）

在 60 秒预热之后，每 10 秒采样一次，共 30 个样本。

指标	JVM OAP	GraalVM OAP	差异
CPU 中位数（millicores）	101	68	-33%
CPU 平均值（millicores）	107	67	-37%
内存中位数（MiB）	2068	629	-70%
内存平均值（MiB）	2082	624	-70%

两个版本报告的 entry-service CPM 一致，说明在这个测试负载下，两者的流量处理能力相同。

我们每 30 秒通过 swctl 对所有已发现服务收集这些指标： service_cpm、service_resp_time、service_sla、service_apdex、service_percentile。

完整的基准测试脚本和原始数据位于发行版仓库中的 benchmark/ 目录。

当前状态

这个项目已经是一个可运行的实验性发行版，托管在独立仓库中：apache/skywalking-graalvm-distro。

当前发行版有意聚焦在一种现代、高性能的运行模式上：

存储： BanyanDB
集群模式： Standalone 和 Kubernetes
配置方式： 无配置或 Kubernetes ConfigMap
运行模型： 固定模块集合、预编译产物和 AOT 友好的装配方式

这种聚焦是刻意的。要把迁移做成一套可重复的系统，第一步必须先把边界收清楚，做出一个真正能跑起来的版本，然后再在不失控的前提下逐步扩展。

快速开始

由于 SkyWalking GraalVM Distro 的设计目标就是追求极致性能，它目前最适合与 BanyanDB 存储后端搭配使用。当前发布的镜像已经可以在 Docker Hub 获取，你可以直接用下面这个 docker-compose.yml 启动整套系统。

version: '3.8'

services:
  banyandb:
    image: ghcr.io/apache/skywalking-banyandb:e1ba421bd624727760c7a69c84c6fe55878fb526
    container_name: banyandb
    restart: always
    ports:
      - "17912:17912"
      - "17913:17913"
    command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data --measure-metadata-cache-wait-duration 1m --stream-metadata-cache-wait-duration 1m
    healthcheck:
      test: ["CMD", "sh", "-c", "nc -nz 127.0.0.1 17912"]
      interval: 5s
      timeout: 10s
      retries: 120

  oap:
    image: apache/skywalking-graalvm-distro:0.1.1
    container_name: oap
    depends_on:
      banyandb:
        condition: service_healthy
    restart: always
    ports:
      - "11800:11800"
      - "12800:12800"
    environment:
      SW_STORAGE: banyandb
      SW_STORAGE_BANYANDB_TARGETS: banyandb:17912
      SW_HEALTH_CHECKER: default
    healthcheck:
      test: ["CMD-SHELL", "nc -nz 127.0.0.1 11800 || exit 1"]
      interval: 5s
      timeout: 10s
      retries: 120

  ui:
    image: ghcr.io/apache/skywalking/ui:10.3.0
    container_name: ui
    depends_on:
      oap:
        condition: service_healthy
    restart: always
    ports:
      - "8080:8080"
    environment:
      SW_OAP_ADDRESS: http://oap:12800

只需要执行：

docker compose up -d

欢迎社区来测试这个新发行版、提交 issue，并帮助我们推动它走向生产可用。

特别感谢 GraalVM 团队提供的技术基础。

Blog: How AI Changed the Economics of Architecture

Fri, 13 Mar 2026 00:00:00 +0000

SkyWalking GraalVM Distro: A case study in turning runnable PoCs into a repeatable migration pipeline.

The most important lesson from this project is not that AI can generate a large amount of code. It is that AI changes the economics of architecture. When runnable PoCs become cheap to build, compare, discard, and rebuild, architects can push further toward the design they actually want instead of stopping early at a compromise they can afford to implement.

That shift matters a lot in mature open source systems. Apache SkyWalking OAP has long been a powerful and production-proven observability backend, but it also carries all the realities of a large Java platform: runtime bytecode generation, reflection-heavy initialization, classpath scanning, SPI-based module wiring, and dynamic DSL execution that are friendly to extensibility but hostile to GraalVM native image.

SkyWalking GraalVM Distro is the result of treating that challenge as a design-system problem instead of a one-off porting exercise. The goal was not only to make OAP run as a native binary, but to turn GraalVM migration itself into a repeatable automation pipeline that can stay aligned with upstream evolution.

For the full technical design, benchmark data, and getting-started guide, see the companion post: SkyWalking GraalVM Distro: Design and Benchmarks.

From Paused Idea to Runnable System

This journey actually began years ago. Shortly after this repository was created, yswdqz spent several months exploring the transition. The project proved much harder in practice than the individual GraalVM limitations sounded on paper, and the work eventually paused for years.

That pause is important. The missing ingredient was not ideas. Mature maintainers usually have more ideas than time. The real constraint was implementation economics. Even when the architect can see several promising directions, limited developer resources force an earlier trade-off: choose the path that is cheapest to implement, not necessarily the path that is cleanest, most reusable, or most future-proof.

This is a very common reality, not an exceptional one. In open source communities, much of the work depends on volunteers or limited company sponsorship. In commercial products, the pressure is different but the constraint is still real: roadmap commitments, staffing limits, and delivery deadlines keep engineering resources tight. In both worlds, good ideas are often abandoned not because they are wrong, but because they are too expensive to validate and implement thoroughly.

There is another constraint that matters just as much: the architect is usually also a very senior engineer, not a full-time implementation machine. That means limited personal coding energy, fragmented time, and a constant need to explain ideas to other senior engineers before the code exists. Traditionally, that explanation happens through diagrams, documents, and conversations. It is slow, lossy, and unpredictable. We all know some version of the Telephone Game: even simple words are easy to misunderstand, and by the time the misunderstanding becomes visible, a lot of time has already passed.

What changed in late 2025 was that AI engineering made multiple runnable ideas affordable. Instead of picking an early compromise because implementation capacity was scarce, we could switch repeatedly between designs, validate them with code, discard weak directions quickly, and keep iterating until the architecture became solid, practical, and efficient enough to hold.

That design freedom was critical. GraalVM documentation gives clear guidance on isolated limitations, but a mature OSS platform hits them as a connected system. Fixing only one dynamic mechanism is not enough. To make native image practical, we had to turn whole categories of runtime behavior into build-time artifacts and automated metadata generation.

There was also a very concrete mountain in front of us in the early history of this distro. In the first several commits of the repository, upstream SkyWalking still relied heavily on Groovy for LAL, MAL, and Hierarchy scripts. In theory, that was just one more unsupported runtime-heavy component. In practice, Groovy was the biggest obstacle in the whole path. It represented not only script execution, but a whole dynamic model that was deeply convenient on the JVM side and deeply unfriendly to native image.

To bridge the gap, we re-architected the core engines of OAP around an AOT-first model. Earlier experiments had to confront Groovy-era runtime behavior directly and explore alternative script-compilation approaches to get around it. The finalized direction went further: align with the upstream compiler pipeline, move dynamic generation to build time, and add automation so the migration stays controllable as upstream keeps moving. Concretely, that meant turning OAL, MAL, LAL, and Hierarchy generation into build-time precompiler outputs instead of leaving them as startup-time dynamic behavior.

AI Speed Changed the Design Loop

The scale of this transformation was not only about coding faster. AI changed the loop between idea, prototype, validation, and redesign. We could build runnable PoCs for different approaches, throw away weak ones quickly, and preserve the promising abstractions until they formed a coherent migration system.

That does not reduce the role of human architecture. It raises the value of it. Human judgment was still required to decide what should become build-time, what should stay configurable, where to introduce same-FQCN replacements, how to keep upstream sync controllable, and which abstractions were worth preserving. But AI speed made it realistic to pursue those better designs instead of settling for a simpler compromise too early.

This is the real change in the economics of architecture. In the past, an architect might already know the cleaner direction, but limited engineering capacity often forced that vision back toward a cheaper compromise. Now the architect can return much closer to being a fast developer again: building code, shaping high-abstraction interfaces, and using design patterns to prove the vision directly in the real world.

That changes communication as much as implementation. In open source, we often say, talk is cheap, show me the code. With AI engineering, showing the code becomes much more straightforward. The design no longer depends so heavily on a slow top-down translation from idea to documents to interpretation to implementation. The code can appear earlier, and it can run earlier.

Other senior engineers benefit from this too. They do not need to reconstruct the whole design only from diagrams, meetings, or long explanations. They can review the actual abstraction, see the behavior in code, run it, challenge it, and refine it from something concrete. That makes architectural collaboration faster, clearer, and less lossy.

This is also where I think the current AI discussion is often noisy. Many projects are fun, surprising, and worth exploring, but advanced engineering work is not improved merely by attaching an agent to a codebase. The important question is not which demo looks most magical. The important question is which engineering capabilities are actually being accelerated without losing the discipline of software development itself.

For architects and senior engineers, the capabilities that mattered most here were:

Fast comparative prototyping: Building several runnable approaches in code instead of defending one idea with slides and documents.
Large-scale code comprehension: Reading across many modules quickly enough to keep the whole system in view.
Systematic refactoring: Converting reflection-heavy or runtime-dynamic paths into designs that fit AOT constraints.
Automation construction: When a migration step must be repeated every upstream sync, doing it manually once is already expensive. Doing it manually again next time is even more expensive. AI made it practical to invest in generators, inventories, consistency checks, and drift detectors that turn repeated manual work into repeatable automation.
Review at breadth: Checking edge cases, compatibility boundaries, and repeatability across a large surface area.

Those capabilities were visible in the resulting design. Same-FQCN replacements created a controlled boundary for GraalVM-specific behavior. Reflection metadata was generated from build outputs instead of maintained as a hand-written guess list. Inventories and drift detectors turned upstream sync from a vague maintenance risk into an explicit engineering workflow.

For junior engineers, I think the lesson is equally important. AI does not remove the need to learn architecture, invariants, interfaces, testing, or maintenance. It makes those skills more valuable, because they determine whether accelerated implementation produces a durable system or just more code faster. The leverage comes from engineering judgment, not from novelty.

Claude Code and Gemini AI acted as engineering accelerators throughout this process. In the GraalVM Distro specifically, they helped us:

Explore migration strategies as running code: Instead of debating which approach might work, we built and compared multiple real prototypes, discarded the weak ones, and kept what held up.
Refactor reflection-heavy and dynamic code paths: Replace runtime-hostile patterns with AOT-friendly alternatives across the codebase.
Make upstream sync sustainable: Every time the distro pulls from upstream SkyWalking, metadata scanning, config regeneration, and recompilation must happen again. AI helped build the pipeline so that each sync is a controlled, largely automated process rather than a fresh manual effort that grows longer each time.
Review logic and edge cases at scale: Especially in places where feature parity mattered more than raw implementation speed.

The result was not just a large rewrite. It was a repeatable system: precompilers, manifest-driven loading, reflection-config generation, replacement boundaries, and drift detectors that make upstream migration reviewable and automatable.

For the broader methodology behind this style of development, see Agentic Vibe Coding in a Mature OSS Project. This post is the next step in that story: not only enhancing an active mature codebase, but reviving a paused effort and making it actually runnable.

What Actually Changed

The most important outcome of this project is not a benchmark table. The benchmark results belong to the distro itself, and they matter because they prove the system is real. But for this post, the deeper result is methodological: AI engineering changed how architecture could be explored, validated, and refined.

Instead of treating architecture as a mostly document-driven activity followed by a long and expensive implementation phase, we were able to move much faster between idea, prototype, comparison, and redesign. That made it realistic to pursue higher-abstraction solutions, preserve cleaner boundaries, and build the automation needed to keep the migration maintainable over time.

The technical evidence for that work is the SkyWalking GraalVM Distro itself: not only a runnable system, but a migration pipeline expressed as precompilers, generated reflection metadata, controlled replacement boundaries, and drift checks. The benchmark data matter because they prove the system works in practice, but the architectural result is that the migration became a repeatable system rather than a one-time port. For detailed benchmark methodology, per-pod data, and the full technical design, see SkyWalking GraalVM Distro: Design and Benchmarks.

The project is hosted at apache/skywalking-graalvm-distro. We invite the community to test it, report issues, and help move it toward production readiness.

For me, the deeper takeaway is broader than this distro. AI engineering does not make architecture less important. It makes architecture more worth pursuing. When implementation speed rises enough, we can afford to test more ideas in code, keep the good abstractions, and build systems that would previously have been judged too expensive to finish well.

For senior engineers, that means the bottleneck shifts away from raw typing speed and toward taste, system judgment, and the ability to define stable boundaries. For junior engineers, it means the path forward is not to chase every exciting AI workflow, but to become stronger at the fundamentals that let acceleration compound: understanding requirements, reading unfamiliar systems, questioning assumptions, and recognizing what must remain correct as everything around it changes. AI changed the economics of architecture because it lowered the cost of validating better designs without lowering the bar for engineering judgment.

Blog: SkyWalking GraalVM Distro: Design and Benchmarks

Fri, 13 Mar 2026 00:00:00 +0000

A technical deep-dive into how we migrated Apache SkyWalking OAP to GraalVM Native Image — not as a one-off port, but as a repeatable pipeline that stays aligned with upstream.

For the broader story of how AI engineering made this project economically viable, see How AI Changed the Economics of Architecture.

Why GraalVM Is Not Optional

GraalVM Native Image compiles Java applications Ahead-of-Time (AOT) into standalone executables. For an observability backend like SkyWalking OAP, this is not a performance optimization — it is an operational necessity.

An observability platform must be the most reliable component in the infrastructure. It has to survive the failures it is supposed to observe. In cloud-native environments where workloads scale, migrate, and restart constantly, the backend that watches everything cannot itself be the slow, heavy process that takes seconds to recover and gigabytes to idle.

Our benchmarks make the case concrete:

Startup: ~5 ms vs ~635 ms. In a Kubernetes cluster where an OAP pod gets evicted or rescheduled, a 635 ms gap means lost telemetry — traces, metrics, and logs that arrive during that window are simply dropped. At 5 ms, the new pod is receiving data before most clients even notice the disruption.
Idle memory: ~41 MiB vs ~1.2 GiB. Observability backends run 24/7. In a multi-tenant or edge deployment, a 97% reduction in baseline RSS is the difference between fitting the observability stack on a small node and needing a dedicated one.
Memory under load: ~629 MiB vs ~2.0 GiB at 20 RPS. A 70% reduction at production-like traffic means fewer nodes, lower cloud bills, and more headroom before the backend itself becomes a scaling bottleneck.
No warm-up penalty: Peak throughput is available from the first request. The JVM’s JIT compiler needs minutes of traffic before it optimizes hot paths — during that window, tail latency is worse and data processing lags behind. A native binary has no such phase.
Smaller attack surface: No JDK runtime means fewer CVEs to track and patch. For a component that ingests data from every service in the cluster, that matters.

These are not incremental improvements. They change what deployment topologies are practical. Serverless observability backends, sidecar-model collectors, edge nodes with tight memory budgets — all become realistic when the backend is this light and this fast.

The Challenge: A Mature, Dynamic Java Platform

SkyWalking OAP carries all the realities of a large Java platform: runtime bytecode generation, reflection-heavy initialization, classpath scanning, SPI-based module wiring, and dynamic DSL execution. These patterns are friendly to extensibility but hostile to GraalVM native image.

The documented GraalVM limitations are only the beginning. In a mature OSS platform, those limitations are deeply entangled with years of runtime design decisions. Standard GraalVM native images struggle with runtime class generation, reflection, dynamic discovery, and script execution — all of which had deep roots in SkyWalking OAP.

There was also a very concrete mountain in the early history of this distro. Upstream SkyWalking relied heavily on Groovy for LAL, MAL, and Hierarchy scripts. In theory, that was just one more unsupported runtime-heavy component. In practice, Groovy was the biggest obstacle in the whole path. It represented not only script execution, but a whole dynamic model that was deeply convenient on the JVM side and deeply unfriendly to native image.

The Design Goal: Make Migration Repeatable

The final design is not just “run native-image successfully.” It is a system that keeps migration work repeatable:

Pre-compile runtime-generated assets at build time. OAL, MAL, LAL, Hierarchy rules, and meter-related generated classes are compiled during the build and packaged as artifacts instead of being generated at startup.
Replace dynamic discovery with deterministic loading. Classpath scanning and runtime registration paths are converted into manifest-driven loading.
Reduce runtime reflection and generate native metadata from the build. Reflection configuration is produced from actual manifests and scanned classes instead of being maintained as a hand-written guess list.
Keep the upstream sync boundary explicit. Same-FQCN replacements are intentionally packaged, inventoried, and guarded with staleness checks.
Make drift visible immediately. If upstream providers, rule files, or replaced source files change, tests fail and force explicit review.

That is the architectural shift that matters most. Reusable abstraction and foresight did not become less important in the AI era. They became more important, because they determine whether AI speed produces a maintainable system or just a fast-growing pile of code.

Turning Runtime Dynamism into Build-Time Assets

SkyWalking OAP has several dynamic subsystems that are natural in a JVM world but problematic for native image:

OAL generates classes at runtime.
LAL, MAL, and Hierarchy were historically tied to Groovy-heavy runtime behavior, which became one of the biggest practical blockers in the early distro work.
MAL, LAL, and Hierarchy rules depend on runtime compilation behavior.
Guava-based classpath scanning discovers annotations, dispatchers, decorators, and meter functions.
SPI-based module/provider discovery expects a more dynamic runtime environment.
YAML/config initialization and framework integrations depend on reflective access.

In SkyWalking GraalVM Distro, these are not solved one by one as isolated patches. They are pulled into a build-time pipeline.

The precompiler runs the DSL engines during the build, exports generated classes, writes manifests, serializes config data, and generates native-image metadata. That means startup becomes class loading and registration, not runtime code generation. The runtime path is simpler because the build path became richer.

This is also why the project is more than a performance exercise. The design goal was to move complexity into a place where it is easier to verify, easier to automate, and easier to repeat.

Same-FQCN Replacements as a Controlled Boundary

One of the most practical design choices in this distro is the use of same-FQCN replacement classes. We do not rely on vague startup tricks or undocumented ordering assumptions. Instead, the GraalVM-specific jars are repackaged so the original upstream classes are excluded and the replacement classes occupy the exact same fully-qualified names.

This matters for maintainability. It creates a very clear boundary:

the upstream class still defines the behavior contract,
the GraalVM replacement provides a compatible implementation strategy,
and the packaging makes that swap explicit.

For example, OAL loading changes from runtime compilation into manifest-driven loading of precompiled classes. Similar replacements handle MAL and LAL DSL loading, module wiring, config initialization, and several reflection-sensitive paths. The goal is not to fork everything. The goal is to replace only the places where the runtime model is fundamentally unfriendly to native image.

That boundary is then guarded by tests that hash the upstream source files corresponding to the replacements. When upstream changes one of those files, the build fails and tells us exactly which replacement needs review. This is what turns “keeping up with upstream” from an anxiety problem into a visible engineering task.

Reflection Config Is Generated, Not Guessed

In many GraalVM migrations, reflect-config.json becomes a manually accumulated artifact. It grows over time, gets stale, and nobody is fully sure whether it is complete or why each entry exists. That approach does not scale well for a large, evolving OSS platform.

In this distro, reflection metadata is generated from the build outputs and scanned classes:

manifests for OAL, MAL, LAL, Hierarchy, and meter-generated classes,
annotation-scanned classes,
Armeria HTTP handlers,
GraphQL resolvers and schema-mapped types,
and accepted ModuleConfig classes.

This is a much healthier model. Instead of asking people to remember every reflective access path, the system derives reflection metadata from the actual migration pipeline. The build becomes the source of truth.

Keeping Upstream Sync Practical

If this distro were only a one-time engineering sprint, it would be much less interesting. The real challenge is keeping it alive while upstream SkyWalking continues to evolve.

That is why the repo includes explicit inventories and drift detectors:

provider inventories that force new upstream providers to be categorized,
rule-file inventories that force new DSL inputs to be acknowledged,
SHA watchers for precompiled YAML inputs,
and SHA watchers for upstream source files with GraalVM-specific replacements.

Good abstraction is not only about elegant code structure. It is about choosing a migration design that can survive contact with future change.

Benchmark Results

We benchmarked the standard JVM OAP against the GraalVM Distro on an Apple M3 Max (macOS, Docker Desktop, 10 CPUs / 62.7 GB), both connecting to BanyanDB.

Boot Test (Docker Compose, no traffic, median of 3 runs)

Metric	JVM OAP	GraalVM OAP	Delta
Cold boot startup	635 ms	5 ms	~127x faster
Warm boot startup	630 ms	5 ms	~126x faster
Idle RSS	~1.2 GiB	~41 MiB	~97% reduction

Boot time is measured from OAP’s first application log timestamp to the listening on 11800 log line (gRPC server ready).

Under Sustained Load (Kind + Istio 1.25.2 + Bookinfo at ~20 RPS, 2 OAP replicas)

30 samples at 10s intervals after 60s warmup.

Metric	JVM OAP	GraalVM OAP	Delta
CPU median (millicores)	101	68	-33%
CPU avg (millicores)	107	67	-37%
Memory median (MiB)	2068	629	-70%
Memory avg (MiB)	2082	624	-70%

Both variants reported identical entry-service CPM, confirming equivalent traffic processing capability.

Service metrics collected every 30s via swctl for all discovered services: service_cpm, service_resp_time, service_sla, service_apdex, service_percentile.

Full benchmark scripts and raw data are in the benchmark/ directory of the distro repository.

Current Status

The project is a runnable experimental distribution, hosted in its own repository: apache/skywalking-graalvm-distro.

The current distro intentionally focuses on a modern, high-performance operating model:

Storage: BanyanDB
Cluster modes: Standalone and Kubernetes
Configuration: none or Kubernetes ConfigMap
Runtime model: fixed module set, precompiled assets, and AOT-friendly wiring

This focus is deliberate. A repeatable migration system starts by making a clear scope runnable, then expanding without losing control.

Getting Started

Because the SkyWalking GraalVM Distro is designed for peak performance, it is optimized to work with BanyanDB as its storage backend. The current published image is available on Docker Hub, and you can boot the stack using the following docker-compose.yml.

version: '3.8'

services:
  banyandb:
    image: ghcr.io/apache/skywalking-banyandb:e1ba421bd624727760c7a69c84c6fe55878fb526
    container_name: banyandb
    restart: always
    ports:
      - "17912:17912"
      - "17913:17913"
    command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data --measure-metadata-cache-wait-duration 1m --stream-metadata-cache-wait-duration 1m
    healthcheck:
      test: ["CMD", "sh", "-c", "nc -nz 127.0.0.1 17912"]
      interval: 5s
      timeout: 10s
      retries: 120

  oap:
    image: apache/skywalking-graalvm-distro:0.1.1
    container_name: oap
    depends_on:
      banyandb:
        condition: service_healthy
    restart: always
    ports:
      - "11800:11800"
      - "12800:12800"
    environment:
      SW_STORAGE: banyandb
      SW_STORAGE_BANYANDB_TARGETS: banyandb:17912
      SW_HEALTH_CHECKER: default
    healthcheck:
      test: ["CMD-SHELL", "nc -nz 127.0.0.1 11800 || exit 1"]
      interval: 5s
      timeout: 10s
      retries: 120

  ui:
    image: ghcr.io/apache/skywalking/ui:10.3.0
    container_name: ui
    depends_on:
      oap:
        condition: service_healthy
    restart: always
    ports:
      - "8080:8080"
    environment:
      SW_OAP_ADDRESS: http://oap:12800

Simply run:

docker compose up -d

We invite the community to test this new distribution, report issues, and help us move it toward a production-ready state.

Special thanks to the GraalVM team for the technology foundation.

Blog: SkyWalking 10 Release: Service Hierarchy, Kubernetes Network Monitoring by eBPF, BanyanDB, and More

Mon, 13 May 2024 00:00:00 +0000

The Apache SkyWalking team today announced the 10 release. SkyWalking 10 provides a host of groundbreaking features and enhancements. The introduction of Layer and Service Hierarchy streamlines monitoring by organizing services and metrics into distinct layers and providing seamless navigation across them. Leveraging eBPF technology, Kubernetes Network Monitoring delivers granular insights into network traffic, topology, and TCP/HTTP metrics. BanyanDB emerges as a high-performance native storage solution, while expanded monitoring support encompasses Apache RocketMQ, ClickHouse, and Apache ActiveMQ Classic. Support for Multiple Labels Names enhances flexibility in metrics analysis, while enhanced exporting and querying capabilities streamline data dissemination and processing.

This release blog briefly introduces these new features and enhancements as well as some other notable changes.

Layer and Service Hierarchy

Layer concept was introduced in SkyWalking 9.0.0, it represents an abstract framework in computer science, such as Operating System(OS_LINUX layer), Kubernetes(k8s layer). It organizes services and metrics into different layers based on their roles and responsibilities in the system. SkyWalking provides a suite of monitoring and diagnostic tools for each layer, but there is a gap between the layers, which can not easily bridge the data across different layers.

In SkyWalking 10, SkyWalking provides new abilities to jump/connect across different layers and provide a seamless monitoring experience for users.

Layer Jump

In the topology graph, users can click on a service node to jump to the dashboard of the service in another layer. The following figures show the jump from the GENERAL layer service topology to the VIRTUAL_DATABASE service layer dashboard by clicking the topology node. Figure 1: Layer Jump

Figure 2: Layer jump Dashboard

Service Hierarchy

SkyWalking 10 introduces a new concept called Service Hierarchy, which defines the relationships of existing logically same services in various layers. OAP will detect the services from different layers, and try to build the connections. Users can click the Hierarchy Services in any layer’s service topology node or service dashboard to get the Hierarchy Topology. In this topology graph, users can see the relationships between the services in different layers and the summary of the metrics and also can jump to the service dashboard in the layer. When a service occurs performance issue, users can easily analyze the metrics from different layers and track down the root cause:

The examples of the Service Hierarchy relationships:

The application song deployed in the Kubernetes cluster with SkyWalking agent and Service Mesh at the same time. So the application song across the GENERAL, MESH, MESH_DP and K8S_SERVICE layers which could be monitored by SkyWalking, the Service Hierarchy topology as below: Figure 3: Service Hierarchy Agent With K8s Service And Mesh With K8s Service.

And can also have the Service Instance Hierarchy topology to get the single instance status across the layers as below: Figure 4: Instance Hierarchy Agent With K8s Service(Pod)
The PostgreSQL database psql deployed in the Kubernetes cluster and used by the application song. So the database psql across the VIRTUAL_DATABASE, POSTGRESQL and K8S_SERVICE layers which could be monitored by SkyWalking, the Service Hierarchy topology as below: Figure 5: Service Hierarchy Agent(Virtual Database) With Real Database And K8s Service

For more supported layers and how to detect the relationships between services in different layers please refer to the Service Hierarchy. how to configure the Service Hierarchy in SkyWalking, please refer to the Service Hierarchy Configuration section.

Monitoring Kubernetes Network Traffic by using eBPF

In the previous version, skyWalking provides Kubernetes (K8s) monitoring from kube-state-metrics and cAdvisor, which can monitor the Kubernetes cluster status and the metrics of the Kubernetes resources.

In SkyWalking 10, by leverage Apache SkyWalking Rover 0.6+, SkyWalking has the ability to monitor the Kubernetes network traffic by using eBPF, which can collect and map access logs from applications in Kubernetes environments. Through these data, SkyWalking can analyze and provide the Service Traffic, Topology, TCP/HTTP level metrics from the Kubernetes aspect.

The following figures show the Topology and TCP Dashboard of the Kubernetes network traffic:

Figure 6: Kubernetes Network Traffic Topology

Figure 7: Kubernetes Network Traffic TCP Dashboard

More details about how to monitor the Kubernetes network traffic by using eBPF, please refer to the Monitoring Kubernetes Network Traffic by using eBPF.

BanyanDB - Native APM Database

BanyanDB 0.6.0 and BanyanDB Java client 0.6.0 are released with SkyWalking 10, As a native storage solution for SkyWalking, BanyanDB is going to be SkyWalking’s next-generation storage solution. This is recommended to use for medium-scale deployments from 0.6 until 1.0.
It has shown high potential performance improvement. Less than 50% CPU usage and 50% memory usage with 40% disk volume compared to Elasticsearch in the same scale.

Apache RocketMQ Server Monitoring

Apache RocketMQ is an open-source distributed messaging and streaming platform, which is widely used in various scenarios including Internet, big data, mobile Internet, IoT, and other fields. SkyWalking provides a basic monitoring dashboard for RocketMQ, which includes the following metrics:

Cluster Metrics: including messages produced/consumed today, total producer/consumer TPS, producer/consumer message size, messages produced/consumed until yesterday, max consumer latency, max commitLog disk ratio, commitLog disk ratio, pull/send threadPool queue head wait time, topic count, and broker count.
Broker Metrics: including produce/consume TPS, producer/consumer message size.
Topic Metrics: including max producer/consumer message size, consumer latency, producer/consumer TPS, producer/consumer offset, producer/consumer message size, consumer group count, and broker count.

The following figure shows the RocketMQ Cluster Metrics dashboard: Figure 8: Apache RocketMQ Server Monitoring

For more metrics and details about the RocketMQ monitoring, please refer to the Apache RocketMQ Server Monitoring,

ClickHouse Server Monitoring

ClickHouse is an open-source column-oriented database management system that allows generating analytical data reports in real-time, it is widely used for online analytical processing (OLAP). ClickHouse monitoring provides monitoring of the metrics 、events and asynchronous metrics of the ClickHouse server, which includes the following parts of metrics:

Server Metrics
Query Metrics
Network Metrics
Insert Metrics
Replica Metrics
MergeTree Metrics
ZooKeeper Metrics
Embedded ClickHouse Keeper Metrics

The following figure shows the ClickHouse Cluster Metrics dashboard: Figure 9: ClickHouse Server Monitoring

For more metrics and details about the ClickHouse monitoring, please refer to the ClickHouse Server Monitoring, and here is a blog that can help for a quick start Monitoring ClickHouse through SkyWalking.

Apache ActiveMQ Server Monitoring

Apache ActiveMQ Classic is a popular and powerful open-source messaging and integration pattern server. SkyWalking provides a basic monitoring dashboard for ActiveMQ, which includes the following metrics:

Cluster Metrics: including memory usage, rates of write/read, and average/max duration of write.
Broker Metrics: including node state, number of connections, number of producers/consumers, and rate of write/read under the broker. Depending on the cluster mode, one cluster may include one or more brokers.
Destination Metrics: including number of producers/consumers, messages in different states, queues, and enqueue duration in a queue/topic.

The following figure shows the ActiveMQ Cluster Metrics dashboard: Figure 10: Apache ActiveMQ Server Monitoring

For more metrics and details about the ActiveMQ monitoring, please refer to the Apache ActiveMQ Server Monitoring, and here is a blog that can help for a quick start Monitoring ActiveMQ through SkyWalking.

Support Multiple Labels Names

Before SkyWalking 10, SkyWalking does not store the labels names in the metrics data, which makes MQE have to use _ as the generic label name, it can’t query the metrics data with multiple labels names.

SkyWalking 10 supports storing the labels names in the metrics data, and MQE can query or calculate the metrics data with multiple labels names. For example: The k8s_cluster_deployment_status metric has labels namespace, deployment and status. If we want to query all deployment metric values with namespace=skywalking-showcase and status=true, we can use the following expression:

k8s_cluster_deployment_status{namespace='skywalking-showcase', status='true'}

related enhancement:

Since Alarm rule configuration had migrated to the MQE in SkyWalking 9.6.0, the alarm rule also supports multiple labels names.
PromeQL service supports multiple labels names query.

Metrics gRPC exporter

SkyWalking 10 enhanced the metrics gPRC exporter, it supports exporting all types of metrics data to the gRPC server.

SkyWalking Native UI Metrics Query Switch to V3 APIs

SkyWalking Native UI metrics query deprecate the V2 APIs, and all migrated to V3 APIs and MQE.

Other Notable Enhancements

Support Java 21 runtime and oap-java21 image for Java 21 runtime.
Remove CLI(swctl) from the image.
More MQE functions and operators supported.
Enhance the native UI and improve the user experience.
Several bugs and CVEs fixed.

Blog: How to Use SkyWalking for Distributed Tracing in Istio?

Wed, 14 Dec 2022 00:00:00 +0000

In cloud native applications, a request often needs to be processed through a series of APIs or backend services, some of which are parallel and some serial and located on different platforms or nodes. How do we determine the service paths and nodes a call goes through to help us troubleshoot the problem? This is where distributed tracing comes into play.

This article covers:

How distributed tracing works
How to choose distributed tracing software
How to use distributed tracing in Istio
How to view distributed tracing data using Bookinfo and SkyWalking as examples

Distributed Tracing Basics

Distributed tracing is a method for tracing requests in a distributed system to help users better understand, control, and optimize distributed systems. There are two concepts used in distributed tracing: TraceID and SpanID. You can see them in Figure 1 below.

TraceID is a globally unique ID that identifies the trace information of a request. All traces of a request belong to the same TraceID, and the TraceID remains constant throughout the trace of the request.
SpanID is a locally unique ID that identifies a request’s trace information at a certain time. A request generates different SpanIDs at different periods, and SpanIDs are used to distinguish trace information for a request at different periods.

TraceID and SpanID are the basis of distributed tracing. They provide a uniform identifier for request tracing in distributed systems and facilitate users’ ability to query, manage, and analyze the trace information of requests.

Figure 1: Trace and span

The following is the process of distributed tracing:

When a system receives a request, the distributed tracing system assigns a TraceID to the request, which is used to chain together the entire chain of invocations.
The distributed trace system generates a SpanID and ParentID for each service call within the system for the request, which is used to record the parent-child relationship of the call; a Span without a ParentID is used as the entry point of the call chain.
TraceID and SpanID are to be passed during each service call.
When viewing a distributed trace, query the full process of a particular request by TraceID.

How Istio Implements Distributed Tracing

Istio’s distributed tracing is based on information collected by the Envoy proxy in the data plane. After a service request is intercepted by Envoy, Envoy adds tracing information as headers to the request forwarded to the destination workload. The following headers are relevant for distributed tracing:

As TraceID: x-request-id
Used to establish parent-child relationships for Span in the LightStep trace: x-ot-span-context</li
Used for Zipkin, also for Jaeger, SkyWalking, see b3-propagation:
- x-b3-traceid
- x-b3-traceid
- x-b3-spanid
- x-b3-parentspanid
- x-b3-sampled
- x-b3-flags
- b3
For Datadog:
- x-datadog-trace-id
- x-datadog-parent-id
- x-datadog-sampling-priority
For SkyWalking: sw8
For AWS X-Ray: x-amzn-trace-id

For more information on how to use these headers, please see the Envoy documentation.

Regardless of the language of your application, Envoy will generate the appropriate tracing headers for you at the Ingress Gateway and forward these headers to the upstream cluster. However, in order to utilize the distributed tracing feature, you must modify your application code to attach the tracing headers to upstream requests. Since neither the service mesh nor the application can automatically propagate these headers, you can integrate the agent for distributed tracing into the application or manually propagate these headers in the application code itself. Once the tracing headers are propagated to all upstream requests, Envoy will send the tracing data to the tracer’s back-end processing, and then you can view the tracing data in the UI.

For example, look at the code of the Productpage service in the Bookinfo application. You can see that it integrates the Jaeger client library and synchronizes the header generated by Envoy with the HTTP requests to the Details and Reviews services in the getForwardHeaders (request) function.

def getForwardHeaders(request):
    headers = {}

    # Using Jaeger agent to get the x-b3-* headers
    span = get_current_span()
    carrier = {}
    tracer.inject(
        span_context=span.context,
        format=Format.HTTP_HEADERS,
        carrier=carrier)

    headers.update(carrier)

    # Dealing with the non x-b3-* header manually
    if 'user' in session:
        headers['end-user'] = session['user']
    incoming_headers = [
        'x-request-id',
        'x-ot-span-context',
        'x-datadog-trace-id',
        'x-datadog-parent-id',
        'x-datadog-sampling-priority',
        'traceparent',
        'tracestate',
        'x-cloud-trace-context',
        'grpc-trace-bin',
        'sw8',
        'user-agent',
        'cookie',
        'authorization',
        'jwt',
    ]

    for ihdr in incoming_headers:
        val = request.headers.get(ihdr)
        if val is not None:
            headers[ihdr] = val

    return headers

For more information, the Istio documentation provides answers to frequently asked questions about distributed tracing in Istio.

How to Choose A Distributed Tracing System

Distributed tracing systems are similar in principle. There are many such systems on the market, such as Apache SkyWalking, Jaeger, Zipkin, Lightstep, Pinpoint, and so on. For our purposes here, we will choose three of them and compare them in several dimensions. Here are our inclusion criteria:

They are currently the most popular open-source distributed tracing systems.
All are based on the OpenTracing specification.
They support integration with Istio and Envoy.

Items	Apache SkyWalking	Jaeger	Zipkin
Implementations	Language-based probes, service mesh probes, eBPF agent, third-party instrumental libraries (Zipkin currently supported)	Language-based probes	Language-based probes
Database	ES, H2, MySQL, TiDB, Sharding-sphere, BanyanDB	ES, MySQL, Cassandra, Memory	ES, MySQL, Cassandra, Memory
Supported Languages	Java, Rust, PHP, NodeJS, Go, Python, C++, .Net, Lua	Java, Go, Python, NodeJS, C#, PHP, Ruby, C++	Java, Go, Python, NodeJS, C#, PHP, Ruby, C++
Initiator	Personal	Uber	Twitter
Governance	Apache Foundation	CNCF	CNCF
Version	9.3.0	1.39.0	2.23.19
Stars	20.9k	16.8k	15.8k

Although Apache SkyWalking’s agent does not support as many languages as Jaeger and Zipkin, SkyWalking’s implementation is richer and compatible with Jaeger and Zipkin trace data, and development is more active, so it is one of the best choices for building a telemetry platform.

Demo

Refer to the Istio documentation to install and configure Apache SkyWalking.

Environment Description

The following is the environment for our demo:

Kubernetes 1.24.5
Istio 1.16
SkyWalking 9.1.0

Install Istio

Before installing Istio, you can check the environment for any problems:

$ istioctl experimental precheck
✔ No issues found when checking the cluster. Istio is safe to install or upgrade!
  To get started, check out https://istio.io/latest/docs/setup/getting-started/

Then install Istio and configure the destination for sending tracing messages as SkyWalking:

# Initial Istio Operator
istioctl operator init
# Configure tracing destination
kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istio-with-skywalking
spec:
  meshConfig:
    defaultProviders:
      tracing:
      - "skywalking"
    enableTracing: true
    extensionProviders:
    - name: "skywalking"
      skywalking:
        service: tracing.istio-system.svc.cluster.local
        port: 11800
EOF

Deploy Apache SkyWalking

Istio 1.16 supports distributed tracing using Apache SkyWalking. Install SkyWalking by executing the following code:

kubectl apply -f 
https://raw.githubusercontent.com/istio/istio/release-1.16/samples/addons/extras/skywalking.yaml

It will install the following components under the istio-system namespace:

SkyWalking Observability Analysis Platform (OAP): Used to receive trace data, supports SkyWalking native data formats, Zipkin v1 and v2 and Jaeger format.
UI: Used to query distributed trace data.

For more information about SkyWalking, please refer to the SkyWalking documentation.

Deploy the Bookinfo Application

Execute the following command to install the bookinfo application:

kubectl label namespace default istio-injection=enabled
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

Launch the SkyWalking UI:

istioctl dashboard skywalking

Figure 2 shows all the services available in the bookinfo application:

Figure 2: SkyWalking General Service page

You can also see information about instances, endpoints, topology, tracing, etc. For example, Figure 3 shows the service topology of the bookinfo application:

Figure 3: Topology diagram of the Bookinfo application

Tracing views in SkyWalking can be displayed in a variety of formats, including list, tree, table, and statistics. See Figure 4:

Figure 4: SkyWalking General Service trace supports multiple display formats

To facilitate our examination, set the sampling rate of the trace to 100%:

kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
  - randomSamplingPercentage: 100.00
EOF

Important: It’s generally not good practice to set the sampling rate to 100% in a production environment. To avoid the overhead of generating too many trace logs in production, please adjust the sampling strategy (sampling percentage).

Uninstall

After experimenting, uninstall Istio and SkyWalking by executing the following command.

samples/bookinfo/platform/kube/cleanup.sh
istioctl unintall --purge
kubectl delete namespace istio-system

Understanding the Bookinfo Tracing Information

Navigate to the General Service tab in the Apache SkyWalking UI, and you can see the trace information for the most recent istio-ingressgateway service, as shown in Figure 5. Click on each span to see the details.

Figure 5: The table view shows the basic information about each span.

Switching to the list view, you can see the execution order and duration of each span, as shown in Figure 6:

Figure 6: List display

You might want to know why such a straightforward application generates so much span data. Because after we inject the Envoy proxy into the pod, every request between services will be intercepted and processed by Envoy, as shown in Figure 7:

Figure 7: Envoy intercepts requests to generate a span

The tracing process is shown in Figure 8:

Figure 8: Trace of the Bookinfo application

We give each span a label with a serial number, and the time taken is indicated in parentheses. For illustration purposes, we have summarized all spans in the table below.

No.	Endpoint	Total Duration (ms)	Component Duration (ms)	Current Service	Description
1	/productpage	190	0	istio-ingressgateway	Envoy Outbound
2	/productpage	190	1	istio-ingressgateway	Ingress -> Productpage network transmission
3	/productpage	189	1	productpage	Envoy Inbound
4	/productpage	188	21	productpage	Application internal processing
5	/details/0	8	1	productpage	Envoy Outbound
6	/details/0	7	3	productpage	Productpage -> Details network transmission
7	/details/0	4	0	details	Envoy Inbound
8	/details/0	4	4	details	Application internal processing
9	/reviews/0	159	0	productpage	Envoy Outbound
10	/reviews/0	159	14	productpage	Productpage -> Reviews network transmission
11	/reviews/0	145	1	reviews	Envoy Inbound
12	/reviews/0	144	109	reviews	Application internal processing
13	/ratings/0	35	2	reviews	Envoy Outbound
14	/ratings/0	33	16	reviews	Reviews -> Ratings network transmission
15	/ratings/0	17	1	ratings	Envoy Inbound
16	/ratings/0	16	16	ratings	Application internal processing

From the above information, it can be seen that:

The total time consumed for this request is 190 ms.
In Istio sidecar mode, each traffic flow in and out of the application container must pass through the Envoy proxy once, each time taking 0 to 2 ms.
Network requests between Pods take between 1 and 16ms.
This is because the data itself has errors and the start time of the Span is not necessarily equal to the end time of the parent Span.
We can see that the most time-consuming part is the Reviews application, which takes 109 ms so that we can optimize it for that application.

Summary

Distributed tracing is an indispensable tool for analyzing performance and troubleshooting modern distributed applications. In this tutorial, we’ve seen how, with just a few minor changes to your application code to propagate tracing headers, Istio makes distributed tracing simple to use. We’ve also reviewed Apache SkyWalking as one of the best distributed tracing systems that Istio supports. It is a fully functional platform for cloud native application analytics, with features such as metrics and log collection, alerting, Kubernetes monitoring, service mesh performance diagnosis using eBPF, and more.

If you’re new to service mesh and Kubernetes security, we have a bunch of free online courses available at Tetrate Academy that will quickly get you up to speed with Istio and Envoy.

If you’re looking for a fast way to get to production with Istio, check out Tetrate Istio Distribution (TID). TID is Tetrate’s hardened, fully upstream Istio distribution, with FIPS-verified builds and support available. It’s a great way to get started with Istio knowing you have a trusted distribution to begin with, have an expert team supporting you, and also have the option to get to FIPS compliance quickly if you need to.

Once you have Istio up and running, you will probably need simpler ways to manage and secure your services beyond what’s available in Istio, that’s where Tetrate Service Bridge comes in. You can learn more about how Tetrate Service Bridge makes service mesh more secure, manageable, and resilient here, or contact us for a quick demo.

Zh: 如何在 Istio 中使用 SkyWalking 进行分布式追踪？

Wed, 14 Dec 2022 00:00:00 +0000

在云原生应用中，一次请求往往需要经过一系列的 API 或后台服务处理才能完成，这些服务有些是并行的，有些是串行的，而且位于不同的平台或节点。那么如何确定一次调用的经过的服务路径和节点以帮助我们进行问题排查？这时候就需要使用到分布式追踪。

本文将向你介绍：

分布式追踪的原理
如何选择分布式追踪软件
在 Istio 中如何使用分布式追踪
以 Bookinfo 和 SkyWalking 为例说明如何查看分布式追踪数据

分布式追踪基础

分布式追踪是一种用来跟踪分布式系统中请求的方法，它可以帮助用户更好地理解、控制和优化分布式系统。分布式追踪中用到了两个概念：TraceID 和 SpanID。

TraceID 是一个全局唯一的 ID，用来标识一个请求的追踪信息。一个请求的所有追踪信息都属于同一个 TraceID，TraceID 在整个请求的追踪过程中都是不变的；
SpanID 是一个局部唯一的 ID，用来标识一个请求在某一时刻的追踪信息。一个请求在不同的时间段会产生不同的 SpanID，SpanID 用来区分一个请求在不同时间段的追踪信息；

TraceID 和 SpanID 是分布式追踪的基础，它们为分布式系统中请求的追踪提供了一个统一的标识，方便用户查询、管理和分析请求的追踪信息。

下面是分布式追踪的过程：

当一个系统收到请求后，分布式追踪系统会为该请求分配一个 TraceID，用于串联起整个调用链；
分布式追踪系统会为该请求在系统内的每一次服务调用生成一个 SpanID 和 ParentID，用于记录调用的父子关系，没有 ParentID 的 Span 将作为调用链的入口；
每个服务调用过程中都要传递 TraceID 和 SpanID；
在查看分布式追踪时，通过 TraceID 查询某次请求的全过程；

Istio 如何实现分布式追踪

Istio 中的分布式追踪是基于数据平面中的 Envoy 代理实现的。服务请求在被劫持到 Envoy 中后，Envoy 在转发请求时会附加大量 Header，其中与分布式追踪相关的有：

作为 TraceID：x-request-id
用于在 LightStep 追踪系统中建立 Span 的父子关系：x-ot-span-context
用于 Zipkin，同时适用于 Jaeger、SkyWalking，详见 b3-propagation：
- x-b3-traceid
- x-b3-spanid
- x-b3-parentspanid
- x-b3-sampled
- x-b3-flags
- b3
用于 Datadog：
- x-datadog-trace-id
- x-datadog-parent-id
- x-datadog-sampling-priority
用于 SkyWalking：sw8
用于 AWS X-Ray：x-amzn-trace-id

关于这些 Header 的详细用法请参考 Envoy 文档。

Envoy 会在 Ingress Gateway 中为你产生用于追踪的 Header，不论你的应用程序使用何种语言开发，Envoy 都会将这些 Header 转发到上游集群。但是，你还要对应用程序代码做一些小的修改，才能为使用分布式追踪功能。这是因为应用程序无法自动传播这些 Header，可以在程序中集成分布式追踪的 Agent，或者在代码中手动传播这些 Header。Envoy 会将追踪数据发送到 tracer 后端处理，然后就可以在 UI 中查看追踪数据了。

例如在 Bookinfo 应用中的 Productpage 服务，如果你查看它的代码可以发现，其中集成了 Jaeger 客户端库，并在 getForwardHeaders (request) 方法中将 Envoy 生成的 Header 同步给对 Details 和 Reviews 服务的 HTTP 请求：

def getForwardHeaders(request):
    headers = {}

    # 使用 Jaeger agent 获取 x-b3-* header
    span = get_current_span()
    carrier = {}
    tracer.inject(
        span_context=span.context,
        format=Format.HTTP_HEADERS,
        carrier=carrier)

    headers.update(carrier)

    # 手动处理非 x-b3-* header
    if 'user' in session:
        headers['end-user'] = session['user']
    incoming_headers = [
        'x-request-id',
        'x-ot-span-context',
        'x-datadog-trace-id',
        'x-datadog-parent-id',
        'x-datadog-sampling-priority',
        'traceparent',
        'tracestate',
        'x-cloud-trace-context',
        'grpc-trace-bin',
        'sw8',
        'user-agent',
        'cookie',
        'authorization',
        'jwt',
    ]

    for ihdr in incoming_headers:
        val = request.headers.get(ihdr)
        if val is not None:
            headers[ihdr] = val

    return headers

关于 Istio 中分布式追踪的常见问题请见 Istio 文档。

分布式追踪系统如何选择

分布式追踪系统的原理类似，市面上也有很多这样的系统，例如 Apache SkyWalking 、Jaeger 、Zipkin 、LightStep 、Pinpoint 等。我们将选择其中三个，从多个维度进行对比。之所以选择它们是因为：

它们是当前最流行的开源分布式追踪系统；
都是基于 OpenTracing 规范；
都支持与 Istio 及 Envoy 集成；

类别	Apache SkyWalking	Jaeger	Zipkin
实现方式	基于语言的探针、服务网格探针、eBPF agent、第三方指标库（当前支持 Zipkin）	基于语言的探针	基于语言的探针
数据存储	ES、H2、MySQL、TiDB、Sharding-sphere、BanyanDB	ES、MySQL、Cassandra、内存	ES、MySQL、Cassandra、内存
支持语言	Java、Rust、PHP、NodeJS、Go、Python、C++、.NET、Lua	Java、Go、Python、NodeJS、C#、PHP、Ruby、C++	Java、Go、Python、NodeJS、C#、PHP、Ruby、C++
发起者	个人	Uber	Twitter
治理方式	Apache Foundation	CNCF	CNCF
版本	9.3.0	1.39.0	2.23.19
Star 数量	20.9k	16.8k	15.8k

分布式追踪系统对比表（数据截止时间 2022-12-07）

虽然 Apache SkyWalking 的 Agent 支持的语言没有 Jaeger 和 Zipkin 多，但是 SkyWalking 的实现方式更丰富，并且与 Jaeger、Zipkin 的追踪数据兼容，开发更为活跃，且为国人开发，中文资料丰富，是构建遥测平台的最佳选择之一。

实验

参考 Istio 文档来安装和配置 Apache SkyWalking。

环境说明

以下是我们实验的环境：

Kubernetes 1.24.5
Istio 1.16
SkyWalking 9.1.0

安装 Istio

安装之前可以先检查下环境是否有问题:

$ istioctl experimental precheck
✔ No issues found when checking the cluster. Istio is safe to install or upgrade!
  To get started, check out https://istio.io/latest/docs/setup/getting-started/

然后安装 Istio 同时配置发送追踪信息的目的地为 SkyWalking：

# 初始化 Istio Operator
istioctl operator init
# 安装 Istio 并配置使用 SkyWalking
kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: istio-with-skywalking
spec:
  meshConfig:
    defaultProviders:
      tracing:
      - "skywalking"
    enableTracing: true
    extensionProviders:
    - name: "skywalking"
      skywalking:
        service: tracing.istio-system.svc.cluster.local
        port: 11800
EOF

部署 Apache SkyWalking

Istio 1.16 支持使用 Apache SkyWalking 进行分布式追踪，执行下面的代码安装 SkyWalking：

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.16/samples/addons/extras/skywalking.yaml

它将在 istio-system 命名空间下安装：

SkyWalking OAP (Observability Analysis Platform) ：用于接收追踪数据，支持 SkyWalking 原生数据格式，Zipkin v1 和 v2 以及 Jaeger 格式。
UI ：用于查询分布式追踪数据。

关于 SkyWalking 的详细信息请参考 SkyWalking 文档。

部署 Bookinfo 应用

执行下面的命令安装 bookinfo 示例：

kubectl label namespace default istio-injection=enabled
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

打开 SkyWalking UI：

istioctl dashboard skywalking

SkyWalking 的 General Service 页面展示了 bookinfo 应用中的所有服务。

你还可以看到实例、端点、拓扑、追踪等信息。例如下图展示了 bookinfo 应用的服务拓扑。

SkyWalking 的追踪视图有多种显示形式，如列表、树形、表格和统计。

SkyWalking 通用服务追踪支持多种显示样式

为了方便我们检查，将追踪的采样率设置为 100%：

kubectl apply -f - <<EOF
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
  - randomSamplingPercentage: 100.00
EOF

卸载

在实验完后，执行下面的命令卸载 Istio 和 SkyWalking：

samples/bookinfo/platform/kube/cleanup.sh
istioctl unintall --purge
kubectl delete namespace istio-system

Bookinfo demo 追踪信息说明

在 Apache SkyWalking UI 中导航到 General Service 分页，查看最近的 istio-ingressgateway 服务的追踪信息，表视图如下所示。图中展示了此次请求所有 Span 的基本信息，点击每个 Span 可以查看详细信息。

切换为列表视图，可以看到每个 Span 的执行顺序及持续时间，如下图所示。

你可能会感到困惑，为什么这么简单的一个应用会产生如此多的 Span 信息？因为我们为 Pod 注入了 Envoy 代理之后，每个服务间的请求都会被 Envoy 拦截和处理，如下图所示。

整个追踪流程如下图所示。

图中给每一个 Span 标记了序号，并在括号里注明了耗时。为了便于说明我们将所有 Span 汇总在下面的表格中。

序号	方法	总耗时（ms）	组件耗时（ms）	当前服务	说明
1	/productpage	190	0	istio-ingressgateway	Envoy Outbound
2	/productpage	190	1	istio-ingressgateway	Ingress -> Productpage 网络传输
3	/productpage	189	1	productpage	Envoy Inbound
4	/productpage	188	21	productpage	应用内部处理
5	/details/0	8	1	productpage	Envoy Outbound
6	/details/0	7	3	productpage	Productpage -> Details 网络传输
7	/details/0	4	0	details	Envoy Inbound
8	/details/0	4	4	details	应用内部
9	/reviews/0	159	0	productpage	Envoy Outbound
10	/reviews/0	159	14	productpage	Productpage -> Reviews 网络传输
11	/reviews/0	145	1	reviews	Envoy Inbound
12	/reviews/0	144	109	reviews	应用内部处理
13	/ratings/0	35	2	reviews	Envoy Outbound
14	/ratings/0	33	16	reviews	Reviews -> Ratings 网络传输
15	/ratings/0	17	1	ratings	Envoy Inbound
16	/ratings/0	16	16	ratings	应用内部处理

从以上信息可以发现：

本次请求总耗时 190ms；
在 Istio sidecar 模式下，每次流量在进出应用容器时都需要经过一次 Envoy 代理，每次耗时在 0 到 2 ms；
在 Pod 间的网络请求耗时在 1 到 16ms 之间；
将耗时做多的调用链 Ingress Gateway -> Productpage -> Reviews -> Ratings 上的所有耗时累计 182 ms，小于请求总耗时 190ms，这是因为数据本身有误差，以及 Span 的开始时间并不一定等于父 Span 的结束时间，如果你在 SkyWalking 的追踪页面，选择「列表」样式查看追踪数据（见图 2）可以更直观的发现这个问题；
我们可以查看到最耗时的部分是 Reviews 应用，耗时 109ms，因此我们可以针对该应用进行优化；

总结

只要对应用代码稍作修改就可以在 Istio 很方便的使用分布式追踪功能。在 Istio 支持的众多分布式追踪系统中，Apache SkyWalking 是其中的佼佼者。它不仅支持分布式追踪，还支持指标和日志收集、报警、Kubernetes 和服务网格监控，使用 eBPF 诊断服务网格性能等功能，是一个功能完备的云原生应用分析平台。本文中为了方便演示，将追踪采样率设置为了 100%，在生产使用时请根据需要调整采样策略（采样百分比），防止产生过多的追踪日志。

如果您不熟悉服务网格和 Kubernetes 安全性，我们在 Tetrate Academy 提供了一系列免费在线课程，可以让您快速了解 Istio 和 Envoy。

如果您正在寻找一种快速将 Istio 投入生产的方法，请查看 Tetrate Istio Distribution (TID)。TID 是 Tetrate 的强化、完全上游的 Istio 发行版，具有经过 FIPS 验证的构建和支持。这是开始使用 Istio 的好方法，因为您知道您有一个值得信赖的发行版，有一个支持您的专家团队，并且如果需要，还可以选择快速获得 FIPS 合规性。

一旦启动并运行 Istio，您可能需要更简单的方法来管理和保护您的服务，而不仅仅是 Istio 中可用的方法，这就是 Tetrate Service Bridge 的用武之地。您可以在这里详细了解 Tetrate Service Bridge 如何使服务网格更安全、更易于管理和弹性，或联系我们进行快速演示。

Blog: How to run Apache SkyWalking on AWS EKS and RDS/Aurora

Tue, 13 Dec 2022 00:00:00 +0000

Introduction

Apache SkyWalking is an open source APM tool for monitoring and troubleshooting distributed systems, especially designed for microservices, cloud native and container-based (Docker, Kubernetes, Mesos) architectures. It provides distributed tracing, service mesh observability, metric aggregation and visualization, and alarm.

In this article, I will introduce how to quickly set up Apache SkyWalking on AWS EKS and RDS/Aurora, as well as a couple of sample services, monitoring services to observe SkyWalking itself.

Prerequisites

AWS account
AWS CLI
Terraform
kubectl

We can use the AWS web console or CLI to create all resources needed in this tutorial, but it can be too tedious and hard to debug when something goes wrong. So in this artical I will use Terraform to create all AWS resources, deploy SkyWalking, sample services, and load generator services (Locust).

Architecture

The demo architecture is as follows:

graph LR
    subgraph AWS
        subgraph EKS
          subgraph istio-system namespace
              direction TB
              OAP[[SkyWalking OAP]]
              UI[[SkyWalking UI]]
            Istio[[istiod]]
          end
          subgraph sample namespace
              Service0[[Service0]]
              Service1[[Service1]]
              ServiceN[[Service ...]]
          end
          subgraph locust namespace
              LocustMaster[[Locust Master]]
              LocustWorkers0[[Locust Worker 0]]
              LocustWorkers1[[Locust Worker 1]]
              LocustWorkersN[[Locust Worker ...]]
          end
        end
        RDS[[RDS/Aurora]]
    end
    OAP --> RDS
    Service0 -. telemetry data -.-> OAP
    Service1 -. telemetry data -.-> OAP
    ServiceN -. telemetry data -.-> OAP
    UI --query--> OAP
    LocustWorkers0 -- traffic --> Service0
    LocustWorkers1 -- traffic --> Service0
    LocustWorkersN -- traffic --> Service0
    Service0 --> Service1 --> ServiceN
    LocustMaster --> LocustWorkers0
    LocustMaster --> LocustWorkers1
    LocustMaster --> LocustWorkersN
    User --> LocustMaster

As shown in the architecture diagram, we need to create the following AWS resources:

EKS cluster
RDS instance or Aurora cluster

Sounds simple, but there are a lot of things behind the scenes, such as VPC, subnets, security groups, etc. You have to configure them correctly to make sure the EKS cluster can connect to RDS instance/Aurora cluster otherwise the SkyWalking won’t work. Luckily, Terraform can help us to create and destroy all these resources automatically.

I have created a Terraform module to create all AWS resources needed in this tutorial, you can find it in the GitHub repository.

Create AWS resources

First, we need to clone the GitHub repository and cd into the folder:

git clone https://github.com/kezhenxu94/oap-load-test.git

Then, we need to create a file named terraform.tfvars to specify the AWS region and other variables:

cat > terraform.tfvars <<EOF
aws_access_key = ""
aws_secret_key = ""
cluster_name   = "skywalking-on-aws"
region         = "ap-east-1"
db_type        = "rds-postgresql"
EOF

If you have already configured the AWS CLI, you can skip the aws_access_key and aws_secret_key variables. To install SkyWalking with RDS postgresql, set the db_type to rds-postgresql, to install SkyWalking with Aurora postgresql, set the db_type to aurora-postgresql.

There are a lot of other variables you can configure, such as tags, sample services count, replicas, etc., you can find them in the variables.tf.

Then, we can run the following commands to initialize the Terraform module and download the required providers, then create all AWS resources:

terraform init
terraform apply -var-file=terraform.tfvars

Type yes to confirm the creation of all AWS resources, or add the -auto-approve flag to the terraform apply to skip the confirmation:

terraform apply -var-file=terraform.tfvars -auto-approve

Now what you need to do is to wait for the creation of all AWS resources to complete, it may take a few minutes. You can check the progress of the creation in the AWS web console, and check the deployment progress of the services inside the EKS cluster.

Generate traffic

Besides creating necessary AWS resources, the Terraform module also deploys SkyWalking, sample services, and Locust load generator services to the EKS cluster.

You can access the Locust web UI to generate traffic to the sample services:

open http://$(kubectl get svc -n locust -l app=locust-master -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}'):8089

The command opens the browser to the Locust web UI, you can configure the number of users and hatch rate to generate traffic.

Observe SkyWalking

You can access the SkyWalking web UI to observe the sample services.

First you need to forward the SkyWalking UI port to local

kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=skywalking -l component=ui -o name) 8080:8080

And then open the browser to http://localhost:8080 to access the SkyWalking web UI.

Observe RDS/Aurora

You can also access the RDS/Aurora web console to observe the performance of RDS/Aurora instance/Aurora cluste.

Test Results

Test 1: SkyWalking with EKS and RDS PostgreSQL

Service Traffic

RDS Performance

SkyWalking Performance

Test 2: SkyWalking with EKS and Aurora PostgreSQL

Service Traffic

RDS Performance

SkyWalking Performance

Clean up

When you are done with the demo, you can run the following command to destroy all AWS resources:

terraform destroy -var-file=terraform.tfvars -auto-approve

Zh: 如何在 AWS EKS 和 RDS/Aurora 上运行 Apache SkyWalking

Tue, 13 Dec 2022 00:00:00 +0000

介绍

Apache SkyWalking 是一个开源的 APM 工具，用于监控分布式系统和排除故障，特别是为微服务、云原生和基于容器（Docker、Kubernetes、Mesos）的架构而设计。它提供分布式跟踪、服务网格可观测性、指标聚合和可视化以及警报。

在本文中，我将介绍如何在 AWS EKS 和 RDS/Aurora 上快速设置 Apache SkyWalking，以及几个示例服务，监控服务以观察 SkyWalking 本身。

先决条件

AWS 账号
AWS CLI
Terraform
kubectl

我们可以使用 AWS Web 控制台或 CLI 来创建本教程所需的所有资源，但是当出现问题时，它可能过于繁琐且难以调试。因此，在本文中，我将使用 Terraform 创建所有 AWS 资源、部署 SkyWalking、示例服务和负载生成器服务 (Locust)。

架构

演示架构如下：

graph LR
    subgraph AWS
        subgraph EKS
          subgraph istio-system namespace
              direction TB
              OAP[[SkyWalking OAP]]
              UI[[SkyWalking UI]]
            Istio[[istiod]]
          end
          subgraph sample namespace
              Service0[[Service0]]
              Service1[[Service1]]
              ServiceN[[Service ...]]
          end
          subgraph locust namespace
              LocustMaster[[Locust Master]]
              LocustWorkers0[[Locust Worker 0]]
              LocustWorkers1[[Locust Worker 1]]
              LocustWorkersN[[Locust Worker ...]]
          end
        end
        RDS[[RDS/Aurora]]
    end
    OAP --> RDS
    Service0 -. telemetry data -.-> OAP
    Service1 -. telemetry data -.-> OAP
    ServiceN -. telemetry data -.-> OAP
    UI --query--> OAP
    LocustWorkers0 -- traffic --> Service0
    LocustWorkers1 -- traffic --> Service0
    LocustWorkersN -- traffic --> Service0
    Service0 --> Service1 --> ServiceN
    LocustMaster --> LocustWorkers0
    LocustMaster --> LocustWorkers1
    LocustMaster --> LocustWorkersN
    User --> LocustMaster

如架构图所示，我们需要创建以下 AWS 资源：

EKS 集群
RDS 实例或 Aurora 集群

听起来很简单，但背后有很多东西，比如 VPC、子网、安全组等。你必须正确配置它们以确保 EKS 集群可以连接到 RDS 实例 / Aurora 集群，否则 SkyWalking 不会不工作。幸运的是，Terraform 可以帮助我们自动创建和销毁所有这些资源。

我创建了一个 Terraform 模块来创建本教程所需的所有 AWS 资源，您可以在 GitHub 存储库中找到它。

创建 AWS 资源

首先，我们需要将 GitHub 存储库克隆 cd 到文件夹中：

git clone https://github.com/kezhenxu94/oap-load-test.git

然后，我们需要创建一个文件 terraform.tfvars 来指定 AWS 区域和其他变量：

cat > terraform.tfvars <<EOF
aws_access_key = ""
aws_secret_key = ""
cluster_name   = "skywalking-on-aws"
region         = "ap-east-1"
db_type        = "rds-postgresql"
EOF

如果您已经配置了 AWS CLI，则可以跳过 aws_access_key 和 aws_secret_key 变量。要使用 RDS postgresql 安装 SkyWalking，请将 db_type 设置为 rds-postgresql，要使用 Aurora postgresql 安装 SkyWalking，请将 db_type 设置为 aurora-postgresql。

您可以配置许多其他变量，例如标签、示例服务计数、副本等，您可以在 variables.tf 中找到它们。

然后，我们可以运行以下命令来初始化 Terraform 模块并下载所需的提供程序，然后创建所有 AWS 资源：

terraform init
terraform apply -var-file=terraform.tfvars

键入 yes 以确认所有 AWS 资源的创建，或将标志 -auto-approve 添加到 terraform apply 以跳过确认：

terraform apply -var-file=terraform.tfvars -auto-approve

现在你需要做的就是等待所有 AWS 资源的创建完成，这可能需要几分钟的时间。您可以在 AWS Web 控制台查看创建进度，也可以查看 EKS 集群内部服务的部署进度。

产生流量

除了创建必要的 AWS 资源外，Terraform 模块还将 SkyWalking、示例服务和 Locust 负载生成器服务部署到 EKS 集群。

您可以访问 Locust Web UI 以生成到示例服务的流量：

open http://$(kubectl get svc -n locust -l app=locust-master -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}'):8089

该命令将浏览器打开到 Locust web UI，您可以配置用户数量和孵化率以生成流量。

观察 SkyWalking

您可以访问 SkyWalking Web UI 来观察示例服务。

首先需要将 SkyWalking UI 端口转发到本地：

kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=skywalking -l component=ui -o name) 8080:8080

然后在浏览器中打开 http://localhost:8080 访问 SkyWalking web UI。

观察 RDS/Aurora

您也可以访问 RDS/Aurora web 控制台，观察 RDS/Aurora 实例 / Aurora 集群的性能。

试验结果

测试 1：使用 EKS 和 RDS PostgreSQL 的 SkyWalking

服务流量

RDS 性能

SkyWalking 性能

测试 2：使用 EKS 和 Aurora PostgreSQL 的 SkyWalking

服务流量

RDS 性能

SkyWalking 性能

清理

完成演示后，您可以运行以下命令销毁所有 AWS 资源：

terraform destroy -var-file=terraform.tfvars -auto-approve

Blog: Pinpoint Service Mesh Critical Performance Impact by using eBPF

Tue, 05 Jul 2022 00:00:00 +0000

Content

Background

Apache SkyWalking observes metrics, logs, traces, and events for services deployed into the service mesh. When troubleshooting, SkyWalking error analysis can be an invaluable tool helping to pinpoint where an error occurred. However, performance problems are more difficult: It’s often impossible to locate the root cause of performance problems with pre-existing observation data. To move beyond the status quo, dynamic debugging and troubleshooting are essential service performance tools. In this article, we’ll discuss how to use eBPF technology to improve the profiling feature in SkyWalking and analyze the performance impact in the service mesh.

Trace Profiling in SkyWalking

Since SkyWalking 7.0.0, Trace Profiling has helped developers find performance problems by periodically sampling the thread stack to let developers know which lines of code take more time. However, Trace Profiling is not suitable for the following scenarios:

Thread Model: Trace Profiling is most useful for profiling code that executes in a single thread. It is less useful for middleware that relies heavily on async execution models. For example Goroutines in Go or Kotlin Coroutines.
Language: Currently, Trace Profiling is only supported in Java and Python, since it’s not easy to obtain the thread stack in the runtimes of some languages such as Go and Node.js.
Agent Binding: Trace Profiling requires Agent installation, which can be tricky depending on the language (e.g., PHP has to rely on its C kernel; Rust and C/C++ require manual instrumentation to make install).
Trace Correlation: Since Trace Profiling is only associated with a single request it can be hard to determine which request is causing the problem.
Short Lifecycle Services: Trace Profiling doesn’t support short-lived services for (at least) two reasons:
1. It’s hard to differentiate system performance from class code manipulation in the booting stage.
2. Trace profiling is linked to an endpoint to identify performance impact, but there is no endpoint to match these short-lived services.

Fortunately, there are techniques that can go further than Trace Profiling in these situations.

Introduce eBPF

We have found that eBPF — a technology that can run sandboxed programs in an operating system kernel and thus safely and efficiently extend the capabilities of the kernel without requiring kernel modifications or loading kernel modules — can help us fill gaps left by Trace Profiling. eBPF is a trending technology because it breaks the traditional barrier between user and kernel space. Programs can now inject bytecode that runs in the kernel, instead of having to recompile the kernel to customize it. This is naturally a good fit for observability.

In the figure below, we can see that when the system executes the execve syscalls, the eBPF program is triggered, and the current process runtime information is obtained by using function calls.

Using eBPF technology, we can expand the scope of Skywalking’s profiling capabilities:

Global Performance Analysis: Before eBPF, data collection was limited to what agents can observe. Since eBPF programs run in the kernel, they can observe all threads. This is especially useful when you are not sure whether a performance problem is caused by a particular request.
Data Content: eBPF can dump both user and kernel space thread stacks, so if a performance issue happens in kernel space, it’s easier to find.
Agent Binding: All modern Linux kernels support eBPF, so there is no need to install anything. This means it is an orchestration-free vs an agent model. This reduces friction caused by built-in software which may not have the correct agents installed, such as Envoy in a Service Mesh.
Sampling Type: Unlike Trace Profiling, eBPF is event-driven and, therefore, not constrained by interval polling. For example, eBPF can trigger events and collect more data depending on a transfer size threshold. This can allow the system to triage and prioritize data collection under extreme load.

eBPF Limitations

While eBPF offers significant advantages for hunting performance bottlenecks, no technology is perfect. eBPF has a number of limitations described below. Fortunately, since SkyWalking does not require eBPF, the impact is limited.

Linux Version Requirement: eBPF programs require a Linux kernel version above 4.4, with later kernel versions offering more data to be collected. The BCC has documented the features supported by different Linux kernel versions, with the differences between versions usually being what data can be collected with eBPF.
Privileges Required: All processes that intend to load eBPF programs into the Linux kernel must be running in privileged mode. As such, bugs or other issues in such code may have a big impact.
Weak Support for Dynamic Language: eBPF has weak support for JIT-based dynamic languages, such as Java. It also depends on what data you want to collect. For Profiling, eBPF does not support parsing the symbols of the program, which is why most eBPF-based profiling technologies only support static languages like C, C++, Go, and Rust. However, symbol mapping can sometimes be solved through tools provided by the language. For example, in Java, perf-map-agent can be used to generate the symbol mapping. However, dynamic languages don’t support the attach (uprobe) functionality that would allow us to trace execution events through symbols.

Introducing SkyWalking Rover

SkyWalking Rover introduces the eBPF profiling feature into the SkyWalking ecosystem. The figure below shows the overall architecture of SkyWalking Rover. SkyWalking Rover is currently supported in Kubernetes environments and must be deployed inside a Kubernetes cluster. After establishing a connection with the SkyWalking backend server, it saves information about the processes on the current machine to SkyWalking. When the user creates an eBPF profiling task via the user interface, SkyWalking Rover receives the task and executes it in the relevant C, C++, Golang, and Rust language-based programs.

Other than an eBPF-capable kernel, there are no additional prerequisites for deploying SkyWalking Rover.

CPU Profiling with Rover

CPU profiling is the most intuitive way to show service performance. Inspired by Brendan Gregg‘s blog post, we’ve divided CPU profiling into two types that we have implemented in Rover:

On-CPU Profiling: Where threads are spending time running on-CPU.
Off-CPU Profiling: Where time is spent waiting while blocked on I/O, locks, timers, paging/swapping, etc.

Profiling Envoy with eBPF

Envoy is a popular proxy, used as the data plane by the Istio service mesh. In a Kubernetes cluster, Istio injects Envoy into each service’s pod as a sidecar where it transparently intercepts and processes incoming and outgoing traffic. As the data plane, any performance issues in Envoy can affect all service traffic in the mesh. In this scenario, it’s more powerful to use eBPF profiling to analyze issues in production caused by service mesh configuration.

Demo Environment

If you want to see this scenario in action, we’ve built a demo environment where we deploy an Nginx service for stress testing. Traffic is intercepted by Envoy and forwarded to Nginx. The commands to install the whole environment can be accessed through GitHub.

On-CPU Profiling

On-CPU profiling is suitable for analyzing thread stacks when service CPU usage is high. If the stack is dumped more times, it means that the thread stack occupies more CPU resources.

When installing Istio using the demo configuration profile, we found there are two places where we can optimize performance:

Zipkin Tracing: Different Zipkin sampling percentages have a direct impact on QPS.
Access Log Format: Reducing the fields of the Envoy access log can improve QPS.

Zipkin Tracing

Zipkin with 100% sampling

In the default demo configuration profile, Envoy is using 100% sampling as default tracing policy. How does that impact the performance?

As shown in the figure below, using the on-CPU profiling, we found that it takes about 16% of the CPU overhead. At a fixed consumption of 2 CPUs, its QPS can reach 5.7K.

Disable Zipkin tracing

At this point, we found that if Zipkin is not necessary, the sampling percentage can be reduced or we can even disable tracing. Based on the Istio documentation, we can disable tracing when installing the service mesh using the following command:

istioctl install -y --set profile=demo \
   --set 'meshConfig.enableTracing=false' \
   --set 'meshConfig.defaultConfig.tracing.sampling=0.0'

After disabling tracing, we performed on-CPU profiling again. According to the figure below, we found that Zipkin has disappeared from the flame graph. With the same 2 CPU consumption as in the previous example, the QPS reached 9K, which is an almost 60% increase.

Tracing with Throughput

With the same CPU usage, we’ve discovered that Envoy performance greatly improves when the tracing feature is disabled. Of course, this requires us to make trade-offs between the number of samples Zipkin collects and the desired performance of Envoy (QPS).

The table below illustrates how different Zipkin sampling percentages under the same CPU usage affect QPS.

Zipkin sampling %	QPS	CPUs	Note
100% (default)	5.7K	2	16% used by Zipkin
1%	8.1K	2	0.3% used by Zipkin
disabled	9.2K	2	0% used by Zipkin

Access Log Format

Default Log Format

In the default demo configuration profile, the default Access Log format contains a lot of data. The flame graph below shows various functions involved in parsing the data such as request headers, response headers, and streaming the body.

Simplifying Access Log Format

Typically, we don’t need all the information in the access log, so we can often simplify it to get what we need. The following command simplifies the access log format to only display basic information:

istioctl install -y --set profile=demo \
   --set meshConfig.accessLogFormat="[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE%\n"

After simplifying the access log format, we found that the QPS increased from 5.7K to 5.9K. When executing the on-CPU profiling again, the CPU usage of log formatting dropped from 2.4% to 0.7%.

Simplifying the log format helped us to improve the performance.

Off-CPU Profiling

Off-CPU profiling is suitable for performance issues that are not caused by high CPU usage. For example, when there are too many threads in one service, using off-CPU profiling could reveal which threads spend more time context switching.

We provide data aggregation in two dimensions:

Switch count: The number of times a thread switches context. When the thread returns to the CPU, it completes one context switch. A thread stack with a higher switch count spends more time context switching.
Switch duration: The time it takes a thread to switch the context. A thread stack with a higher switch duration spends more time off-CPU.

Write Access Log

Enable Write

Using the same environment and settings as before in the on-CPU test, we performed off-CPU profiling. As shown below, we found that access log writes accounted for about 28% of the total context switches. The “__write” shown below also indicates that this method is the Linux kernel method.

Disable Write

SkyWalking implements Envoy’s Access Log Service (ALS) feature which allows us to send access logs to the SkyWalking Observability Analysis Platform (OAP) using the gRPC protocol. Even by disabling the access logging, we can still use ALS to capture/aggregate the logs. We’ve disabled writing to the access log using the following command:

istioctl install -y --set profile=demo --set meshConfig.accessLogFile=""

After disabling the Access Log feature, we performed the off-CPU profiling. File writing entries have disappeared as shown in the figure below. Envoy throughput also increased from 5.7K to 5.9K.

Conclusion

In this article, we’ve examined the insights Apache Skywalking’s Trace Profiling can give us and how much more can be achieved with eBPF profiling. All of these features are implemented in skywalking-rover. In addition to on- and off-CPU profiling, you will also find the following features:

Continuous profiling, helps you automatically profile without manual intervention. For example, when Rover detects that the CPU exceeds a configurable threshold, it automatically executes the on-CPU profiling task.
More profiling types to enrich usage scenarios, such as network, and memory profiling.

Blog: Chaos Mesh + SkyWalking: Better Observability for Chaos Engineering

Tue, 21 Dec 2021 00:00:00 +0000

Chaos Mesh is an open-source cloud-native chaos engineering platform. You can use Chaos Mesh to conveniently inject failures and simulate abnormalities that might occur in reality, so you can identify potential problems in your system. Chaos Mesh also offers a Chaos Dashboard which allows you to monitor the status of a chaos experiment. However, this dashboard cannot let you observe how the failures in the experiment impact the service performance of applications. This hinders us from further testing our systems and finding potential problems.

Apache SkyWalking is an open-source application performance monitor (APM), specially designed to monitor, track, and diagnose cloud native, container-based distributed systems. It collects events that occur and then displays them on its dashboard, allowing you to observe directly the type and number of events that have occurred in your system and how different events impact the service performance.

When you use SkyWalking and Chaos Mesh together during chaos experiments, you can observe how different failures impact the service performance.

This tutorial will show you how to configure SkyWalking and Chaos Mesh. You’ll also learn how to leverage the two systems to monitor events and observe in real time how chaos experiments impact applications’ service performance.

Preparation

Before you start to use SkyWalking and Chaos Mesh, you have to:

Set up a SkyWalking cluster according to the SkyWalking configuration guide.
Deploy Chao Mesh using Helm.
Install JMeter or other Java testing tools (to increase service loads).
Configure SkyWalking and Chaos Mesh according to this guide if you just want to run a demo.

Now, you are fully prepared, and we can cut to the chase.

Step 1: Access the SkyWalking cluster

After you install the SkyWalking cluster, you can access its user interface (UI). However, no service is running at this point, so before you start monitoring, you have to add one and set the agents.

In this tutorial, we take Spring Boot, a lightweight microservice framework, as an example to build a simplified demo environment.

Create a SkyWalking demo in Spring Boot by referring to this document.
Execute the command kubectl apply -f demo-deployment.yaml -n skywalking to deploy the demo.

After you finish deployment, you can observe the real-time monitoring results at the SkyWalking UI.

Note: Spring Boot and SkyWalking have the same default port number: 8080. Be careful when you configure the port forwarding; otherise, you may have port conflicts. For example, you can set Spring Boot’s port to 8079 by using a command like kubectl port-forward svc/spring-boot-skywalking-demo 8079:8080 -n skywalking to avoid conflicts.

Step 2: Deploy SkyWalking Kubernetes Event Exporter

SkyWalking Kubernetes Event Exporter is able to watch, filter, and send Kubernetes events into the SkyWalking backend. SkyWalking then associates the events with the system metrics and displays an overview about when and how the metrics are affected by the events.

If you want to deploy SkyWalking Kubernetes Event Explorer with one line of commands, refer to this document to create configuration files in YAML format and then customize the parameters in the filters and exporters. Now, you can use the command kubectl apply to deploy SkyWalking Kubernetes Event Explorer.

Step 3: Use JMeter to increase service loads

To better observe the change in service performance, you need to increase the service loads on Spring Boot. In this tutorial, we use JMeter, a widely adopted Java testing tool, to increase the service loads.

Perform a stress test on localhost:8079 using JMeter and add five threads to continuously increase the service loads.

Open the SkyWalking Dashboard. You can see that the access rate is 100%, and that the service loads reach about 5,300 calls per minute (CPM).

Step 4: Inject failures via Chaos Mesh and observe results

After you finish the three steps above, you can use the Chaos Dashboard to simulate stress scenarios and observe the change in service performance during chaos experiments.

The following sections describe how service performance varies under the stress of three chaos conditions:

CPU load: 10%; memory load: 128 MB

The first chaos experiment simulates low CPU usage. To display when a chaos experiment starts and ends, click the switching button on the right side of the dashboard. To learn whether the experiment is Applied to the system or Recovered from the system, move your cursor onto the short, green line.

During the time period between the two short, green lines, the service load decreases to 4,929 CPM, but returns to normal after the chaos experiment ends.
CPU load: 50%; memory load: 128 MB

When the application’s CPU load increases to 50%, the service load decreases to 4,307 CPM.
CPU load: 100%; memory load: 128 MB

When the CPU usage is at 100%, the service load decreases to only 40% of what it would be if no chaos experiments were taking place.

Because the process scheduling under the Linux system does not allow a process to occupy the CPU all the time, the deployed Spring Boot Demo can still handle 40% of the access requests even in the extreme case of a full CPU load.

Summary

By combining SkyWalking and Chaos Mesh, you can clearly observe when and to what extent chaos experiments affect application service performance. This combination of tools lets you observe the service performance in various extreme conditions, thus boosting your confidence in your services.

Chaos Mesh has grown a lot in 2021 thanks to the unremitting efforts of all PingCAP engineers and community contributors. In order to continue to upgrade our support for our wide variety of users and learn more about users’ experience in Chaos Engineering, we’d like to invite you to take this survey and give us your valuable feedback.

If you want to know more about Chaos Mesh, you’re welcome to join the Chaos Mesh community on GitHub or our Slack discussions (#project-chaos-mesh). If you find any bugs or missing features when using Chaos Mesh, you can submit your pull requests or issues to our GitHub repository.

Blog: Observe VM Service Meshes with Apache SkyWalking and the Envoy Access Log Service

Sun, 21 Feb 2021 00:00:00 +0000

Origin: Observe VM Service Meshes with Apache SkyWalking and the Envoy Access Log Service - The New Stack

Apache SkyWalking: an APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based (Docker, Kubernetes, Mesos) architectures.

Envoy Access Log Service: Access Log Service (ALS) is an Envoy extension that emits detailed access logs of all requests going through Envoy.

Background

In the previous post, we talked about the observability of service mesh under Kubernetes environment, and applied it to the bookinfo application in practice. We also mentioned that, in order to map the IP addresses into services, SkyWalking needs access to the service metadata from a Kubernetes cluster, which is not available for services deployed in virtual machines (VMs). In this post, we will introduce a new analyzer in SkyWalking that leverages Envoy’s metadata exchange mechanism to decouple with Kubernetes. The analyzer is designed to work in Kubernetes environments, VM environments, and hybrid environments. If there are virtual machines in your service mesh, you might want to try out this new analyzer for better observability, which we will demonstrate in this tutorial.

How it works

The mechanism of how the analyzer works is the same as what we discussed in the previous post. What makes VMs different from Kubernetes is that, for VM services, there are no places where we can fetch the metadata to map the IP addresses into services.

The basic idea we present in this article is to carry the metadata along with Envoy’s access logs, which is called metadata-exchange mechanism in Envoy. When Istio pilot-agent starts an Envoy proxy as a sidecar of a service, it collects the metadata of that service from the Kubernetes platform, or a file on the VM where that service is deployed, and injects the metadata into the bootstrap configuration of Envoy. Envoy will carry the metadata transparently when emitting access logs to the SkyWalking receiver.

But how does Envoy compose a piece of a complete access log that involves the client side and server side? When a request goes out from Envoy, a plugin of istio-proxy named “metadata-exchange” injects the metadata into the http headers (with a prefix like x-envoy-downstream-), and the metadata is propagated to the server side. The Envoy sidecar of the server side receives the request and parses the headers into metadata, and puts the metadata into the access log, keyed by wasm.downstream_peer. The server side Envoy also puts its own metadata into the access log keyed by wasm.upstream_peer. Hence the two sides of a single request are completed.

With the metadata-exchange mechanism, we can use the metadata directly without any extra query.

Example

In this tutorial, we will use another demo application Online Boutique that consists of 10+ services so that we can deploy some of them in VMs and make them communicate with other services deployed in Kubernetes.

Topology of Online Boutique In order to cover as many cases as possible, we will deploy CheckoutService and PaymentService on VM and all the other services on Kubernetes, so that we can cover the cases like Kubernetes → VM (e.g. Frontend → CheckoutService), VM → Kubernetes (e.g. CheckoutService → ShippingService), and VM → VM ( e.g. CheckoutService → PaymentService).

NOTE: All the commands used in this tutorial are accessible on GitHub.

git clone https://github.com/SkyAPMTest/sw-als-vm-demo-scripts
cd sw-als-vm-demo-scripts

Make sure to init the gcloud SDK properly before moving on. Modify the GCP_PROJECT in file env.sh to your own project name. Most of the other variables should be OK to work if you keep them intact. If you would like to use ISTIO_VERSION >/= 1.8.0, please make sure this patch is included.

Prepare Kubernetes cluster and VM instances 00-create-cluster-and-vms.sh creates a new GKE cluster and 2 VM instances that will be used through the entire tutorial, and sets up some necessary firewall rules for them to communicate with each other.
Install Istio and SkyWalking 01a-install-istio.sh installs Istio Operator with spec resources/vmintegration.yaml. In the YAML file, we enable the meshExpansion that supports VM in mesh. We also enable the Envoy access log service and specify the address skywalking-oap.istio-system.svc.cluster.local:11800 to which Envoy emits the access logs. 01b-install-skywalking.sh installs Apache SkyWalking and sets the analyzer to mx-mesh.
Create files to initialize the VM 02-create-files-to-transfer-to-vm.sh creates necessary files that will be used to initialize the VMs. 03-copy-work-files-to-vm.sh securely transfers the generated files to the VMs with gcloud scp command. Now use ./ssh.sh checkoutservice and ./ssh.sh paymentservice to log into the two VMs respectively, and cd to the ~/work directory, execute ./prep-checkoutservice.sh on checkoutservice VM instance and ./prep-paymentservice.sh on paymentservice VM instance. The Istio sidecar should be installed and started properly. To verify that, use tail -f /var/logs/istio/istio.log to check the Istio logs. The output should be something like:
```
2020-12-12T08:07:07.348329Z	info	sds	resource:default new connection
2020-12-12T08:07:07.348401Z	info	sds	Skipping waiting for gateway secret
2020-12-12T08:07:07.348401Z	info	sds	Skipping waiting for gateway secret
2020-12-12T08:07:07.568676Z	info	cache	Root cert has changed, start rotating root cert for SDS clients
2020-12-12T08:07:07.568718Z	info	cache	GenerateSecret default
2020-12-12T08:07:07.569398Z	info	sds	resource:default pushed key/cert pair to proxy
2020-12-12T08:07:07.949156Z	info	cache	Loaded root cert from certificate ROOTCA
2020-12-12T08:07:07.949348Z	info	sds	resource:ROOTCA pushed root cert to proxy
2020-12-12T20:12:07.384782Z	info	sds	resource:default pushed key/cert pair to proxy
2020-12-12T20:12:07.384832Z	info	sds	Dynamic push for secret default
```
The dnsmasq configuration address=/.svc.cluster.local/{ISTIO_SERVICE_IP_STUB} also resolves the domain names ended with .svc.cluster.local to Istio service IP, so that you are able to access the Kubernetes services in the VM by fully qualified domain name (FQDN) such as httpbin.default.svc.cluster.local.
Deploy demo application Because we want to deploy CheckoutService and PaymentService manually on VM, resources/google-demo.yaml removes the two services from the original YAML . 04a-deploy-demo-app.sh deploys the other services on Kubernetes. Then log into the 2 VMs, run ~/work/deploy-checkoutservice.sh and ~/work/deploy-paymentservice.sh respectively to deploy CheckoutService and PaymentService.
Register VMs to Istio Services on VMs can access the services on Kubernetes by FQDN, but that’s not the case when the Kubernetes services want to talk to the VM services. The mesh has no idea where to forward the requests such as checkoutservice.default.svc.cluster.local because checkoutservice is isolated in the VM. Therefore, we need to register the services to the mesh. 04b-register-vm-with-istio.sh registers the VM services to the mesh by creating a “dummy” service without running Pods, and a WorkloadEntry to bridge the “dummy” service with the VM service.

Done!

The demo application contains a load generator service that performs requests repeatedly. We only need to wait a few seconds, and then open the SkyWalking web UI to check the results.

export POD_NAME=$(kubectl get pods --namespace istio-system -l "app=skywalking,release=skywalking,component=ui" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl port-forward $POD_NAME 8080:8080 --namespace istio-system

Navigate the browser to http://localhost:8080 . The metrics, topology should be there.

Troubleshooting

If you face any trouble when walking through the steps, here are some common problems and possible solutions:

VM service cannot access Kubernetes services? It’s likely the DNS on the VM doesn’t correctly resolve the fully qualified domain names. Try to verify that with nslookup istiod.istio-system.svc.cluster.local. If it doesn’t resolve to the Kubernetes CIDR address, recheck the step in prep-checkoutservice.sh and prep-paymentservice.sh. If the DNS works correctly, try to verify that Envoy has fetched the upstream clusters from the control plane with curl http://localhost:15000/clusters. If it doesn’t contain the target service, recheck prep-checkoutservice.sh.
Services are normal but nothing on SkyWalking WebUI? Check the SkyWalking OAP logs via kubectl -n istio-system logs -f $(kubectl get pod -A -l "app=skywalking,release=skywalking,component=oap" -o name) and WebUI logs via kubectl -n istio-system logs -f $(kubectl get pod -A -l "app=skywalking,release=skywalking,component=ui" -o name) to see whether there are any error logs . Also, make sure the time zone at the bottom-right of the browser is set to UTC +0.

Additional Resources

Observe a Service Mesh with Envoy ALS.

Blog: Observe Service Mesh with SkyWalking and Envoy Access Log Service

Thu, 03 Dec 2020 00:00:00 +0000

Author: Zhenxu Ke, Sheng Wu, and Tevah Platt. tetrate.io
Original link, Tetrate.io blog
Dec. 03th, 2020

Apache SkyWalking: an APM (application performance monitor) system, especially designed for microservices, cloud native, and container-based (Docker, Kubernetes, Mesos) architectures.

Envoy Access Log Service: Access Log Service (ALS) is an Envoy extension that emits detailed access logs of all requests going through Envoy.

Background

Apache SkyWalking has long supported observability in service mesh with Istio Mixer adapter. But since v1.5, Istio began to deprecate Mixer due to its poor performance in large scale clusters. Mixer’s functionalities have been moved into the Envoy proxies, and is supported only through the 1.7 Istio release. On the other hand, Sheng Wu and Lizan Zhou presented a better solution based on the Apache SkyWalking and Envoy ALS on KubeCon China 2019, to reduce the performance impact brought by Mixer, while retaining the same observability in service mesh. This solution was initially implemented by Sheng Wu, Hongtao Gao, Lizan Zhou, and Dhi Aurrahman at Tetrate.io. If you are looking for a more efficient solution to observe your service mesh instead of using a Mixer-based solution, this is exactly what you need. In this tutorial, we will explain a little bit how the new solution works, and apply it to the bookinfo application in practice.

How it works

From a perspective of observability, Envoy can be typically deployed in 2 modes, sidecar, and router. As a sidecar, Envoy mostly represents a single service to receive and send requests (2 and 3 in the picture below). While as a proxy, Envoy may represent many services (1 in the picture below).

In both modes, the logs emitted by ALS include a node identifier. The identifier starts with router~ (or ingress~) in router mode and sidecar~ in sidecar proxy mode.

Apart from the node identifier, there are several noteworthy properties in the access logs that will be used in this solution:

downstream_direct_remote_address: This field is the downstream direct remote address on which the request from the user was received. Note: This is always the physical peer, even if the remote address is inferred from for example the x-forwarded-for header, proxy protocol, etc.
downstream_remote_address: The remote/origin address on which the request from the user was received.
downstream_local_address: The local/destination address on which the request from the user was received.
upstream_remote_address: The upstream remote/destination address that handles this exchange.
upstream_local_address: The upstream local/origin address that handles this exchange.
upstream_cluster: The upstream cluster that upstream_remote_address belongs to.

We will discuss more about the properties in the following sections.

Sidecar

When serving as a sidecar, Envoy is deployed alongside a service, and delegates all the incoming/outgoing requests to/from the service.

Delegating incoming requests: in this case, Envoy acts as a server side sidecar, and sets the upstream_cluster in form of inbound|portNumber|portName|Hostname[or]SidecarScopeID.

The SkyWalking analyzer checks whether either downstream_remote_address can be mapped to a Kubernetes service:

a. If there is a service (say Service B) whose implementation is running in this IP(and port), then we have a service-to-service relation, Service B -> Service A, which can be used to build the topology. Together with the start_time and duration fields in the access log, we have the latency metrics now.

b. If there is no service that can be mapped to downstream_remote_address, then the request may come from a service out of the mesh. Since SkyWalking cannot identify the source service where the requests come from, it simply generates the metrics without source service, according to the topology analysis method. The topology can be built as accurately as possible, and the metrics detected from server side are still correct.
Delegating outgoing requests: in this case, Envoy acts as a client-side sidecar, and sets the upstream_cluster in form of outbound|<port>|<subset>|<serviceFQDN>.

Client side detection is relatively simpler than (1. Delegating incoming requests). If upstream_remote_address is another sidecar or proxy, we simply get the mapped service name and generate the topology and metrics. Otherwise, we have no idea what it is and consider it an UNKNOWN service.

Proxy role

When Envoy is deployed as a proxy, it is an independent service itself and doesn’t represent any other service like a sidecar does. Therefore, we can build client-side metrics as well as server-side metrics.

Example

In this section, we will use the typical bookinfo application to demonstrate how Apache SkyWalking 8.3.0+ (the latest version up to Nov. 30th, 2020) works together with Envoy ALS to observe a service mesh.

Installing Kubernetes

SkyWalking 8.3.0 supports the Envoy ALS solution under both Kubernetes environment and virtual machines (VM) environment, in this tutorial, we’ll only focus on the Kubernetes scenario, for VM solution, please stay tuned for our next blog, so we need to install Kubernetes before taking further steps.

In this tutorial, we will use the Minikube tool to quickly set up a local Kubernetes(v1.17) cluster for testing. In order to run all the needed components, including the bookinfo application, the SkyWalking OAP and WebUI, the cluster may need up to 4GB RAM and 2 CPU cores.

minikube start --memory=4096 --cpus=2

Next, run kubectl get pods --namespace=kube-system --watch to check whether all the Kubernetes components are ready. If not, wait for the readiness before going on.

Installing Istio

Istio provides a very convenient way to configure the Envoy proxy and enable the access log service. The built-in configuration profiles free us from lots of manual operations. So, for demonstration purposes, we will use Istio through this tutorial.

export ISTIO_VERSION=1.7.1
curl -L https://istio.io/downloadIstio | sh - 
sudo mv $PWD/istio-$ISTIO_VERSION/bin/istioctl /usr/local/bin/
istioctl  install --set profile=demo
kubectl label namespace default istio-injection=enabled

Run kubectl get pods --namespace=istio-system --watch to check whether all the Istio components are ready. If not, wait for the readiness before going on.

Enabling ALS

The demo profile doesn’t enable ALS by default. We need to reconfigure it to enable ALS via some configuration.

istioctl  manifest install \
               --set meshConfig.enableEnvoyAccessLogService=true \
               --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800

The example command --set meshConfig.enableEnvoyAccessLogService=true enables the Envoy access log service in the mesh. And as we said earlier, ALS is essentially a gRPC service that emits requests logs. The config meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800 tells this gRPC service where to emit the logs, say skywalking-oap.istio-system:11800, where we will deploy the SkyWalking ALS receiver later.

NOTE: You can also enable the ALS when installing Istio so that you don’t need to restart Istio after installation:

istioctl install --set profile=demo \
               --set meshConfig.enableEnvoyAccessLogService=true \
               --set meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800
kubectl label namespace default istio-injection=enabled

Deploying Apache SkyWalking

The SkyWalking community provides a Helm Chart to make it easier to deploy SkyWalking and its dependent services in Kubernetes. The Helm Chart can be found at the GitHub repository.

# Install Helm
curl -sSLO https://get.helm.sh/helm-v3.0.0-linux-amd64.tar.gz
sudo tar xz -C /usr/local/bin --strip-components=1 linux-amd64/helm -f helm-v3.0.0-linux-amd64.tar.gz
# Clone SkyWalking Helm Chart
git clone https://github.com/apache/skywalking-kubernetes
cd skywalking-kubernetes/chart
git reset --hard dd749f25913830c47a97430618cefc4167612e75
# Update dependencies
helm dep up skywalking
# Deploy SkyWalking
helm -n istio-system install skywalking skywalking \
               --set oap.storageType='h2'\
               --set ui.image.tag=8.3.0 \
               --set oap.image.tag=8.3.0-es7 \
               --set oap.replicas=1 \
               --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \
               --set oap.env.JAVA_OPTS='-Dmode=' \
               --set oap.envoy.als.enabled=true \
               --set elasticsearch.enabled=false

We deploy SkyWalking to the namespace istio-system, so that SkyWalking OAP service can be accessed by skywalking-oap.istio-system:11800, to which we told ALS to emit their logs, in the previous step.

We also enable the ALS analyzer in the SkyWalking OAP: oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh. The analyzer parses the access logs and maps the IP addresses in the logs to the real service names in the Kubernetes, to build a topology.

In order to retrieve the metadata (such as Pod IP and service names) from a Kubernetes cluster for IP mappings, we also set oap.envoy.als.enabled=true, to apply for a ClusterRole that has access to the metadata.

export POD_NAME=$(kubectl get pods -A -l "app=skywalking,release=skywalking,component=ui" -o name)
echo $POD_NAME
kubectl -n istio-system port-forward $POD_NAME 8080:8080

Now navigate your browser to http://localhost:8080 . You should be able to see the SkyWalking dashboard. The dashboard is empty for now, but after we deploy the demo application and generate traffic, it should be filled up later.

Deploying Bookinfo application

Run:

export ISTIO_VERSION=1.7.1
kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/platform/kube/bookinfo.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/$ISTIO_VERSION/samples/bookinfo/networking/bookinfo-gateway.yaml
kubectl wait --for=condition=Ready pods --all --timeout=1200s
minikube tunnel

Then navigate your browser to http://localhost/productpage. You should be able to see the typical bookinfo application. Refresh the webpage several times to generate enough access logs.

Done!

And you’re all done! Check out the SkyWalking WebUI again. You should see the topology of the bookinfo application, as well the metrics of each individual service of the bookinfo application.

Troubleshooting

Check all pods status: kubectl get pods -A.
SkyWalking OAP logs: kubectl -n istio-system logs -f $(kubectl get pod -A -l "app=skywalking,release=skywalking,component=oap" -o name).
SkyWalking WebUI logs: kubectl -n istio-system logs -f $(kubectl get pod -A -l "app=skywalking,release=skywalking,component=ui" -o name).
Make sure the time zone at the bottom-right of the WebUI is set to UTC +0.

Customizing Service Names

The SkyWalking community brought more improvements to the ALS solution in the 8.3.0 version. You can decide how to compose the service names when mapping from the IP addresses, with variables service and pod. For instance, configuring K8S_SERVICE_NAME_RULE to the expression ${service.metadata.name}-${pod.metadata.labels.version} gets service names with version label such as reviews-v1, reviews-v2, and reviews-v3, instead of a single service reviews, see the PR.

Working ALS with VM

Kubernetes is popular, but what about VMs? From what we discussed above, in order to map the IPs to services, SkyWalking needs access to the Kubernetes cluster, fetching service metadata and Pod IPs. But in a VM environment, there is no source from which we can fetch those metadata. In the next post, we will introduce another ALS analyzer based on the Envoy metadata exchange mechanism. With this analyzer, you are able to observe a service mesh in the VM environment. Stay tuned! If you want to have commercial support for the ALS solution or hybrid mesh observability, Tetrate Service Bridge, TSB is another good option out there.

Additional Resources

KubeCon 2019 Recorded Video.
Get more SkyWalking updates on the official website.

Apache SkyWalking founder Sheng Wu, SkyWalking core maintainer Zhenxu Ke are Tetrate engineers, and Tevah Platt is a content writer for Tetrate. Tetrate helps organizations adopt open source service mesh tools, including Istio, Envoy, and Apache SkyWalking, so they can manage microservices, run service mesh on any infrastructure, and modernize their applications.

Blog: The Apdex Score for Measuring Service Mesh Health

Sun, 26 Jul 2020 00:00:00 +0000

Author: Srinivasan Ramaswamy, tetrate
Original link, Tetrate.io blog

Asking `How are you` is more profound than `What are your symptoms`

Background

Recently I visited my preferred doctor. Whenever I visit, the doctor greets me with a series of light questions: How’s your day? How about the week before? Any recent trips? Did I break my cycling record? How’s your workout regimen? _Finally _he asks, “Do you have any problems?" On those visits when I didn’t feel ok, I would say something like, “I’m feeling dull this week, and I’m feeling more tired towards noon….” It’s at this point that he takes out his stethoscope, his pulse oximeter, and blood pressure apparatus. Then, if he feels he needs a more in-depth insight, he starts listing out specific tests to be made.

When I asked him if the first part of the discussion was just an ice-breaker, he said, “That’s the essential part. It helps me find out how you feel, rather than what your symptoms are.” So, despite appearances, our opening chat about life helped him structure subsequent questions on symptoms, investigations and test results.

On the way back, I couldn’t stop asking myself, “Shouldn’t we be managing our mesh this way, too?”

If I strike parallels between my own health check and a health check, “tests” would be log analysis, “investigations” would be tracing, and “symptoms” would be the traditional RED (Rate, Errors and Duration) metrics. That leaves the “essential part,” which is what we are talking about here: the Wellness Factor, primarily the health of our mesh.

Health in the context of service mesh

We can measure the performance of any observed service through RED metrics. RED metrics offer immense value in understanding the performance, reliability, and throughput of every service. Compelling visualizations of these metrics across the mesh make monitoring the entire mesh standardized and scalable. Also, setting alerts based on thresholds for each of these metrics helps to detect anomalies as and when they arise.

To establish the context of any service and observe them, it’s ideal to visualize the mesh as a topology.

A topology visualization of the mesh not only allows for picking any service and watching its metrics, but also gives vital information about service dependencies and the potential impact of a given service on the mesh.

While RED metrics of each service offer tremendous insights, the user is more concerned with the overall responsiveness of the mesh rather than each of these services in isolation.

To describe the performance of any service, right from submitting the request to receiving a completed http response, we’d be measuring the user’s perception of responsiveness. This measure of response time compared with a set threshold is called Apdex. This Apdex is an indicator of the health of a service in the mesh.

Apdex

Apdex is a measure of response time considered against a set threshold**. **It is the ratio of satisfactory response times and unsatisfactory response times to total response times.

Apdex is an industry standard to measure the satisfaction of users based on the response time of applications and services. It measures how satisfied your users are with your services, as traditional metrics such as average response time could get skewed quickly.

Satisfactory response time indicates the number of times when the roundtrip response time of a particular service was less than this threshold. Unsatisfactory response time while meaning the opposite, is further categorized as Tolerating and Frustrating. Tolerating accommodates any performance that is up to four times the threshold, and anything over that or any errors encountered is considered Frustrating. The threshold mentioned here is an ideal roundtrip performance that we expect from any service. We could even start with an organization-wide limit of say, 500ms.

The Apdex score is a ratio of satisfied and tolerating requests to the total requests made.

Each satisfied request counts as one request, while each tolerating request counts as half a satisfied request.

An Apdex score takes values from 0 to 1, with 0 being the worst possible score indicating that users were always frustrated, and ‘1’ as the best possible score (100% of response times were Satisfactory).

A percentage representation of this score also serves as the Health Indicator of the service.

The Math

The actual computation of this Apdex score is achieved through the following formula.

		            SatisfiedCount +  ( ToleratingCount / 2 )

Apdex Score  =  ------------------------------------------------------

                                TotalSamples

A percentage representation of this score is known as the Health Indicator of a service.

Example Computation

During a 2-minute period, a host handles 200 requests.

The Apdex threshold T = 0.5 seconds (500ms).

170 of the requests were handled within 500ms, so they are classified as Satisfied.
20 of the requests were handled between 500ms and 2 seconds (2000 ms), so they are classified as Tolerating.
The remaining 10 were not handled properly or took longer than 2 seconds, so they are classified as Frustrated.

The resulting Apdex score is 0.9: (170 + (20/2))/200 = 0.9.

The next level

At the next level, we can attempt to improve our topology visualization by coloring nodes based on their health. Also, we can include health as a part of the information we show when the user taps on a service.

Apdex specifications recommend the following Apdex Quality Ratings by classifying Apdex Score as Excellent (0.94 - 1.00), Good (0.85 - 0.93), Fair (0.70 - 0.84), Poor (0.50 - 0.69) and Unacceptable (0.00 - 0.49).

To visualize this, let’s look at our topology using traffic light colors, marking our nodes as Healthy, At-Risk and Unhealthy, where Unhealthy indicates health that falls below 80%. A rate between 80% and 95% indicates At-Risk, and health at 95% and above is termed Healthy.

Let’s incorporate this coloring into our topology visualization and take its usability to the next level. If implemented, we will be looking at something like this.

Moving further

Apdex provides tremendous visibility into customer satisfaction on the responsiveness of our services. Even more, by extending the implementation to the edges calling this service we get further insight into the health of the mesh itself.

Two services with similar Apdex scores offer the same customer satisfaction to the customer. However, the size of traffic that flows into the service can be of immense help in prioritizing between services to address. A service with higher traffic flow is an indication that this experience is impacting a significant number of users on the mesh.

While health relates to a service, we can also analyze the interactions between two services and calculate the health of the interaction. This health calculation of every interaction on the mesh helps us establish a critical path, based on the health of all interactions in the entire topology.

In a big mesh, showing traffic as yet another number will make it more challenging to visualize and monitor. We can, with a bit of creativity, improve the entire visualization by rendering the edges that connect services with different thickness depending on the throughput of the service.

An unhealthy service participating in a high throughput transaction could lead to excessive consumption of resources. On the other hand, this visualization also offers a great tip to maximize investment in tuning services.

Tuning service that is a part of a high throughput transaction offers exponential benefits when compared to tuning an occasionally used service.

If we look at implementing such a visualization, which includes the health of interactions and throughput of such interactions, we would be looking at something like below :

The day is not far

These capabilities are already available to users today as one of the UI features of Tetrate’s service mesh platform, using the highly configurable and performant observability and performance management framework: Apache SkyWalking (https://skywalking.apache.org), which monitors traffic across the mesh, aggregates RED metrics for both services and their interactions, continuously computes and monitors health of the services, and enables users to configure alerts and notifications when services cross specific thresholds, thereby having a comprehensive health visibility of the mesh.

With such tremendous visibility into our mesh performance, the day is not far when we at our NOC (Network Operations Center) for the mesh have this topology as our HUD (Heads Up Display).

This HUD, with the insights and patterns gathered over time, would predict situations and proactively prompt us on potential focus areas to improve customer satisfaction.

The visualization with rich historical data can also empower the Network Engineers to go back in time and look at the performance of the mesh on a similar day in the past.

An earnest implementation of such a visualization would be something like below :

To conclude

With all the discussion so far, the health of a mesh is more about how our users feel, and what we can proactively do as service providers to sustain, if not enhance, the experience of our users.

As the world advances toward personalized medicine, we’re not far from a day when my doctor will text me: “How about feasting yourself with ice cream today and take the Gray Butte Trail to Mount Shasta!” Likewise, we can do more for our customers by having better insight into their overall wellness.

Tetrate’s approach to “service mesh health” is not only to offer management, monitoring and support but to make infrastructure healthy from the start to reduce the probability of incidents. Powered by the Istio, Envoy, and SkyWalking, Tetrate’s solutions enable consistent end-to-end observability, runtime security, and traffic management for any workload in any environment.

Our customers deserve healthy systems! Please do share your thoughts on making service mesh an exciting and robust experience for our customers.

References

Blog: SkyWalking v6 is Service Mesh ready

Wed, 05 Dec 2018 00:00:00 +0000

Original link, Tetrate.io blog

Context

The integration of SkyWalking and Istio Service Mesh yields an essential open-source tool for resolving the chaos created by the proliferation of siloed, cloud-based services.

Apache SkyWalking is an open, modern performance management tool for distributed services, designed especially for microservices, cloud native and container-based (Docker, K8s, Mesos) architectures. We at Tetrate believe it is going to be an important project for understanding the performance of microservices. The recently released v6 integrates with Istio Service Mesh and focuses on metrics and tracing. It natively understands the most common language runtimes (Java, .Net, and NodeJS). With its new core code, SkyWalking v6 also supports Istrio telemetry data formats, providing consistent analysis, persistence, and visualization.

SkyWalking has evolved into an Observability Analysis Platform that enables observation and monitoring of hundreds of services all at once. It promises solutions for some of the trickiest problems faced by system administrators using complex arrays of abundant services: Identifying why and where a request is slow, distinguishing normal from deviant system performance, comparing apples-to-apples metrics across apps regardless of programming language, and attaining a complete and meaningful view of performance.

SkyWalking History

Launched in China by Wu Sheng in 2015, SkyWalking started as just a distributed tracing system, like Zipkin, but with auto instrumentation from a Java agent. This enabled JVM users to see distributed traces without any change to their source code. In the last two years, it has been used for research and production by more than 50 companies. With its expanded capabilities, we expect to see it adopted more globally.

What’s new

Service Mesh Integration

Istio has picked up a lot of steam as the framework of choice for distributed services. Based on all the interest in the Istio project, and community feedback, some SkyWalking (P)PMC members decided to integrate with Istio Service Mesh to move SkyWalking to a higher level.

So now you can use Skywalking to get metrics and understand the topology of your applications. This works not just for Java, .NET and Node using our language agents, but also for microservices running under the Istio service mesh. You can get a full topology of both kinds of applications.

Observability analysis platform

With its roots in tracing, SkyWalking is now transitioning into an open-standards based Observability Analysis Platform, which means the following:

It can accept different kinds and formats of telemetry data from mesh like Istio telemetry.
Its agents support various popular software technologies and frameworks like Tomcat, Spring, Kafka. The whole supported framework list is here.
It can accept data from other compliant sources like Zipkin-formatted traces reported from Zipkin, Jaeger, or OpenCensus clients.

SkyWalking is logically split into four parts: Probes, Platform Backend, Storage and UI:

There are two kinds of probes:

Language agents or SDKs following SkyWalking across-thread propagation formats and trace formats, run in the user’s application process.
The Istio mixer adaptor, which collects telemetry from the Service Mesh.

The platform backend provides gRPC and RESTful HTTP endpoints for all SkyWalking-supported trace and metric telemetry data. For example, you can stream these metrics into an analysis system.

Storage supports multiple implementations such as ElasticSearch, H2 (alpha), MySQL, and Apache ShardingSphere for MySQL Cluster. TiDB will be supported in next release.

SkyWalking’s built-in UI with a GraphQL endpoint for data allows intuitive, customizable integration.

Some examples of SkyWalking’s UI:

Observe a Spring app using the SkyWalking JVM-agent

Observe on Istio without any agent, no matter what langugage the service is written in

See fine-grained metrics like request/Call per Minute, P99/95/90/75/50 latency, avg response time, heatmap

Service dependencies and metrics

Service Focused

At Tetrate, we are focused on discovery, reliability, and security of your running services. This is why we are embracing Skywalking, which makes service performance observable.

Behind this admittedly cool UI, the aggregation logic is very easy to understand, making it easy to customize SkyWalking in its Observability Analysis Language (OAL) script.

We’ll post more about OAL for developers looking to customize SkyWalking, and you can read the official OAL introduction document.

Scripts are based on three core concepts:

Service represents a group of workloads that provide the same behaviours for incoming requests. You can define the service name whether you are using instrument agents or SDKs. Otherwise, SkyWalking uses the name you defined in the underlying platform, such as Istio.
Service Instance Each workload in the Service group is called an instance. Like Pods in Kubernetes, it doesn’t need to be a single OS process. If you are using an instrument agent, an instance does map to one OS process.
Endpoint is a path in a certain service that handles incoming requests, such as HTTP paths or a gRPC service + method. Mesh telemetry and trace data are formatted as source objects (aka scope). These are the input for the aggregation, with the script describing how to aggregate, including input, conditions, and the resulting metric name.

Core Features

The other core features in SkyWalking v6 are:

Service, service instance, endpoint metrics analysis.
Consistent visualization in Service Mesh and no mesh.
Topology discovery, Service dependency analysis.
Distributed tracing.
Slow services and endpoints detected.
Alarms.

Of course, SkyWalking has some more upgrades from v5, such as:

ElasticSearch 6 as storage is supported.
H2 storage implementor is back.
Kubernetes cluster management is provided. You don’t need Zookeeper to keep the backend running in cluster mode.
Totally new alarm core. Easier configuration.
More cloud native style.
MySQL will be supported in the next release.

Please: Test and Provide Feedback!

We would love everyone to try to test our new version. You can find everything you need in our Apache repository,read the document for further details. You can contact the project team through the following channels:

Submit an issue on GitHub repository
Mailing list: dev@skywalking.apache.org . Send to dev-subscribe@kywalking.apache.org to subscribe the mail list.
Gitter
Project twitter

Oh, and one last thing! If you like our project, don’t forget to give us a star on GitHub.

Apache SkyWalking – Cloud Native

Blog: Meet Horizon UI · 7/16: The Log Explorer

The stored log stream

Reading the stream

Into a single line

Pod Logs: tailing what’s printing right now

Where to go next

Blog: Meet Horizon UI · 5/16: The 3D Infrastructure Map

One scene for the whole estate

Tiers are the spine

Reading the map: cubes, zones, traffic

Alarms, and Beacon mode for incidents

The lines between things

Moving around

How it builds

Configured, not coded

Where to go next

Blog: Meet Horizon UI · 6/16: The Trace Explorer

Built for triage, not tailing

Conditions, not a query language

A distribution you can box-select

Three ways to read one trace

Inside a span

Native and Zipkin, side by side

From a slow row to the trace behind it

Where to go next

Blog: Meet Horizon UI · 1/16: SkyWalking's New Observability Console

The sidebar is your estate

Never a blank page

One new tier makes the rest possible

A 3D look at the whole thing

What’s in this series

Try it against your OAP today

Other notable points

Blog: Meet Horizon UI · 2/16: Dashboards That Adapt — MQE, Smart Widgets, and Numbers Humans Can Read

Every widget is one MQE expression

A dashboard that adapts to the entity in front of you

Numbers humans can read

Read the whole grid as one timeline

From a slow row to its trace, in one click

Pin and compare entities

The time picker moves the whole dashboard

Where to go next

Blog: Meet Horizon UI · 3/16: Topology & Service Dependency

One topology engine, repainted per layer

Cut the noise

Drill from a call into its instances

Walk the request chain, endpoint by endpoint

One service, every layer it reports through

Where to go next

Blog: Meet Horizon UI · 4/16: The Deployment Tab & BanyanDB Self-Observability

The Deployment tab: a map of one service’s own instances

BanyanDB, watched like everything else

Cluster, Container, Group

Edges that know their role pair, and a Flows table

Configured, not coded

Blog: Monitoring Envoy AI Gateway with Apache SkyWalking

The Problem: Flying Blind with LLM Traffic

Why an AI Gateway

Data Flow

Try It Locally

Prerequisites

Step 1: Start Ollama

Step 2: Start the Stack

Step 3: Run the Demo App

Step 4: View in SkyWalking UI

Cleanup

Deploying on Kubernetes

GenAI Observability Without an AI Gateway

References

Zh: 用 Apache SkyWalking 监控 Envoy AI Gateway

问题：LLM 流量缺乏统一观测

为什么选择 AI 网关

数据流

本地体验

前置条件

第一步：启动 Ollama

第二步：启动服务栈

第三步：运行 Demo 应用

第四步：在 SkyWalking UI 中查看