GitHub has announced that it will be using the open source project OpenTelemetry to manage telemetry data. Up to now, statsd – a syslog format for plain text logs – and OpenTracing for request traces have been used for metrics.
Apparently, each component did its job well, but interoperability was a challenge, according to the announcement on GitHub. For example, different parts of the GitHub infrastructure used different statsd dialects, which had resulted in the telemetry code having to be tagged with special characters in various places. Different components can have different "Vocabulary" use for similar observability concepts, making it harder to find.
Unity with Open Telemetry
GitHub seems to have found a way to standardize the use of telemetry data with OpenTelemetry, an open source project that provides a uniform and vendor-independent signal. The Open Telemetry Protocol (OTLP) also allows telemetry signals to be interconnected.
For now, the code hoster wants to focus on support for the OpenTelemetry tracing signal. Tracing is well suited as a main entry point in a world of distributed systems. The community behind OpenTelemetry has already begun writing libraries that automatically add distributed tracing functions to other libraries, which GitHub in turn relies on, according to the blog post. The plan is to build up an automatic correlation of the existing signals with OpenTelemetry tracing as a basis. By automatically adding trace identifiers to log lines, request traces can be linked to logs, for example. According to the blog entry, the responsible development team wanted to automatically create dashboards, alerts and tools for teams based on these signals.
What is OpenTelemetry?
The communities behind the open-source OpenCensus and OpenTracing projects joined forces in 2019 and have since been running OpenTelemetry. Under the umbrella of the Cloud Native Computing Foundation (CNCF), the project has increasingly emerged as a new standard for telemetry data observability. OpenTelemetry provides a unified set of language-specific APIs, SDKs and other components for collecting and processing distributed traces, metrics and related metadata from any type of application. Developers will be able to record and analyze observability data from their applications with little effort.