Go Tracing Roadmap

Go Tracing Roadmap golang 分布式追踪的路演, 今后可能直接从语言层面支持这些特性，golang真是太棒了, 原始文档在 google doc 上，可能有些同学网络不太方面，所以转一份在这里。

Jaana B. Dogan (jbd@golang.org), May 15, 2017

This document summarizes the state of the distributed tracing related work for the Go ecosystem. It is recommended to read the Dapper (google 2010年发表的一个关于分布式追踪系统架构的论文) paper before reading this document to understand the higher level concepts and terminology in distributed tracing.

Dapper, a Large-Scale Distributed Systems Tracing Infrastructure

Background

Go ecosystem provides a large set of solutions for distributed tracing such as x/net/trace, OpenTracing, tracing backend-specific clients, etc. Few months ago, we wanted to see if we can unify the existing solutions and use a single set of unified API and utilities to back the current tracing providers. Our main goals were to:

• Defrag the ecosystem and make instrumentation portable regardless of which tracing backend users depend on.
• Enable the library ecosystem to utilize the current trace context if any without having to depend on a specific tracing backend.

It soon became clear that there are two critical barriers to achieve a common API:

• Tracing is a multi language problem. Our design decisions needs to be reflected in other language ecosystems to provide a proper end-to-end tracing solution. As the Go authors, it is beyond our scope.
• Tracing backends and their data models are still widely different and achieving a common API that covers 80-90% of the use-cases is hard.

Additional to these critical barriers, depending on the initial research and several 1:1s we made with some of the Go users, our conclusion is that language ecosystems are suffering from the lack of a common data model and wire format rather than the lack of a common tracing library.

As we were having the initial conversation on tracing, there has been an ongoing effort in the tracing community to establish a trace context standard and a common encoding/decoding algorithm for span identifiers and the propagated labels.

As a result, our focus has shifted from achieving a common API to supporting open standard initiatives and making sure Go is well presented. We are currently collaborating to the broader tracing standardization effort. The ideal outcome of our collaboration will:

• Allow users to propagate traces among Go and non-Go services regardless of the backend.
• Allow users to be able to propagate traces within ecosystem libraries. Currently, trace contexts are dropped during medium/transport changes, e.g. when an HTTP request/response is wrapped by a Go API or a gRPC request is followed by an HTTP request. Provide a minimal Go building block to represent a trace context and enable the ecosystem propagating traces using that building block.
• Allow users to be able to discover the current trace context from a context.Context.
• Start discussion about where standard library can benefit from out-of-the-box tracing support.
• Establish utility libraries to make it easier to enable tracing out-of-the-box without manual instrumentation: Tracing-enabled handlers, http.Client, and similar.
• Enable a solution that scales both small- and large-scale in terms of performance characteristics.

Roadmap

Our roadmap is currently primarily blocked by the the emerging standards group because our priority is to align well with the tracing community and adopting their solutions rather than inventing anything Go-specific. Our next steps can be summarized as:

• Work with the standards group to represent Go user’s concerns. If standards are not achieved or widely adopted, a backend-agnostic generic Go-specific solution will be considered.
• Implement a trace package with the standard trace context representation and propagation primitives (e.g. propagating in context.Context and as an HTTP header) .
• Open a discussion in the community to utilize the trace propagation package, help current tracing libraries to adopt these types where possible. Open discussion for the standard library tracing support.

The following sections will give more details about the specifics of the work required for some items.

Trace context

In a tracing system, trace context is the smallest state carrying object. It identifies a unit of work and carries labels that represent additional metrics/data in the scope of a the work.

Each tracing system has a trace context format and a spec how it should be carried in wire (e.g. via an HTTP request). An example of a trace context propagated via an HTTP header:

    GET /service HTTP/1.1
    Trace-Id: 4bf92f3577b34da6a3ce929d0e0e4736/e1;o=0

Today, trace context formats are often incompatible with each other; a context generated from a tracing backend cannot be parsed by another one. Compatibility issues restrict users to depend on a single backend end-to-end and use backend-specific instrumentation libraries to utilize the trace context.

Until a standard establishes, the trace context can only be represented by an identifier and a labels byte slice that both can later be parsed by the backend specific instrumentation libraries; given backend-specific library is the only layer knows about the encoding/decoding algorithm.

package trace

// Span represents a unit of work.
//
// ID identifies a span globally in a tracing system.
// Annotations return the labels propagated with the span.
// Encoding algorithm might be tracing-backend specific.
type Span interface{
    ID() []byte
    Labels() []byte
}

This overly generic representation of trace context is not very useful, especially for the library ecosystem that wants to instrument code (e.g. creating new child spans, attaching additional labels) without any dependencies to specific backends.

At this point, we propose to contribute to the establishment of the industry-wide wire standard and draft our design according to this standard rather than inventing a Go-specific representation. The outcome of this work will be:

• A span context encoding format standard that is well accepted by top tracing backends.
• A concrete span context type, and encoders/decoders in Go.

Propagation via context.Context

Being able to discover the trace contexts from the current context also is widely required. By blessing a canonical way to access trace context, we are enabling different libraries cooperate on the same trace.

package trace

// NewContext returns a derived span from the current context.
func NewContext(ctx context.Context, s Span) context.Context

// FromContext returns the span contained in the context or nil.
func FromContext(ctx context.Context) Span

Standard library support

If the proposed trace package gets into the standard library, with the first-class availability, we may be able to enable tracing support for some of the existing packages.

For example, net/http package can automatically extract the trace context from incoming HTTP request’s headers and put it in the request context. Similarly, it can inject trace context header if the request context contains a trace context.

We will open discussions and follow up with proposals once there is a trace package.

Caveats

• We cannot foresee when an open context trace standard will be mature and finalized. Before such standard is finalized, we are blocked on designing the trace context type.
• There are already several custom trace context types contributed by the community, e.g. OpenTracing’s SpanContext. Unifying the APIs around the new type will require a lot of consensus and work.
• Propagation primitives only provide a building block. Without a fully featured solution, it is hard to pitch the building block’s itself. We need to work closely with the current tracing libraries and make sure the building block is utilized before we
• If the tracing community cannot achieve an open standard, our generic trace context type ([]byte, []byte) is not very useful.