<p>I've not yet come across some recommendations for how granular you should get with distributed tracing. I know this isn't strictly a Go question, but I am working with Jaeger via OpenTracing in Go.</p>
<p>I'm working on some services that use <a href="https://blog.golang.org/pipelines" rel="nofollow">pipelines</a> to process a bunch of data. Usually this data comes from a third-party, so the processing is the transformation of the data returned from a third-party back into a normalised format. Lots of items are returned, so the pipeline approach is used to ensure that as many items are processed as possible before a timeout. If we ever do reach a timeout, at least we can have finished processing some of the data, and there's something to send back normalised, even if it's missing some things.</p>
<p>The problem with this approach is that each of these transformations requires calls to downstream services, or database queries. These are all the kinds of things that I'd like to be able to trace, along with how long each stage in the pipeline took overall, from the first item being processed to the last.</p>
<p>When I've added tracing that does accomplish this goal, I've ended up with thousands of spans. Even with the newer Jaeger UI that has better performance it's a bit much...</p>
<p>I'd been toying with the idea of swapping the approach to batch things up instead so there were larger chunks of work to trace, but there are too many advantages for my use-case in pipelines.</p>
<p>I understand that in production use only <em>some</em> of the requests would actually be sampled in Jaeger, as long as I use probabilistic sampling. That helps to alleviate some of my concern, however; what I'd prefer really is some way to say, when a request is done "Was this request slower than we expected? Yes? Okay, then this trace is worth sending over, even if it does have thousands of spans" (on top of the probabilistic sampling).</p>
<p>I'm keen to hear about how other people may have tackled similar issues before.</p>
