<p>tl;dr Do you use protobufs to represent your data - even at rest and not only in transmission? If so do you use vanilla protobufs or adopt alternatives such as <a href="https://github.com/gogo/protobuf">gogo/protobuf</a> or use other solutions such as <a href="https://capnproto.org">capnproto</a></p>
<p>I've been playing around with a simplistic toy project; Simplistic REST API (toying with gRPC too) which interacts with GCP's Datastore. I decided to try and fully adopt Protobufs for the project as a way to further learn the more interesting features of Protobufs.</p>
<p>At first I started with "proto2" syntax as I was attracted to the idea of default values and optional vs required - later found out optional and required are compiled the same way with Go so it didn't matter, but default values still were nice.</p>
<p>This has worked great for the API side, but as soon as I began to try and integrate with Datastore (<a href="https://godoc.org/google.golang.org/appengine/datastore">Package</a> for the curious) I found that the compiled structs were pointer heavy and Datastore can't store any pointers of types. By updating to the "proto3" syntax you lose the optional vs required and the default values but it causes all non-struct values to be non-pointers which is good. But structs (messages within messages) are still pointers and as such it still doesn't work with Datastore.</p>
<p>I've looked into gogo/protobuf a bit but the additional syntax felt out of place and a little wrong to me - but they seem to be able to strip pointers so might be my best option.</p>
<p>What other Go + Protobuf recommendations do people have?</p>
<hr/>**评论：**<br/><br/>nsd433: <pre><p>I did lots of enhancements to the Google Go protobuf package to support all the missing datatypes[1] last year. It would let you handle the struct-in-struct that you need.</p>
<p><a href="https://github.com/mistsys/protobuf3">github.com/mistsys/protobuf3</a></p>
<p>protobuf has been fine as an encoding for data "at rest" for me, but it's only been a year.</p>
<p>[1] Google's protobuf only supports the types which their protoc tool emits, which isn't everything. No arrays, no slices of structs, etc. I wanted to go from existing Go structs, which used all the features of Go, to a marshaler, an unmarshaler, and a .proto to share with others.</p></pre>siwu: <pre><p>We our all our modelling for all platforms in with protobuf 3 (Go, Objc, Android, Java, Swift, Python) and use it in place of struct when representing object of our data model.</p>
<p>For go we use gogo.</p>
<p>Added benefit is serialisation is free.</p></pre>soapysops: <pre><p>If you store stuff in datastore, you should use datastore's format so you can do proper querying and indexing. You should not use protobufs.</p>
<p>Protobufs are useful if you have to create messages of data to send to something or to store somewhere if all you're allowed to store is a string (like if you were implementing a database on top of leveldb, or if you were writing each message to a file). That's why they are used in grpc, to provide a standard for constructing messages.</p>
<p>You probably should not use the protobuf generated structs in most of your program's code. Generally you'll have some sort of request handler (acting as a view, in MVC), which treats the protobuf message as part of the request. Then you deconstruct the message and use its components to do work. It's not a good idea to pass the original protobuf generated struct around because it introduces unnecessary coupling and makes the entire message's data available to functions that probably don't need to know everything.</p></pre>Mteigers: <pre><p>Message visibility is a good point I hadn't considered.</p>
<p>Also Datastore has no issues storing entire structs that are entirely queryable, it chokes on pointers which Protobuf relies on.</p></pre>fakeNAcsgoPlayer: <pre><p>Don't use protobufs, every thing is pointer for some stupid reason, obviously this means a lot of CPU spent in encoding and decoding. </p>
<p>Use gogoprotbuf because Google is lazy to optimize protobufs for Go.</p>
<p>It may not matter if you are doing just single digits kQPS. If you want more throughput with better latency, avoid protobufs and use gogoprotbuf. </p>
<p>Good luck</p></pre>robpike: <pre><p>Use them or not, but you are wrong when you say "everything is a pointer". For some time now the supported version of protobufs uses the proto3 API, which does not use pointers for scalars.</p>
<p>And the reason they were pointers in Go (only) was not stupid, it made sense given the contraints of fitting a C++-oriented model onto Go. But with proto3 one of the most limiting constraints (the "has" property) was lifted and the pointers are no longer needed.</p></pre>siwu: <pre><p>proto3 only uses pointers for messages</p></pre>klauspost: <pre><p>I use MessagePack, for most cases.</p>
<p>The main thing I like is that I can define my data structures in Go, and changing the data structure is backwards compatible. There is at least 3 very good Go packages with different features.</p>
<p>At least for me the convenience far outweighs the extra space the format uses for field names.</p></pre>
