<p>Hey <a href="/r/golang">/r/golang</a>,</p>
<p>I've been experimenting a lot with event sourcing, explaining the concept to people and building small applications using this approach. After many iterations I've finally settled on a way of doing things and cast that into code. </p>
<p>You can find the <a href="https://github.com/dhamidi/ess">code here</a> and the <a href="http://godoc.org/github.com/dhamidi/ess">documentation here</a>.</p>
<p>The main criticism of event sourcing seems to be that it is "complex". My experience is that it is just a different, unfamiliar kind of complexity. The underlying principles are simple.</p>
<p>What I'm looking for is some feedback on the <a href="http://godoc.org/github.com/dhamidi/ess">documentation</a>, especially the <a href="http://godoc.org/github.com/dhamidi/ess#hdr-Tutorial">tutorial part</a>, as to whether it reads well and gets the point across.</p>
<p>Also, if anyone has experience with event sourcing, I'd be glad to hear about them.</p>
<hr/>**评论：**<br/><br/>jerf: <pre><p>So, first of all let me say I'm very sympathetic to the goals and general ideas here. I've been in the business world long enough to know what this is for, where it's coming from, why it's a good idea, and why every business system ends up including a half-assed version of this sooner or later if it doesn't start out with a whole-assed one, and it's neat to try to start with a whole-assed one.</p>
<p>But there's two major issues with this, I think.</p>
<p>First, perhaps ironically considering its strong business sponsor, Go is one of the more hostile languages to this sort of fundamental paradigm shift. A lot of this code is basically dynamically typed without the convenience of being a dynamically typed language. You've even got the basic <code>Variant</code> type in <code>Value</code>. That's not a criticism of this code per se, because you can't do anything about it, it's fundamental to the language. Go wants to be written in idiomatic Go, and in a lot of ways Go makes Ruby look wishy-washy about its opinions.</p>
<p>If you are interested in pursuing these ideas but are not really that tied to Go, you might want to consider Haskell, whose type system will eagerly play along with you, and even enforce things like "modifications to the underlying values can only be done via events being processed in the sanctioned manner", all the while maintaining full strong typing.</p>
<p>Secondly, if you are tied to Go, there are ways of making this more native Go, and simplifying as well.</p>
<p>First, pull the data types out into a subpackage. It clutters up the godoc.</p>
<p>Second... I don't even know exactly how to put this, but... this proscribes a lot more specificity than it really should. <code>Command</code> is probably one clear place... <code>Command</code> should be an interface, rather than a concrete type. Because <code>Command</code> is a concrete object, you find yourself implementing getters (which because they are generic, can not be type safe), an overly-generic setter, the <code>String</code> method (which itself should be an interface and looks an awful lot like TextMarshaler), and so on. What you probably ought to do is turn it into an interface. You can also offer a stock object that easily embeds into implementations that implements the boring stuff like Acknowledge and and AggregateID, but leaves the implementer of <code>Command</code> as a full object, which is allowed to be type-safe.</p>
<p>I'm not reading your source code, but I'm going to assume that the methods that your system cares about are <code>Acknowledge</code>, <code>AggregateId</code>, <code>Execute</code>, <code>Receiver</code>, and <code>String</code>. Of those, a simply-embedded object could be provided for default implemantations of <code>Acknowledge</code>, <code>AggregateId</code>, and <code>Receiver</code> (I think), leaving just <code>Execute</code> and <code>String</code> to be implemented normally.</p>
<p><code>CommandDefinition</code> would actually become part of <code>Command</code>. What it would then be is the specification of the metadata necessary to deal with the command. Basically, you'd be specifying "class methods" in your interface, a powerful technique:</p>
<pre><code>type Command interface {
Name() string
Construct(Form) Command
// ...
}
var commands = map[string]Command
func Register(c Command) {
commands[c.Name()] = c
}
</code></pre>
<p>Notice that <code>Construct</code> method there... this is also a powerful idea. If you put a "class method" in your interface whose job it is to take some input in and return a new value of the "same type" (not expressible in the Go type system but you can document this requirement... if indeed it really <em>is</em> a requirement), you don't need to try to create a generic instantiator yourself... you foist it off on to the object. Which is where it belongs anyhow.</p>
<p>It may be difficult to see my point at first, but I think you'll find that if you propagate this idea through, an entire layer of code will melt away, a lot of the type safety will come back, and it will be simpler to use the library because instead of preparing objects of the correct shape and then jumping through hoops to use it, you'll only have to prepare objects of the correct shape. I think there will still be a bit of the "dynamic language embedded in Go" in the end... but it's possible that with just a dash more of cleverness it might be able to be entirely chased away.</p></pre>farmingyaks: <pre><blockquote>
<p>why every business system ends up including a half-assed version of this sooner or later if it doesn't start out with a whole-assed one, and it's neat to try to start with a whole-assed one.</p>
</blockquote>
<p>This strikes a chord with me. I've been under the same impression but failed to put it into words as clearly as you just did.</p>
<blockquote>
<p>A lot of this code is basically dynamically typed without the convenience of being a dynamically typed language. </p>
</blockquote>
<p>This occurred to me as well while writing the code. At times it felt like implementing the runtime for a dynamic language. However, the dynamism is restricted to the boundaries of the system (i.e. capturing / processing input parameters). This made me realize that I actually want to model dynamic behavior which Go's type system allowed me to. The standard library gives examples for this in the <a href="http://godoc.org/flag" rel="nofollow">flag</a> and <a href="http://godoc.org/reflect" rel="nofollow">reflect</a> packages. Maybe modelling commands should talk more inspiration from the <code>flag</code> package (which is doing the same thing -- capturing typed input parameters).</p>
<blockquote>
<p>you might want to consider Haskell, whose type system will eagerly play along with you</p>
</blockquote>
<p>I did consider Haskell and play around with it from time to time. Then I look at Go's tooling and fall back to it (documentation, editor integration, the oracle, gofmt, go get, ...). Go doesn't make me think about the tool and gets out of the way. Haskell is still too hard to use for me. Ultimately this is a question of familiarity. At the moment my focus is on producing value with the tools I know (Go, Ruby, JavaScript) instead of learning yet another technology, as nice and useful as it may be.</p>
<blockquote>
<p>First, pull the data types out into a subpackage. It clutters up the godoc.</p>
</blockquote>
<p>Good point. I've created an issue on GitHub for this to keep track of it: <a href="https://github.com/dhamidi/ess/issues/1" rel="nofollow">https://github.com/dhamidi/ess/issues/1</a></p>
<blockquote>
<p>this proscribes a lot more specificity than it really should</p>
</blockquote>
<p>That's a valid point? How much specificity should it proscribe? The mental model used by this packages is not unlike <code>net/http</code>: there is some object that processes requests passed to it. In <code>net/http</code> these requests are of a concrete type, <code>*http.Request</code>. That was the motivation for using a concrete type for implementing commands. I don't see the value of <code>Command</code> being an interface yet.</p>
<blockquote>
<p>Because Command is a concrete object, you find yourself implementing getters (which because they are generic, can not be type safe)</p>
</blockquote>
<p>My first reaction was to say "I don't see how to get rid of those" but then it occurred to me that code that produces command objects and the code that consumes them most likely live in the same package. Your suggestion definitely warrants some thought. Given a default implementation of command, this change shouldn't impact a lot.</p>
<blockquote>
<p>instead of preparing objects of the correct shape and then jumping through hoops to use it, you'll only have to prepare objects of the correct shape</p>
</blockquote>
<p>I'm not sure I'm following on this one, but I guess it will become clear once I'm making the changes you suggested.</p>
<p>Thank you for taking the time and writing such an elaborate reply!</p></pre>jerf: <pre><blockquote>
<p>Given a default implementation of command, this change shouldn't impact a lot.</p>
</blockquote>
<p>It is a technique I consider a bit underused in Go at the moment. Go doesn't have traditional inheritance, so in particular you don't get one of my favorite patterns for this, the Template pattern, but you get some elements of it with composition... if you need an object to have a certain behavior you can provide it as an object and implement it for people, allowing them to compose it in from the bottom, rather than driving the rules from the top. It means they might be able to screw up the implementation but they might also be able to extend it successfully too, so... it's hard to make the call on which is better. The combination of a required interface and a provided object that implements as much of it as is sensible is at least a reasonable compromise with the Template pattern.</p>
<p>It's also the sort of thing that leads me to think there's another such pattern or three waiting to be discovered by people. Go is mostly OO so it's mostly OO patterning, but there's enough difference that a few other things might be optimal that one would not normally consider in Java.</p></pre>farmingyaks: <pre><blockquote>
<p>Go doesn't have traditional inheritance</p>
</blockquote>
<p>I am very grateful for that. Traditional inheritance makes it all too easy to end up with a fragile base class that is hard to change and results in a lot of action at a distance. </p>
<blockquote>
<p>The combination of a required interface and a provided object that implements as much of it as is sensible</p>
</blockquote>
<p>There's also the option of embedding an interface into a struct, to get empty implementation for all of the interface's methods. Calling any unimplemented methods results in a panic. That comes pretty close to the template pattern. Example: <a href="http://play.golang.org/p/D_FYgXxRcV" rel="nofollow">http://play.golang.org/p/D_FYgXxRcV</a></p></pre>jerf: <pre><blockquote>
<p>That comes pretty close to the template pattern.</p>
</blockquote>
<p>I'd call that an abstract method. There's nothing particularly driving it to be part of a template, it's just, unimplemented.</p></pre>farmingyaks: <pre><blockquote>
<p>First, pull the data types out into a subpackage. It clutters up the godoc.</p>
</blockquote>
<p>I just tried doing that. Turns out that it leads to a cascade of changes due to import cycles, so doing this is on hold. </p>
<p>Probably this problem disappear once the commands have been refactored.</p></pre>joeshaw: <pre><p>Great job for using godoc for more than just reference material. Having a tutorial in there is a welcome sight.</p></pre>farmingyaks: <pre><p>Thank you! <a href="http://godoc.org/github.com/streadway/amqp">This package</a> set my standards for good godoc documentation.</p></pre>socceroos: <pre><p>Looking good! I'm very interested in event sourced applications.</p>
<p>I've only cast my eye over your documentation, but it doesn't seem that you're doing entity snapshotting at all for your event stream?</p>
<p>I'm not sure if Martin Fowler covers it, but if you've got 200,000,000 events for an Entity, you don't want to replay them all to get your Entity state up-to-date, it's best for the Event stream to have a recent application state snapshot to start from and then just replay all the events that have happened since that snapshot.</p>
<p>Otherwise, I love it!</p>
<p>Edit: to clarify, I'm not saying that you would discard your event stream prior to the snapshot, just that there would be regular snapshots to shorten the required replay events for an entity.</p></pre>farmingyaks: <pre><blockquote>
<p>Looking good!</p>
</blockquote>
<p>Thank you!</p>
<blockquote>
<p>I'm very interested in event sourced applications.</p>
</blockquote>
<p>Glad to hear that! Have you already gotten your feet wet and tried out the idea somewhere? </p>
<blockquote>
<p>I've only cast my eye over your documentation, but it doesn't seem that you're doing entity snapshotting at all for your event stream?</p>
</blockquote>
<p>That is correct.</p>
<blockquote>
<p>I'm not sure if Martin Fowler covers it</p>
</blockquote>
<p>He does cover it. The general consensus on snapshots appears to be "only do them when you need to". Unfortunately I can't cite any references, this is extracted from a lot of material I found by googling for "event sourcing".</p>
<blockquote>
<p>Edit: to clarify</p>
</blockquote>
<p>No confusion caused :)</p>
<p>I haven't had the need for snapshotting yet, hence why it is missing here. I'm trying to avoid putting too many things into this package, lest it grows into a framework (it already is a bit too "frameworky" by prescribing a lot of structure).</p>
<p>One thing that worked well in one of the many discarded prototypes was loading the aggregate state from a file on disk and only reconstructing the state from events if the file did not exist. The file was updated after each successfully executed command.</p>
<p>For my little toy example at the time that worked well. It is essentially a cache, so you get all the usual caching problems (i.e. possibly stale data) with this approach. If need be, I'll probably implement that instead of a stream of snapshots, as just storing the current state on disk is less complex.</p></pre>socceroos: <pre><p>Yeah, a single item on disk is more than enough. If you want to help solve the stale problem, just version your snapshot. If there are any events post that version then you could replay those, thus solving the state issue.</p></pre>farmingyaks: <pre><p>Thanks, somehow I didn't think of that!</p></pre>alecthomas: <pre><p>Great stuff. I think more libraries and articles about event-based systems would be great. At a high level it feels like your library could have leveraged channels more aggressively though. They do fit event-based models very well.</p>
<p>Somewhat related, I've been researching and implementing distributed event-based data synchronisation (primarily for N-way synchronisation between mobile clients and Go backends) and came across a bunch of articles you might find interesting:</p>
<ul>
<li>An <a href="https://medium.com/@btaylor/react-with-c-building-the-quip-mac-and-windows-apps-c63155c1531b" rel="nofollow">interesting article</a> by Bret Taylor on how they perform data synchronisation between the Quip backend and their various clients.</li>
<li><a href="https://www.meteor.com/ddp" rel="nofollow">Meteor DDP</a> is a protocol for synchronising incremental model changes, and there are some non-Meteor-backed implementations.</li>
<li><a href="https://github.com/mikeal/pouchdb" rel="nofollow">PouchDB</a> is another approach, but ties you to CouchDB.</li>
<li>The <a href="http://www.reactivemanifesto.org" rel="nofollow">Reactive</a> model.</li>
<li><a href="http://www.confluent.io/blog/making-sense-of-stream-processing/" rel="nofollow">This is</a> a pretty good overview of the various data flow approaches that are currently popular.</li>
</ul>
<p>As an aside, <a href="https://nats.io" rel="nofollow">NATS</a> doesn't address this problem specifically, but it's a great "event bus" for building these kind of systems out.</p>
<p>My current solution is somewhat similar to Quip's approach, but not as general purpose due to the way the system evolved. But the great thing about event-based systems is how responsive they are. A user initiates some action on their client and the events flow through to the server, and back out to all subscribed clients. Very nice experience.</p>
<p>An open-source general purpose solution to distributed data synchronisation would be great. If I had infinite time and money I'd love to build it, but alas. :)</p></pre>farmingyaks: <pre><p>The link collection is great, thank you! NATS is new to me and it being written in Golang makes me want to investigate it more. It should be easy to integrate.</p></pre>egonelbre: <pre><p>Pretty much jerf covered most of the issues. My main problem with your approach is the dynamic typing.</p>
<p>Here's one of my attempts <a href="https://github.com/egonelbre/event" rel="nofollow">https://github.com/egonelbre/event</a>. Of course, I'm still not happy with -- feels like I'm missing something obvious how to make it better.</p></pre>farmingyaks: <pre><p>I had a cursory look at the code and have to say it looks much simpler than what I have wrought. How did you develop it / with which part did you start?</p></pre>egonelbre: <pre><p>Mostly the process for it is: write it -> unsatisfied -> try to find better approach -> rewrite it -> unsatisfied -> try to find better approach -> rewrite.... etc. I've rewritten it at least 5 times.</p>
<p>The first one pretty much was verbatim translation of <a href="https://github.com/gregoryyoung/m-r" rel="nofollow">https://github.com/gregoryyoung/m-r</a> to Go.</p>
<p>The "try to find better approach" mainly involves figuring out what to call things to make it clearer and remove the dependencies. E.g. such as the <code>event.Aggregate</code>... whether to call it Recorder and remove the Id, Version from it and use callbacks etc. Here's what I have left from those experiments <a href="https://dl.dropboxusercontent.com/u/4300994/Go/events.zip" rel="nofollow">ZIP</a>.</p></pre>
