Passing complex nested structures to function. Pointers, by-value or deep copy?

<p>I&#39;m having hard time figuring out a perfect way how I should design complex data structures in go. I often come up with a design that stores some kind of state in a deep structure. Assuming this struct contains only other structures as values (not pointers), everything is easy: I can pass this structure by-value to channels, goroutine thread safety is ensured and I when I need to modify something inside I can pass it by pointer and I&#39;m happy.</p> <p>However the problems starts when I have some kind of subset of my data which might not be present in every situation. If I use pointers to store this it makes sense that the pointer can either be NULL or that it contains a pointer to the data which is available.</p> <pre><code>type Substate { Foobar int } type State { Something *Substate } </code></pre> <p>Now the problems start right away: I can&#39;t no longer pass a State by-value to a channel unless I manually make a deep copy or I risk breaking thread safety. Go doesn&#39;t have a built-in method to do deep copies, so I need to either build them on my own (which gets complex as the structures evolve to contain more nested structures), or that I need to do some kind of dirty tricks like serialise the structure into intermediate format (for example JSON).</p> <p>The only other trick I have seen is to store the optional structures by-value and then store a boolean value next to it indicating if the structure is there or not:</p> <pre><code>type State { IsSomethingPresent bool Something Substate } </code></pre> <p>Pretty? Don&#39;t think so and this doesn&#39;t help if I need to store Maps or Slices. How have other programmers solved these kind of problems?</p> <hr/>**评论：**<br/><br/>egonelbre: <pre><p>What is the actual context where you are having this problem? There might be other solutions to the problem, i.e. using mutexes/goroutines in some way.</p></pre>jerf: <pre><p>&#34;It depends.&#34;</p> <p>First, you are clearly aware of this on at least some level, but it&#39;s worth bringing it out explicitly: Here&#39;s where the dragons live. Fine-grained shared-state concurrency is how threading blows up in your face. Unlike some other languages like Erlang, Go won&#39;t actually stop you at the language level from using it; it &#34;only&#34; provides some tools to help with avoiding it.</p> <p>The most &#34;Go-like&#34; answer is to figure out how to break the State up into pieces, give the pieces to the goroutines that need them, and do any necessary cross-communication between the pieces via communication with simple messages. Messages should either be value-only, or be completely transferring ownership of the full message to the target process. Bear in mind that as long as you fully transfer ownership, you can even move bits of what is now the *State around between goroutines. (Just don&#39;t make the mistake of handing something off, then keeping a reference in the original goroutine.) Big monolithic *State is a code smell.</p> <p>That said, while once you get the hang of it that&#39;s not necessarily all that hard, it&#39;s hard to retrofit onto an existing design, and it does take some practice to learn to pull off. I learned in Erlang, where you haven&#39;t got much choice.</p> <p>Further, code smell or not, sometimes that&#39;s not practical. The next most &#34;Go-like&#34; answer is to wrap your entire State behind a goroutine that owns it, and communicate with that state only with messages and channels. Any bit of the *State handed back out should be copies of the relevant data. You can wrap that behind an object that offers all the methods, and contains nothing but enough state to do the channels/locking and a reference to the state in question.</p> <p>I think a lot of programmers will react with concern about performance issues there, because now you can only manipulate that state with effectively one CPU at a time, but generally if you think about it, &#34;one CPU&#34; for a given state like this turns out to be wildly more than enough, compared to everything else going on, and this would fall into the &#34;wait until the profiler actually says this is a problem&#34; class of problem. If not, well, you may have to try harder to implement my first suggestion.</p> <p>The final solution, which is one I&#39;ve used before but is definitely the most dangerous, is the Big Lock solution; put a big lock around the *State. This is the last choice for me because while taking <em>one</em> mutex in a goroutine is generally a safe operation, taking <em>two</em> is when you start summoning C&#39;thulu on your program, and if your API is built around taking one to start with, you&#39;re halfway there... your users need to be much more careful about what they do. A server goroutine that is responsible for your state basically functions as a goroutine that takes one implicit lock, and won&#39;t take any more if written correctly, so it&#39;s much harder to write deadlocks if you correctly isolate the goroutine.</p> <p>Generally I confine this solution to something where my methods all look like:</p> <pre><code>this.Lock() defer this.Unlock() // no more than two or three lines of operation on my state return result </code></pre> <p>The more complicated the commented out code gets, the more you&#39;re playing with fire.</p> <p>Under no circumstances, if you value your sanity, should you accept any sort of closure into such an API. That&#39;s a Dark Side sort of solution, &#34;forever dominating your destiny&#34; etc.</p> <p>If the performance chips are really down, fine-grained locking in the old style is still available. It&#39;s still as inadvisable as ever, but, if you need it, you need it. Nevertheless, I&#39;d strongly recommend at least the State server solution and holding off on this until you&#39;ve profiled everything and optimized everything else to within an inch of its life and it&#39;s still not enough for some reason.</p></pre>Garo5: <pre><p>Thank you for a well written response. Luckily my system isn&#39;t that bad and not that performance critical that I would need to sacrifice elegance and code structure for speed.</p> <p>But you did a good job on reminding me on the benefits and the Go ideology of communicating with messages. I&#39;ve already sliced my code into different goroutines doing different parts in subsystems, but I bet that I can easily do several refactoring rounds to see how the modules could be made better and the access to shared state could be minimized.</p> <p>I have had my share of problems and bugs in the dark times of old (pre-boost) c++ multithreading, so I&#39;m not keen of repeating any of those issues. I&#39;ve actually been surprised how little amount of libraries explain how/if they are thread/goroutine safe, so usually the best bet has been to assume that they&#39;re not and work around that limitation in the code with channels.</p></pre>similarnuclear: <pre><p>I also ran into this problem and chose the serialization hack to create a Clone() method on my top level data structures, using encoding/gob. I send Clone()&#39;d copies the same data structure to many goroutines from one.</p> <pre><code>package main import ( &#34;bytes&#34; &#34;encoding/gob&#34; &#34;fmt&#34; ) type Substate struct { Foobar int } type State struct { Something *Substate } func (s *State) Clone() (*State, error) { var buf bytes.Buffer enc := gob.NewEncoder(&amp;buf) dec := gob.NewDecoder(&amp;buf) if err := enc.Encode(s); err != nil { return nil, err } var cp State if err := dec.Decode(&amp;cp); err != nil { return nil, err } return &amp;cp, nil } func main() { state := State{&amp;Substate{200}} state2 := state state3, _ := state.Clone() fmt.Printf(&#34;state.Something is %p\n&#34;, state.Something) fmt.Printf(&#34;state2.Something is %p\n&#34;, state2.Something) fmt.Printf(&#34;state3.Something is %p\n&#34;, state3.Something) } </code></pre> <p><a href="http://play.golang.org/p/aM8CZMHrQ1" rel="nofollow">http://play.golang.org/p/aM8CZMHrQ1</a></p> <p>I&#39;d also like to know how to better share memory by communicating <strong>safely</strong>.</p> <p>I build all my programs in development with <code>-race</code> and have caught myself making errors (like map accesses from multiple goroutines), which is scary.</p></pre>thockin: <pre><p>Gob gets it wrong sometimes. I think the case we hit was not differentiation between a nil pointer and the zero value of that painter&#39;s pointee.</p> <p>And reflection is really slow. Now we use code generation for deep copy routines. It feels so dirty.</p></pre>Garo5: <pre><p>What kind of code generator? I tried to google around but found nothing. And I agree, I don&#39;t like code generation at all, it just makes the entire compile phase more complex than it should be.</p></pre>thockin: <pre><p>We wrote one. I think it is not committed yet, but will be soon .</p></pre>Garo5: <pre><p>Yeah. I&#39;ve also used the encoding/gob trick, but it feels really stupid to do so, but it does work. I haven&#39;t benchmarked how big penalty there is to use the encoding/gob when compared to a manually written deep copy routing without reflection, but I&#39;d guess that it usually isn&#39;t too much.</p></pre>klauspost: <pre><p>encoding/gob is not very performant, since it is primary a streaming protocol. Many other serializers are actually faster, and I would expect that it would be at least an order of magnitude faster to copy.</p> <p>Maybe this <a href="https://godoc.org/code.google.com/p/rog-go/exp/deepcopy" rel="nofollow">deepcopy</a> package can help you.</p></pre>