Premature GC on file handles

<p>I&#39;ve run into this problem a few times and each time it really surprises me. So I thought I would share in case others run into the same thing.</p> <p>The gist of the issue is that Go&#39;s garbage collector will collect objects that are still in scope if they aren&#39;t used any more. Normally collecting an object that isn&#39;t used is fine, but if that collection has some side effect (such as closing a file handle) it can cause surprising behavior.</p> <p>Ok, here&#39;s what I was trying to do. My program consisted of a parent process and a child process. The parent process needs to give the child some bits of configuration. My first thought was to use something like pipe(2) which you can do in Go using os.Pipe() and exec.Cmd&#39;s ExtraFiles. Basically the parent process creates a pipe and hands the read end of it to the child process as an ExtraFile. The parent writes the config that the child needs to the pipe and the child reads it before starting to do any work.</p> <p>This also has the nice property of easily signaling to the child process that the parent process has exited. In my case if the parent process exits I want the child process to exit also, so the child can read from the pipe until it gets an EOF and then exit.</p> <p>In practice I was seeing the child process exit claiming that the pipe had been closed, but the parent hadn&#39;t exited.</p> <p>Here is an example program demonstrating the problem: <a href="https://play.golang.org/p/HzKZea48EN" rel="nofollow">https://play.golang.org/p/HzKZea48EN</a></p> <p>If you run this with GODEBUG=gctrace=1 you will see there is a GC that occurs in the parent process right before the child process exits. If you run it with GOGC=off the child process does not exit.</p> <p>Go&#39;s garbage collector intelligently identified that the writer object was no longer used by anything and therefore could be collected, even though the program was actually depending on it to keep the descriptor open.</p> <p>My solution was to add a defer to close the write side of the pipe. That keeps a reference to the file object so it doesn&#39;t get collected.</p> <p>I find it quite unintuitive that a variable that is still in scope can be garbage collected. Perhaps it is because I&#39;ve worked in other languages in which this doesn&#39;t occur. I feel like this gotcha could be documented better, but I&#39;m not really sure where that documentation belongs. In any case I hope this helps when someone else runs into this.</p> <hr/>**评论：**<br/><br/>dominikh: <pre><p>Personally I wish that *os.File didn&#39;t have a finalizer that closed the handle. I feel that it hides buggy code (i.e. code that forgot a Close) and harms code like this.</p></pre>rsc: <pre><p>FWIW, me too. :-(</p></pre>weberc2: <pre><blockquote> <p>I feel that it hides buggy code (i.e. code that forgot a Close) and harms code like this.</p> </blockquote> <p>I tend to agree, though I&#39;m uneducated. Do you know of any bugs filed against Go for this reason?</p></pre>dominikh: <pre><p>It came up before <a href="https://github.com/golang/go/issues/9046" rel="nofollow">in the context of Fd() and a bug similar to OP&#39;s</a>. The current behaviour isn&#39;t going to change, though.</p></pre>kune13: <pre><p>The issue is here that the writer is not closed explicitly. That can be done by defer or by an explicit call to Close at the end of the parent. The garbage collector cannot touch the writer until Close is called.</p></pre>itsmontoya: <pre><p>My team has experienced this quite a bit. We avoid the issue by keeping an active pointer to the file for long running processes which intend to leave the file open.</p></pre>_rlh: <pre><p>Avoiding finalizers whenever possible is good practice. If a finalizer can&#39;t be avoid then limiting what they do is the next best option. As this example shows finalizers can introduce non-deterministic asynchronous behavior. The idiomatic solution of always pairing an open with a close seems bang on. This pairing leaves the finalizer with nothing to do.</p></pre>abcded1234234: <pre><p>I agree that this behaviour could be surprising but it also prevents memory leaks in programs like this:</p> <pre><code>func main() { x := readBigFileAsAString() y := getFirstLineOf(x) // x is never used after this point // never ending event loop for { .... } } </code></pre> <p>GC in Java behaves in the same way as far as I know. I am quite interested in what languages this is different?</p></pre>garaktailor: <pre><p>Ruby was the one I knew off the top of my head that won&#39;t collect the object until it goes out of scope.</p> <p>Also any language that uses reference counting instead of a garbage collector.</p></pre>abcded1234234: <pre><p>Does Ruby guarantee this behavior? Even a reference counting implementation can be (theoretically) smart enough and free variables earlier.</p> <p>The Go spec <a href="https://golang.org/ref/spec" rel="nofollow">https://golang.org/ref/spec</a> only mentions that Go is garbage collected language however GC implementation and behavior are completely unspecified.</p></pre>