Tools for async file IO

<p>Disclaimer: Go beginner, have only used it very briefly for a tiny static web server.</p> <p>What&#39;s the current state of efficient async file IO in Go? Is it just spawning many goroutines and using io.Copy?</p> <p>I&#39;m thinking of implementing a rough version of <code>cp</code> as a toy project, and looking around, have found things like <a href="https://github.com/traetox/goaio" rel="nofollow">goaio</a> that provide interfaces to os-level async file IO, since the more canonical async primitives like epoll/kqueue don&#39;t work for normal files. What else is there?</p> <hr/>**评论：**<br/><br/>jeffrallen: <pre><p>Idiomatic Go code uses blocking reads and writes, and implements concurrency via launching one goroutine per blocking operation. This is how it looks to the Go programmer (and future readers of the code) because it is a simple model to reason about, which makes bugs more rare and maintenance cheaper.</p> <p>It is up to the runtime to arrange that request for blocking I/O is then implemented in an efficient way. For networking file descriptors, they are actually set to nonblocking, and the runtime uses async I/O to multiplex the I/O onto one or a few system threads.</p> <p>I am pretty sure that this does not yet extend to file descriptors opened via os.OpenFile. In that case, each blocking read/write will result in blocking a system thread. Depending on the setting of GOMAXPROCS, the runtime may arrange that other goroutines start running while the one that is blocked on I/O sleeps. If those in turn also block, you will eventually end up with GOMAXPROCS system threads that are blocked in I/O. The effect is the same as async I/O, but maybe with a lower cap on concurrency that you might have implemented via async I/O.</p> <p>As always with performance tradeoffs, you need a real use case with a real measured throughput to decide if you want/need to do extra work beyond programming it the simplest way possible.</p> <p>-jeff</p></pre>ijustwantaredditacct: <pre><blockquote> <p>I am pretty sure that this does not yet extend to file descriptors opened via os.OpenFile. In that case, each blocking read/write will result in blocking a system thread.</p> </blockquote> <p>This is my understanding as well. </p> <blockquote> <p>Depending on the setting of GOMAXPROCS, the runtime may arrange that other goroutines start running while the one that is blocked on I/O sleeps. If those in turn also block, you will eventually end up with GOMAXPROCS system threads that are blocked in I/O. </p> </blockquote> <p>This does not match my understanding -- I&#39;m under the impression that any thread blocked on i/o sleeps does not count toward the number of runnable threads -- that is, the runtime will create or re-use an existing thread for the io operation and mark the goroutine as sleeping, but still keep GOMAXPROCS threads/goroutines running (This would apply to all syscalls). I believe there&#39;s a maximum 10,000 thread limit.</p> <blockquote> <p>The effect is the same as async I/O, but maybe with a lower cap on concurrency that you might have implemented via async I/O.</p> </blockquote> <p>Very close, but not quite, I don&#39;t believe. I think if it were async i/o, we might be able to have better options for timing out/setting a deadline on a file read/write. As it stands, it&#39;s <em>sort of</em> possible. You can do the file operation in another goroutine, and use a channel to signal it&#39;s completion -- at that point you can use <code>select</code> against that channel, and a context, and time out -- the actual i/o would still be underway, but the user-facing end could move on. (Use case here would be, say, properly being able to return a 5xx response if a file operation took more than N seconds). </p> <p>The behavior can be particularly obnoxious if the underlying drive fails, and all threads doing i/o operations against it get stuck in an uninterruptible sleep -- you have to wait for the kernel to time out the i/o operation, which is usually longer than you want to wait.</p> <p>I suppose you could argue the above behavior is an implementation detail -- you could specify a timeout while doing synchronous if the kernel supported it.</p></pre>jeffrallen: <pre><p>I went to play and I made this: <a href="https://play.golang.org/p/Gsp9qzGbdv" rel="nofollow">https://play.golang.org/p/Gsp9qzGbdv</a></p> <p>Using it I was able to look at the execution traces for parallel io.Copy. I found that the call to os.Remove caused the goroutine to be suspended while the potentially blocking call was shunted onto a system thread.</p> <p>But the calls to Read/Write inside of io.Copy are implemented without any preemption, and are actually calls to internal/poll.(<em>FD).Read:121 and internal/poll.(</em>FD).Write:218.(See <a href="https://github.com/golang/go/blob/release-branch.go1.9/src/internal/poll/fd_unix.go#L121" rel="nofollow">https://github.com/golang/go/blob/release-branch.go1.9/src/internal/poll/fd_unix.go#L121</a>)</p> <p>TIL: Something like the netpoller now exists for regular file IO.</p> <p>The execution traces show full utilization of the processors, driving reads and writes. I suppose it could be interesting to compare wall clock timings for 100 file copies with GOMAXPROCS==numcpu and GOMAXPROCS==numcpu*2, but I&#39;m going to leave that to the OP.</p> <p>Tell us what you find!</p></pre>kostix: <pre><p>From the <a href="https://golang.org/doc/go1.9#os" rel="nofollow">Go 1.9 release notes</a>:</p> <blockquote> <p>The os package now uses the internal runtime poller for file I/O. This reduces the number of threads required for read/write operations on pipes, and it eliminates races when one goroutine closes a file while another is using the file for I/O.</p> </blockquote></pre>hudsonyard: <pre><p>May I ask what is the use case here ? as you would probably make this linux only ?</p></pre>