Buffered io.Writer switching to file-backing for larger streams?

<p>With both client or server TCP roles I&#39;ve often had to deal with sending larger streams via Go. Using a temp file allows me to easily stream the contents to disk without using a bunch of memory. However, for smaller payloads it is obviously much faster to avoid the file I/O if it&#39;s not needed.</p> <p>How could I create an io.Writter that saved to a buffer up to X bytes then flushed to disk and switched to using a os.File for the remaining bytes? Furthermore, am I missing anything bad about this idea?</p> <hr/>**评论：**<br/><br/>jerf: <pre><p>It kinda sounds like you&#39;ve got everything you need figured out? You just implement a .Write method that saves the bytes to a slice until it is if a certain size, then write the file.</p> <p>Whether it&#39;s a good idea depends on a lot of details about the relative cost and availability of various resources. Hard to be sure without more details, and the decision tree is too big for me to serialize into a Reddit post. I am a bit suspicious, but it&#39;s not intrinsically a bad idea under all circumstances or anything.</p></pre>kubernever: <pre><p>I&#39;m new to golang myself (more of a java guy), but couldn&#39;t you just create a decorator function that wraps an io.Writer and os.File?then just put your logic in there. </p></pre>soapysops: <pre><p>I think there may be some confusion here about how disk based IO works. When you write to a file, the main overhead is usually in context switching system calls, not in actual writing to the disk. The OS already buffers stuff in memory for you.</p> <p>To reduce the system call cost, usually it&#39;s best to use a buffered writer. Go provides one in the standard library: bufio.Writer. Some languages do this buffering automatically, but in Go you have to opt in.</p> <p>If you use a buffered writer, you shouldn&#39;t need to implement anything else yourself. You can change the size of the buffer in case for some reason the system calls are really slow for you.</p></pre>Xeoncross: <pre><p>I&#39;m not worried about the overhead of disk I/O, I simply want to avoid I/O actions: the slow process of writing (then reading) from disk. So, I am trying to write a &#34;smart&#34; buffer that only writes to disk/reads from disk when it gets to large to comfortable fit in memory.</p></pre>tgulacsi: <pre><p><a href="https://godoc.org/github.com/tgulacsi/go/temp#NewMemorySlurper" rel="nofollow">https://godoc.org/github.com/tgulacsi/go/temp#NewMemorySlurper</a> is such a Writer. Doc is lacking, but the source is short and simple - good for copying :)</p></pre>ahmatkutsuu: <pre><p>Check out the <a href="https://github.com/djherbis/buffer" rel="nofollow">https://github.com/djherbis/buffer</a></p></pre>