HTTP Server Broken Pipe Errors

<p>I&#39;ve been spending some time tracking down issues in a production site that I believe are related to these errors showing up in the logs:</p> <pre><code>write tcp 204.248.121.130:41259: broken pipe </code></pre> <p>It looked like the errors were coming from potentially random places in the HTML template, so on a whim, I decided to try a solution that <em>actually worked</em>, to my surprise. The fix I came up with was to buffer the output by changing this:</p> <pre><code>// where `w` is an instance of http.ResponseWriter if err := views.ExecuteTemplate(w, &#34;template&#34;, context); err != nil { // handle error } </code></pre> <p>to this:</p> <pre><code>var buf bytes.Buffer if err := views.ExecuteTemplate(&amp;buf, &#34;template&#34;, context); err != nil { // handle error } else { io.Copy(w, &amp;buf) } </code></pre> <p>My question is: why? Why does buffering the data written to <code>http.ResponseWriter</code> fix this issue? The only thing I can think of is that there&#39;s some kind of timeout from the first call to <code>w.Write()</code> until the response is sent and the pipe is closed, since the template writes as it goes, but with the buffer that gap is drastically narrowed.</p> <p>It should be noted that it took using a <a href="https://github.com/rakyll/boom">load generator</a> to be able to reproduce the errors outside of production (note to self for all future projects: use a load generator <em>before</em> deploying to production), and the templates themselves are pretty sizeable and have a lot of room for optimization.</p> <p>What gives?</p> <hr/>**评论：**<br/><br/>flambasted: <pre><p>Real-world http clients, browsers trying to accommodate humans typing things into browsers, will be flaky. For example, they&#39;ll attempt to load one page in the background before deciding some other page is actually do desired. This is manifested as a GET on one socket, shortly followed by that socket being closed.</p> <p>Your &#34;fix&#34; isn&#39;t checking for socket errors anymore. Surely if you checked for errors returned by io.Copy, you&#39;d see the same rate of errors.</p></pre>MoneyWorthington: <pre><p>That does seem to be possible, but does that then mean that most of those errors are probably non-issues?</p></pre>flambasted: <pre><p>Exactly.</p></pre>jerf: <pre><p>It is unlikely that there is a timeout issue, unless the template is taking a very long time to render, long enough that you&#39;d probably have mentioned it.</p> <p>To diagnose this problem will require more details. For openers:</p> <ul> <li>Do you have a packet dump and have you found one of the closed connections in it? Who is closing the connection? At what point in the connection? (TCP negotiation, randomly in the stream, at a consistent location in the stream?)</li> <li>What OS is the server and client on? What network connects them? What network &#34;optimizations&#34; are in play? Have you increased whatever relevant limits there may be (maximum open files, maximum open sockets, etc)? (I haven&#39;t hit the limit in Go so I don&#39;t know if any of those would manifest as &#34;broken pipe&#34;.) Speed, topology, etc. You can&#39;t really be too detailed. Is there anything between the client and the server that thinks it understands HTTP and is doing something weird? Are you trying to push more data through the pipe than it can take and you&#39;re just plain seeing the pipe too full? Double-check with the packet dump; there are some ways of getting surprising blow-ups (see paragraph below).</li> <li>How many simultaneous pages are rendering, and on what sort of machine? How loaded is the machine while it&#39;s rendering all these pages?</li> <li>Have you tried simplifying the templates? What happens?</li> <li>Have you tried profiling the system? I wouldn&#39;t be entirely comfortable with just making things more efficient and hoping the errors go away, but, well, it&#39;s not an entirely invalid approach, either.</li> </ul> <p>Question for anyone who knows: Does the net/http server directly hook up .Write calls written to the ResponseHandler to the socket, such that writing one byte of response emits a one-byte-payload packet? It&#39;s a problem with Writers that is not often discussed here, though you can find it in the docs in some places, but you sometimes have to be careful not to make lots of small .Write calls to things that will emit one (packet/disk access/RPC call) etc per call. I&#39;m sort of curious about the answer even if it turns out not to be the problem.</p> <p>As it happens, I&#39;m headed off to bed, but the answers may prompt somebody else who can be helpful, or if you&#39;re really lucky, lead you to the answer yourself.</p></pre>ctcherry: <pre><blockquote> <p>Question for anyone who knows: Does the net/http server directly hook up .Write calls written to the ResponseHandler to the socket, such that writing one byte of response emits a one-byte-payload packet?</p> </blockquote> <p>Thankfully no, its smarter than that, there is quite a bit going on. There is a great description in the source: <a href="https://golang.org/src/net/http/server.go#L1077" rel="nofollow">https://golang.org/src/net/http/server.go#L1077</a></p></pre>