<p>I am trying to download a really large JSON file, 500MB-1GB. </p>
<p>This is the current code, where <code>returnFile(...)</code> is what the relevant handler calls.</p>
<p>This occasionally times out (504).</p>
<pre><code>type results struct {
// fields
}
func getResults(ctx context.Context)(r results){
for {
r, err := getData(ctx)
if err == io.EOF {
break
}
}
}
func (s *Server) returnFile(ctx context.Context, w http.ResponseWriter, r *http.Request, err error) {
results := getResults(...)
data, err := json.Marshal(results)
if err != nil {
handleHttpErrors(ctx, w, req, err)
return
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename"))
w.WriteHeader(http.StatusOK)
_, err = io.Copy(w, bytes.NewReader(data))
if err != nil {
s.handleHttpErrors(ctx, w, req, err)
return
}
return
}
</code></pre>
<p>After doing some reading, I think this is the right way to do this is:</p>
<ul>
<li>make buffer</li>
<li>post response</li>
<li>in the loop: get next data batch and write to buffer</li>
<li>close buffer and response</li>
</ul>
<p>What would be the right way to implement this in Go?
I can't get my head around how should this look like.</p>
<p>Ref to some of the reading:</p>
<p><a href="https://medium.com/stupid-gopher-tricks/streaming-data-in-go-without-buffering-3285ddd2a1e5" rel="nofollow">medium</a> </p>
<p><a href="https://www.reddit.com/r/golang/comments/2ntv6d/help_with_streaming_to_a_file/" rel="nofollow">reddit</a></p>
<p>Thanks for the help</p>
<hr/>**评论：**<br/><br/>nsd433: <pre><p>Try using a json.NewEncoder(), which you can stream into, rather than your 1-shot Marshal(), and try having one go routine read the results and encode them and the other copy the encoded output to w.</p>
<p>PS
io.Copy(w, bytes.NewReader(data) is a complicated way of doing w.Write(data)</p></pre>qu33ksilver: <pre><p>Make your <code>getResults</code> write to a <code>io.Writer</code>. Setup a <code>json.Encoder</code> which directly writes to <code>w</code> and have it encode the output from <code>getResults</code>.</p>
<p>Start with this. If you need further control, use <code>io.Pipe</code>.</p></pre>
