What is the right flow to stream download a large file?

agolangf · · 439 次点击    
这是一个分享于 的资源,其中的信息可能已经有所发展或是发生改变。
<p>I am trying to download a really large JSON file, 500MB-1GB. </p> <p>This is the current code, where <code>returnFile(...)</code> is what the relevant handler calls.</p> <p>This occasionally times out (504).</p> <pre><code>type results struct { // fields } func getResults(ctx context.Context)(r results){ for { r, err := getData(ctx) if err == io.EOF { break } } } func (s *Server) returnFile(ctx context.Context, w http.ResponseWriter, r *http.Request, err error) { results := getResults(...) data, err := json.Marshal(results) if err != nil { handleHttpErrors(ctx, w, req, err) return } w.Header().Set(&#34;Content-Type&#34;, &#34;application/json&#34;) w.Header().Set(&#34;Content-Disposition&#34;, fmt.Sprintf(&#34;attachment; filename&#34;)) w.WriteHeader(http.StatusOK) _, err = io.Copy(w, bytes.NewReader(data)) if err != nil { s.handleHttpErrors(ctx, w, req, err) return } return } </code></pre> <p>After doing some reading, I think this is the right way to do this is:</p> <ul> <li>make buffer</li> <li>post response</li> <li>in the loop: get next data batch and write to buffer</li> <li>close buffer and response</li> </ul> <p>What would be the right way to implement this in Go? I can&#39;t get my head around how should this look like.</p> <p>Ref to some of the reading:</p> <p><a href="https://medium.com/stupid-gopher-tricks/streaming-data-in-go-without-buffering-3285ddd2a1e5" rel="nofollow">medium</a> </p> <p><a href="https://www.reddit.com/r/golang/comments/2ntv6d/help_with_streaming_to_a_file/" rel="nofollow">reddit</a></p> <p>Thanks for the help</p> <hr/>**评论:**<br/><br/>nsd433: <pre><p>Try using a json.NewEncoder(), which you can stream into, rather than your 1-shot Marshal(), and try having one go routine read the results and encode them and the other copy the encoded output to w.</p> <p>PS io.Copy(w, bytes.NewReader(data) is a complicated way of doing w.Write(data)</p></pre>qu33ksilver: <pre><p>Make your <code>getResults</code> write to a <code>io.Writer</code>. Setup a <code>json.Encoder</code> which directly writes to <code>w</code> and have it encode the output from <code>getResults</code>.</p> <p>Start with this. If you need further control, use <code>io.Pipe</code>.</p></pre>

入群交流(和以上内容无关):加入Go大咖交流群,或添加微信:liuxiaoyan-s 备注:入群;或加QQ群:692541889

439 次点击  
加入收藏 微博
暂无回复
添加一条新回复 (您需要 登录 后才能回复 没有账号 ?)
  • 请尽量让自己的回复能够对别人有帮助
  • 支持 Markdown 格式, **粗体**、~~删除线~~、`单行代码`
  • 支持 @ 本站用户;支持表情(输入 : 提示),见 Emoji cheat sheet
  • 图片支持拖拽、截图粘贴等方式上传