<p>Such as image processing, video processing, etc. </p>
<hr/>**评论：**<br/><br/>headzoo: <pre><p>You wouldn't transcode images and videos using the same process that's handling HTTP requests. Those should be two different systems, and should eventually run on different servers or in different containers. You don't want a catastrophic crash while transcoding a video taking down your website, and mixing transcoding and HTTP handling doesn't scale.</p>
<p>So you need at least two apps. Call them web.go and transcode.go, and you need a way for those apps to communicate with each other. As others have said, you should use a message queue. For small to medium size projects I recommend <a href="http://kr.github.io/beanstalkd/">beanstalkd</a>. (<a href="https://github.com/nutrun/lentil">Go library</a>) It's easy to install (<code>sudo apt-get install beanstalkd</code>), and easy to use. I've been using it for 5+ years to process millions of messages a day, and it hasn't crashed on me even once.</p>
<p>When web.go receives an upload, it should save the file to a central location (could be /tmp on a single server), create a database entry for the file, and then insert the row id into beanstalkd.</p>
<p>The transcode.go app continuously polls beanstalkd for new messages. When it gets a new message (a row id) it fetches the row from the database, transcodes the video/image, and updates the row to reflect the status of the transcode task, e.g. "success" or "failure".</p>
<p>What happens in web.go after inserting the beanstalkd job depends on your needs. You could send a response and close the connection. You could poll the database for the success/failure flag and send a response when the status changes. You could send a response, close the connection, and use ajax/websockets to poll for the status of the transcode job. It depends on what you're trying to do.</p></pre>cridenour: <pre><p>That said, I found adding a http interface to our long running services very helpful. </p></pre>headzoo: <pre><p>I agree. We're at a point where adding an web based administrative front end to a service is so easy that most services have something like that. Although from my experience those front ends are usually only meant for viewing statistics and statuses.</p></pre>dAnjou: <pre><p>This is the best way.</p>
<p>But I'd even say that <code>transcode.go</code> should report results back to an HTTP API provided by <code>web.go</code>, not to some DB. This way you have a nicely abstracted interface between both services.</p></pre>headzoo: <pre><p>Sounds like a good plan. Hard to say though without knowing OPs needs. One thing I would keep in mind is what happens if the web.go crashes or gets restarted. Are success/failure messages lost while it's down? Should transcode.go be tied up trying to send a response to web.go while it's unreachable? For my projects I would need a database entry for each upload anyway, though I would think twice about having transcode.go interact with the database.</p></pre>dAnjou: <pre><blockquote>
<p>One thing I would keep in mind is what happens if the web.go crashes or gets restarted. Are success/failure messages lost while it's down? Should transcode.go be tied up trying to send a response to web.go while it's unreachable?</p>
</blockquote>
<p>That's a good point. I guess trying to send the request until it gets a 200 could work. Or you send the result to another task system that retries sending the request every minute or so until it's successful.</p></pre>headzoo: <pre><p>True. Could use some kind of mediator. Let the mediator concern itself with how tasks and results are persisted (database, message queue, etc). web.go and transcode.go only need to concern themselves with the mediator api.</p></pre>SerialMiller: <pre><p>There are several ways to achieve this, using a MQ like miko5054 suggested is one way, other wise we have go routines at our disposal :).</p>
<p>Have a look here, they explain <a href="https://gobyexample.com/worker-pools" rel="nofollow">Worker Pools</a>, you could implement something like that as well. Totally depends on your usecase though.</p></pre>DavidDavidsonsGhost: <pre><p>For something like this had a redis cluster. I pushed the job into redis then pushed its id into a queue. The node doing the job updated its status in redis and would respond to messages to ensure that the node was still alive. The web front end just returns the current status of the job, don't try to do it synchronously as this won't be failure tolerant.</p></pre>albatr0s: <pre><p>You could launch goroutines.</p></pre>robertmeta: <pre><p>Yep. This is by far the most sane way. How you signal completion is up to you -- you can use a websocket and spit out progress as you go, you could ask for a webhook to callback to -- or give the client something they can poll.</p></pre>dhdfdh: <pre><p>I'm shocked this isn't the very first and top-rated answer.</p></pre>dAnjou: <pre><p>It shouldn't be because it doesn't scale at all especially when doing image or video processing. As <a href="/u/headzoo" rel="nofollow">/u/headzoo</a> said, it should be separate components communicating over some network protocol.</p></pre>n1ghtm4n: <pre><p>If your web servers are horizontally scalable, is there any reason to prefer an MQ over goroutines launched by the web server? The only downside I can think of is that you'd lose some jobs if the web server went down. But you can lose jobs in an MQ too. I'm not convinced the complexity of adding an MQ cluster is worth it.</p></pre>albatr0s: <pre><p>And this separate processes could launch goroutines.</p></pre>miko5054: <pre><p>Its not a go related question but Probably using MQ in order to create some asynchronous flow...</p></pre>foxh8er: <pre><p>The concern is that these processes may take more than a few seconds, likely after the time out. </p></pre>Fwippy: <pre><p>Then you need to signal to the client that the request has been logged, and notify them once it's complete. Depending on the length of the task (ten seconds? hours?), different notifications might be appropriate. If it can be expected relatively quickly, consider using long polling. If it will take a while, just send them an email once it's done.</p></pre>newimprovedoriginal: <pre><p>do not use long polling when you can use sockets.</p></pre>Frenchiie: <pre><p>Not every browser can use sockets and if someone is using an outdated browser then you are sol.</p></pre>newimprovedoriginal: <pre><p>good point. socket.io has long polling as a fallback. but I say damn the old browsers, damn them to hell. </p>
<p><a href="http://caniuse.com/#feat=websockets" rel="nofollow">http://caniuse.com/#feat=websockets</a></p></pre>kd7nyq: <pre><p>In a system that I'm working on now, we send an AJAX RPC call to the server (looks remarkably like a JSON+REST PUT), the server shoves the request into an in-memory queue (map[string]thingy), and then a go routine picks items off the queue and processes them. Feedback is provided by sending the results to the client using websockets. The user doesn't even need to stay connected and can come back later because of the websockets goodness. Multiple users can use the system at the same time for the same reason. The goroutine trickiness of shoving data into the map, pulling it out, processing it, and then sending feedback requires a single goroutine in this system, and is really straight-forward once it's written. It only takes about 70 lines of code. It took me several hours to get my brain around using goroutines instead of thread-safe queues/collections like you would in Java.</p></pre>foxh8er: <pre><p>Interesting! Do you know where I could find an example of such as system on GitHub?</p></pre>kd7nyq: <pre><p>I don't. I designed this system myself, but the code is protected by confidentiality agreements. Sorry. :(</p></pre>nowayno: <pre><p>Google "golang job queue" and you'll find a number of projects and other interesting links.</p></pre>
