How should I go about managing a task queue/worker system in Go?

<p>I have about 50 scheduled tasks that I need to run with goroutine every 10 minutes. In Python&#39;s django/flask we can use Celery for monitoring and error handling, and RabbitMQ for message queueing. I am not sure where to start right now, do I even need a message/task queueing in GO? I am also considering using Amazon&#39;s SQS to have a easier time configuring the queue, has anyone used it with Go?</p> <hr/>**评论：**<br/><br/>chrighton: <pre><p>I am using SQS with GO and it works well. And I am using it essentially for a distributed task processor. Although mine are fire-and-forget tasks. The only oddity about SQS is that the message payload is a string rather than a []byte, so you need to marshall out your object, and then base64 encode it. Here is my publisher code. I used refection to determine the struct type, and add that as an attribute of the message. The consumer can then switch on that attribute and unmarshall accordingly. </p> <p><a href="https://gist.github.com/jsokel/22dc8cd07ca9ee1c8c4af46f8db78ca8" rel="nofollow">https://gist.github.com/jsokel/22dc8cd07ca9ee1c8c4af46f8db78ca8</a></p></pre>kT_Madlife: <pre><p>Thanks for your example! What was your reason for using SQS?</p></pre>tmornini: <pre><p>Not speaking for <a href="/u/chrighton" rel="nofollow">/u/chrighton</a>, but for myself, SQS has some fantastic characteristics:</p> <pre><code>1) Highly available 2) Highly resilient 3) Highly scalable 4) Infinite loop handling via Redrive feature 5) Can be a subscriber to SNS, allowing for pubsub 6) Very inexpensive 7) Maintenance free operation </code></pre> <p>The <em>only</em> downside I see with SNS/SQS are:</p> <pre><code>1) Not ultra-low latency </code></pre> <p>Common downsides which are commonly cited:</p> <pre><code>1) Out of order delivery is guaranteed 2) Multiple message delivery is guaranteed </code></pre> <p>Both common downsides are required to deliver the scalability that SNS/SQS guarantee, and are rarely if ever an issue when developing with HTTP semantics.</p></pre>alex_bilbie: <pre><p>Both of your SQS downsides can be mitigated by configuring the queue to be a FIFO queue and setting the visibility timeout to a high number (so when a message is pulled from the queue it is not sent to any other workers for the timeout period assuming the message isn&#39;t removed from the queue)</p></pre>tmornini: <pre><p>What you suggest mitigates what I said, I suggest you recheck the docs.</p></pre>kT_Madlife: <pre><p>Thanks for the breakdown.</p> <p>While I was doing my research I found out another aws service called Simple Workflow Service, have you heard/tried it out?</p></pre>tmornini: <pre><p>I have not.</p></pre>chrighton: <pre><p>Basically the same reasons as <a href="/u/tmornini" rel="nofollow">/u/tmornini</a> except that we use Kinesis for our heavy duty, multi-consumer queuing needs, and SQS for our lighter weight, single consumer, fire-and-forget needs.</p></pre>titpetric: <pre><p>Could you link up a consumer for SQS as well? Are you consuming SQS payloads directly with Go, or passing them on to some external scripts/etc (syscall exec?). Do you have one consumer that can consume all the things that you throw on the queue, or do you have individual consumers for a single queue out of many?</p></pre>chrighton: <pre><p>I have a Go consumer app, that receives all the messages from a particular queue. We can run as many instances of that as we need to handle whatever load we have. It reads a message, checks the type, and calls a processor according to type. Each processor knows how to unmarshall the payload for it&#39;s message type, and then handles the message.</p> <p>One example is sending emails. I don&#39;t want to distribute all the configuration, templates and credentials for generating and sending emails, so I centralize that in my task processor. The clients then just send a small request with template name, destination email, and a map of template variables. That&#39;s all they need to know. </p> <p>Here is a snippet of consumer code: <a href="https://gist.github.com/jsokel/176a026d013d3533f8faff849d7dea12" rel="nofollow">https://gist.github.com/jsokel/176a026d013d3533f8faff849d7dea12</a></p></pre>titpetric: <pre><p>Great, thanks. I&#39;d spread out the processing of it out just to have an idea of a single-responsibility, most likely with multiple queues for each type of task (one queue for emails, etc.).</p></pre>JokerSp3: <pre><p>Do you base64 because the transport is http?</p></pre>chrighton: <pre><p>No, the payload will only accept a certain set of values (although a decent set), and since we deal with UTF-8 data, it&#39;s better to be safe and just encode it. </p> <p>Check out the specs for MessageBody: <a href="https://docs.aws.amazon.com/sdk-for-go/api/service/sqs/#SendMessageInput" rel="nofollow">https://docs.aws.amazon.com/sdk-for-go/api/service/sqs/#SendMessageInput</a></p></pre>JokerSp3: <pre><p>Ah, its transmitted as xml so the charset is restricted, so you have to base64. </p> <p>Thanks!</p></pre>