How should I go about managing a task queue/worker system in Go?

I have about 50 scheduled tasks that I need to run with goroutine every 10 minutes. In Python's django/flask we can use Celery for monitoring and error handling, and RabbitMQ for message queueing. I am not sure where to start right now, do I even need a message/task queueing in GO? I am also considering using Amazon's SQS to have a easier time configuring the queue, has anyone used it with Go?

---

评论：

chrighton:

I am using SQS with GO and it works well. And I am using it essentially for a distributed task processor. Although mine are fire-and-forget tasks. The only oddity about SQS is that the message payload is a string rather than a []byte, so you need to marshall out your object, and then base64 encode it. Here is my publisher code. I used refection to determine the struct type, and add that as an attribute of the message. The consumer can then switch on that attribute and unmarshall accordingly. 

https://gist.github.com/jsokel/22dc8cd07ca9ee1c8c4af46f8db78ca8

kT_Madlife:

Thanks for your example! What was your reason for using SQS?

tmornini:

Not speaking for /u/chrighton, but for myself, SQS has some fantastic characteristics:

1) Highly available
2) Highly resilient
3) Highly scalable
4) Infinite loop handling via Redrive feature
5) Can be a subscriber to SNS, allowing for pubsub
6) Very inexpensive
7) Maintenance free operation

The only downside I see with SNS/SQS are:
1) Not ultra-low latency

Common downsides which are commonly cited:
1) Out of order delivery is guaranteed
2) Multiple message delivery is guaranteed

Both common downsides are required to deliver the scalability that SNS/SQS guarantee, and are rarely if ever an issue when developing with HTTP semantics.

alex_bilbie:

Both of your SQS downsides can be mitigated by configuring the queue to be a FIFO queue and setting the visibility timeout to a high number (so when a message is pulled from the queue it is not sent to any other workers for the timeout period assuming the message isn't removed from the queue)

tmornini:

What you suggest mitigates what I said, I suggest you recheck the docs.

kT_Madlife:

Thanks for the breakdown.
While I was doing my research I found out another aws service called Simple Workflow Service, have you heard/tried it out?

tmornini:

I have not.

chrighton:

Basically the same reasons as /u/tmornini except that we use Kinesis for our heavy duty, multi-consumer queuing needs, and SQS for our lighter weight, single consumer, fire-and-forget needs.

titpetric:

Could you link up a consumer for SQS as well? Are you consuming SQS payloads directly with Go, or passing them on to some external scripts/etc (syscall exec?). Do you have one consumer that can consume all the things that you throw on the queue, or do you have individual consumers for a single queue out of many?

chrighton:

I have a Go consumer app, that receives all the messages from a particular queue. We can run as many instances of that as we need to handle whatever load we have. It reads a message, checks the type, 
and calls a processor according to type. Each processor knows how to unmarshall the payload for it's message type, and then handles the message.
One example is sending emails. I don't want to distribute all the configuration, templates and credentials for generating and sending emails, so I centralize that in my task processor. The clients then just send a small request with template name, destination email, and a map of template variables. That's all they need to know. 
Here is a snippet of consumer code:
https://gist.github.com/jsokel/176a026d013d3533f8faff849d7dea12

titpetric:

Great, thanks. I'd spread out the processing of it out just to have an idea of a single-responsibility, most likely with multiple queues for each type of task (one queue for emails, etc.).

JokerSp3:

Do you base64 because the transport is http?

chrighton:

No, the payload will only accept a certain set of values (although a decent set), and since we deal with UTF-8 data, it's better to be safe and just encode it. 
Check out the specs for MessageBody:
https://docs.aws.amazon.com/sdk-for-go/api/service/sqs/#SendMessageInput

JokerSp3:

Ah, its transmitted as xml so the charset is restricted, so you have to base64. 
Thanks!