<p>Hi all, I have a legacy, monolithic application that I want to transform into microservices. I come from a C background, so I am having a lot of fun learning go. I want to start separating out e-mail sending functions, image processing (resizing/etc) and interaction with AWS' S3 buckets. After some research, there are quite a few options out there: go-kit, kite, etc. I quite like the simplicity of kites but did not find how to integrate it with rabbitmq or another message broker. What would be your initial suggestions for me on that front, please? I'd want messages/tasks to persist and also be able to consult message status (e.g. image thumbnail generation). Many thanks for any help.</p>
<hr/>**评论：**<br/><br/>dazzford: <pre><p>Alternatively to <a href="/u/koffiezet">/u/koffiezet</a>'s suggestion, you can also go with google's protocol boffers, and grpc, the RPC end points it can generate for you. Both will generate to c and go, making interop easy.</p>
<p>As for using go, we started down a similar journey to you about 8 months ago. We made a decision to not use any framework like go-kit, gorillia, etc. and it has not hurt us at all. We've got close to 20 services and things are humming along.</p>
<p>We do use packages for working with amqp, postgresql, mysql, and redis.</p></pre>surekcass: <pre><p>That's sound advice, but it should have generated a lot of work in terms of logging, service discovery and monitoring. How did you guys handle that?</p></pre>dazzford: <pre><p>Logging is handled by a small package we wrote; probably 50 lines of code.</p>
<p>Service monitoring/scaling/resolution is handled by <a href="https://mesosphere.com/product/">DCOS</a>. We dockerize everything. </p>
<p>We then use a small package we wrote to look up the service records which Mesos DNS provides internally. This gives us the IP and port for the services internally within the cluster.</p>
<p>Externally we have consul up and running.</p></pre>dazzford: <pre><p>I should add, we have logstash up and running to aggregate all those logs.</p></pre>surekcass: <pre><p>I've been reading about proto3 and gRPC. I must say I can see how it could do what I need. An ELK instance for logs should work out fine, but I'd need to see how to do the monitoring without having to resort to DCOS.</p></pre>dazzford: <pre><p>I just read you are on AWS so perhaps the container service might be what you could use.</p></pre>surekcass: <pre><p>Yeah, AWS at the moment, but using Elastic Beanstalk. Thanks for all info. It is much appreciated. I will investigate further!</p></pre>dazzford: <pre><p>Good luck!</p></pre>dazzford: <pre><p>Your situation is most likely going to be different than ours.</p>
<p>We're using AWS and want more control over our environment as a whole, hence the DCOS. We also use zabbix to monitor things, which you could use for your services. </p>
<p>There is also <a href="https://deferpanic.com/" rel="nofollow">https://deferpanic.com/</a> for go.</p></pre>jrv: <pre><p>For metrics in a dynamic microservices world, we've created and used <a href="http://prometheus.io/">Prometheus</a> at SoundCloud (now in use at CoreOS, DigitalOcean, Kubernetes is instrumented with it, etc.). It's written in Go and has a great Go instrumentation library.</p>
<p>go-kit offers Prometheus integration in case you don't want to instrument your code with Prometheus directly: <a href="https://github.com/go-kit/kit/tree/master/metrics/prometheus">https://github.com/go-kit/kit/tree/master/metrics/prometheus</a> - direct instrumentation is recommended by Prometheus folks though rather than using wrappers.</p></pre>surekcass: <pre><p>Prometheus looks awesome. I had a quick look and it is rather interesting! I'd rather instrument directly, but will check whether using go-kit is a better option. Thanks! </p></pre>koffiezet: <pre><p>It sounds your microservices need to be accessible from different languages including C. Talking REST (or any HTTP-based interface) can be a pain in plain C, unless you want to fall back to SOAP with libzeep/gsoap/... - but is slow and XML (and not very well supported in Go). In C++ your options are a bit better but still limited.</p>
<p>At work we're evaluating <a href="https://thrift.apache.org/" rel="nofollow">Apache Thrift</a> for this reason, we need interoperable services between Python, C/C++ and Java (no Go, but it appears to be possible to use it)</p></pre>captncraig: <pre><p>Thrift is a good choice. I've had a lot of success with it. If I was starting again though I would definitely look at grpc. It still looks to be pretty cutting-edge, but it is essentially googles take on rpc for protobufs. </p></pre>surekcass: <pre><p>The current app is in PHP so I'd want to push some stuff to the services whilst we transition out of it, but I am indeed wanting to make it language agnostic to some extent, e.g. being able to send queued jobs from python/scala etc, if necessary. Thrift does look cool and will certainly be considered as an option. Need to get my head around it first, though. :)</p></pre>koffiezet: <pre><p>As far as I understood, thrift has the advantage of having a service description file, like what WSDL's are for SOAP - but in a more sane format with far less overhead.</p>
<p>That said, if you only need to consume the API from Python/PHP/Go, REST would do just fine, and I would still choose a "no framework" approach. I made the mistake a few years ago of choosing a certain framework for some service we use internally, and now it's abandoned. It's the downside of Go being very young: there are no established major libraries, except the stdlib and a handful others.</p></pre>schoenobates: <pre><p>+1 for thrift. I'd also like to throw in flatbuffers and protobuf as well: we've been looking at those recently, especially flatbuffers</p></pre>throwawayseq001: <pre><p>thrift is a steaming pile of shit due to weak specification. it is amazing that facebook and twitter can even do anything with it at all. i wrote an end to end black box test for thrift many moons ago that echos messages binary serialized with the official thrift impl. releases to another binary written in a different language/runtime deserialize, reserialize, re-transmit, decode and compare the messages for fidelity, and it was broken. first enum elements are inconsistently zero or one depending on the language.</p>
<p>tread carefully. protocol buffers are much more sane and have been vetted by a larger community.</p></pre>surekcass: <pre><p>Thanks for this input. I was aiming to first have a stab at protobuf indeed. However, it is also interesting that I might have a steaming pile of shit to try with later on. :)</p></pre>random314: <pre><p>I'd recommend porting libvips for image. I wrote an image microservice in golang myself and it's extremely fast and scalable. I think there are already a library posted in this sub just yesterday that does exactly this.</p></pre>surekcass: <pre><p>I presume you mean this one, right? <a href="https://github.com/h2non/imaginary" rel="nofollow">https://github.com/h2non/imaginary</a> </p></pre>random314: <pre><p>Yep, that one.</p></pre>google_you: <pre><p>cron</p></pre>
