[D] Do you use Go-Lang in your Machine Learning Pipeline ?

We are a small team of (data) engineers, and we are working on our new pipeline. While we noticed that Python is the best tech to build our ML model, using the same lang to serve the model is not always good. We started thinking of using Go-lang/Scala/Nodejs to serve the model. The solution provides a predictive analytics, it gets data from sensors, mine and process the data, use the model to predict, and then show the results via notifications.

Please feel free to recommend best use-cases for Go-Lang and why you are using it in production (in similar cases).

Our main real time stack is go servers that basically generate the data (as in, serve stuff, generate log we can postprocess), they generate data that is then processed by Scala (personal recommendation), Python or some Pig Latin (a map reduce thingy)

Interesting! Thanks for answering.
If I understood well, you get data and preprocess it using Go, what do u use to store it or do u do everything on the flow ?

Re-read what he said

My entire pipeline is Go, except exploration. That's done in Jupyter. AMA

Thanks. What are the libraries that you use, Can u describe briefly the stack ?

I use a lot of Gonum and Gorgonia. I write custom VMs that can execute Gorgonia graphs, which may span networks. For a LONG time, models were serialized using gob but now I'm moving towards flatbuffers. 

Transport wise, it's mostly custom Go code. I find Spark ridiculous and overkill. I've used Spark, Hadoop, Mahout, and they're all generally really terrible and inelegant solutions. But for larger teams and "big data"/"big team" work, you might want to use them and consume stuff off them. The connectors in Go are relatively painless and nothing to write about.