Backend developers, what's your workflow when writing servers with go?

agolangf · · 65 次点击    
<p>I&#39;m coming from a NodeJS background. When building a server in NodeJs, we tend to leave caching, gzipping, most of the HTTP part to the web server (nginx or such). In production, we deploy a cluster of the app server (which is asynchronous by design) to multiple cores. Use frameworks like Koa/Express for handling routes and middlewares etc.</p> <p>Python follows a similar structure except for the deployment part (in most cases), where a pool of synchronous app servers are used by Gunicorn or the like to support concurrency. (Also causes complete machine utilisation)</p> <p>Since, I&#39;m just starting out with go, I&#39;d like to know the basics of commonly used patterns, as I&#39;ve had little luck online.</p> <p>What&#39;s the deal with go:</p> <ul> <li>What are things one should keep in mind when putting go in production. Any guides on it? What are the major differences between your production and development environments?</li> <li>Is it common to use a reverse proxy for HTTP heavylifting, or handling it in Go itself?</li> <li>I see a lot of references using go&#39;s http library directly. Is it advisable to use it straight rather than frameworks as in most of the other languages (not very common)?</li> <li>How does go take advantage of multiple cores? (natively, not containers).</li> <li>Testing. Are test runners common, or go has something built in for unit tests. Is the ecosystem mature?</li> <li>C bindings? (JNI/swig in other languages)</li> <li>Anything else you&#39;d like me to know?</li> </ul> <p>I know a lot of you might want to say RTFM, but these are things I need to consider early to get a grasp of the learning curve, timeline, and how to best handle a team deciding between go vs (others) as a tech stack.</p> <p>EDIT: These are a lot of great responses! Thank you guys. It did clear up a lot for me.</p> <hr/>**评论:**<br/><br/>epiris: <pre><ul> <li>Too broad- need to define &#34;it&#34;, prod and dev tend to be the same as far as the Go process, using docker to bring up service dependencies.</li> <li>I don&#39;t speak for everyone but I don&#39;t see a point in it unless your existing infrastructure makes it the path of least resistance.</li> <li>I personally use as much of std lib as possible, making sure any dependency I pull in is worth its weight in gold. Exceptions being a handful of libraries to common problems that I wrote, contributed to, or have reviewed thoroughly to ensure it meets my own criteria for quality.</li> <li>What you mean? You should have heard of goroutines by now, are you asking about how goroutines are scheduled processing time?</li> <li>No test frameworks needed, Go test is all I use .. no assertions libraries or anything, see my dependencies rules.</li> <li>Yes, cgo, but you will probably need it much less often.</li> <li>Yes, I suggest you go through the <a href="">tour</a> , then <a href="">how to write Go code</a> and <a href="">effective Go</a>. Skimming through them doesn&#39;t take any time at all but you can get a good idea about most your questions here and maybe lead you to ones with more substance.</li> </ul> <p>I&#39;ve been doing &#34;backend&#34; development for just about 15 years now and out of every mainstream language (I.e, c, perl, Java, Php, c#, node, Python and lots of random things between) and Go is my favorite. I write so much more software using Go. Every other language is full of people&#39;s diversions they created because they were sick of the diversions other people created. The Go tool chain gives you everything you need to not only produce quality software, but maintain and iterate on. Testing, linting, formatting, Compiling, cross compiling, performance profiling, tracing.. its all maintained by the Go authors and contributors. It&#39;s solidified and everyone agrees and uses these tools, so you immediately solve so many problems that other languages are constantly competeting with themselves in.</p> <p>Just my two cents. Take it easy.</p></pre>mavidser: <pre><p>Thanks for answering this. </p> <p>Yes, I see the multi-core problem doesn&#39;t make sense here. I didn&#39;t know that goroutines supported multiple CPUs.</p> <p>Also, what&#39;s your opinion about dependency management in go? I&#39;m not very fond of the idea of committing all my dependencies. I looked into &#34;dep&#34;, and it seemed pretty good. Any issues I should know about other than the fact that its not official as of yet?</p></pre>natefinch: <pre><p>You don&#39;t have to commit your dependencies with most go package management systems, including dep. You can just commit the one or two files used by the tool and bring down the dependencies at compile time.</p> <p>Dep is good, you should use it.</p></pre>mcouturier: <pre><p>I wouldn&#39;t advise someone asking this kind of question to use Docker. Also my two cents. Take it easy :)</p></pre>epiris: <pre><p>I don&#39;t get it, what does asking about a language you never used have to do with using docker? </p></pre>chmikes: <pre><p>Congratulation to consider Go. It&#39;s worth the exploration effort. You won&#39;t regret it. The ecosystem is rich, but not as much as with npm. Howerver, code is usually good. </p> <p>The rule of thumb is to stick to the standard libraries to avoid the versionning issue until it&#39;s solved properly. </p> <p>The language is very simple to learn. What took me a longer time to learn is to write simple code and get away of classes and the inheritence mind set. </p> <p>It is also good to know that some topics are subject to (excessively) passionate debates like the absence of generics and use of the Iris frameWork for instance. ;)</p> <p>Go is not just a language and compiler. It comes along with a large set of amazing and powerfull tools to help you produce high quality and reliable code. They help you investigate what your program is doing behind the scene. I only scratched the surface. </p></pre>alexfiori: <pre><p>Most of your questions have been addressed by others already but I&#39;ll give you my 2c anyway.</p> <ul> <li>Dev and prod is way to wide to discuss, but in general you&#39;ll be able to run your binary locally and test it. You can also have unit tests, the net/http/httptest package is pretty decent for this.</li> <li>It is common, I believe, to use reverse proxies, just because they&#39;re everywhere - cloud providers, etc. Like others pointed out, net/http is production grade and fine to expose to the world if you at least configure the timeouts.</li> <li>Using net/http directly is perfectly fine, but you should familiarize yourself with the concept of &#34;middlewares&#34;. Your early days trying to understand the http.Handler and http.ResponseWriter interfaces can be frustrating. There&#39;s quite a lot of middlewares out there, I&#39;d recommend looking at the ones in the &#34;chi&#34; package.</li> <li>You already know that goroutines can run in multiple CPUs. The default is to use all CPUs, but you can limit them by setting GOMAXPROCS. If you want to lock on an OS thread you can call runtime.LockOSThread().</li> <li>Most people use just the testing package for unit tests, which ships with the stdlib. I&#39;ve seen, however, quite a lot of code that uses other assisting libraries (packages), but don&#39;t personally like any of them.</li> <li>C bindings work just fine, and you can also export shared libraries written in Go that can be used by C programs. See the docs for build modes.</li> </ul> <p>About your last question: once you get into this with Go, for http and general backends, there&#39;s no way back. If you&#39;re looking into building more complex services you should look into gRPC, and the grpc-gateway package to automatically expose your APIs via http.</p></pre>bonekeeper: <pre><blockquote> <p>When building a server in NodeJs, we tend to leave caching, gzipping, most of the HTTP part to the web server (nginx or such). </p> </blockquote> <p>Some people put nginx/haproxy instances in front of Go for SSL and/or caching or special routing and that&#39;s fine, but not necessary.</p> <p>I personally have two nginx instances load balancing access to my backend (for redundancy) just so I can cache some static assets and do some fancy routing (let&#39;s say, have requests from /api/v1/ to go a set of backends, /api/v2/ go to another set of backends, have the institutional website go to yet another set of backends, etc). If you don&#39;t have any special need for nginx, you CAN deploy Go directly to the world using it&#39;s native http server, which is production-grade, as long as you observe some <a href="" rel="nofollow">good practices</a>, specially in regards to <a href="" rel="nofollow">timeouts</a> to keep misbehaving clients from messing with your resources.</p> <p>For your internal webservices, using Go&#39;s native http server is super-fine and standard (no need to have anything in front of them).</p> <blockquote> <p>In production, we deploy a cluster of the app server (which is asynchronous by design) to multiple cores. </p> </blockquote> <p>There&#39;s no need for that in Go, a single instance will already use multiple cores by default.</p> <blockquote> <p>Use frameworks like Koa/Express for handling routes and middlewares etc.</p> </blockquote> <p>You can use only the standard library if you wish and there are people that advocate for using only the standard library and no other frameworks. I personally like to use frameworks because more often than not they save me time (at the cost of getting tied to it forever - and the framework&#39;s opinion on how things should be done). I personally like <a href="" rel="nofollow">go-rest-json</a> which makes creating REST services a breeze and has support for a lot of middlewares. There are other libraries/frameworks that people use a lot like <a href="" rel="nofollow">Gorilla</a> and <a href="" rel="nofollow">Iris</a>. Iris had some community-related issues a while ago, not sure how they are faring now, but it seems to be <a href="" rel="nofollow">actively maintained</a>. Iris is fast, but it uses <a href="" rel="nofollow">fasthttp</a> as http server, not the standard one. It is focused on raw performance (at the cost of some compatibility with exotic clients, I think?). <a href=";hw=peak&amp;test=plaintext" rel="nofollow">It is pretty fast though</a>.</p> <p><a href="" rel="nofollow">Gin</a> is a popular web framework.</p> <p>Another up-and-coming web framework that seems nice and full-featured is <a href="" rel="nofollow">aah</a>.</p></pre>bonekeeper: <pre><blockquote> <p>What are things one should keep in mind when putting go in production. Any guides on it? What are the major differences between your production and development environments?</p> </blockquote> <p>I already linked to some best-practice guides on how to expose Go directly to the public Internet. As for environments, I try to keep my development and production environments as close as possible - usually there will be an environment variable that I pass to it that points to an <a href="" rel="nofollow">etcd server</a> from where configuration is pulled. If I&#39;m running under development, it will be pointing to my &#39;development etcd&#39; (database passwords, etc, are pulled from there and cached locally). I have all my services use <a href="" rel="nofollow">NATS</a> so I get service discovery and message queueing/passing for free with it between the components of my system. It also supports streaming (think Kafka).</p> <p>Imagine that you have a webservice in your system called &#39;urlsnapper&#39; that receives an URL and dimensions (width and height) and a destination (S3 or minio or local filesystem or some gluster mountpoint or whatever) and it takes a snapshot of the webpage at the URL using a viewframe of widthXheight and saves it to the destination and returns a message signaling success of failure to the caller.</p> <p>The backend workers for &#39;urlsnapper&#39; will run on their containers. When initialized, they check an environment variable (let&#39;s say, &#39;ETCD_URL&#39;) and fail if it&#39;s not present. From that ETCD_URL, it pulls all the environment information (so it knows it&#39;s running in &#34;development&#34; mode, knows where the ELK stack is to send event metrics and logs to, etc). From etcd, it also pulls the address of the <a href="" rel="nofollow">gnatsd</a> server(s) from the development environment. It connects to that gnatsd and announces that it is online and that it is serving &#39;urlsnapper&#39; requests.</p> <p>In your main application, whenever you want to take a screenshot of a page, you can call on NATS to either run it in the background (fire and forget), asynchronously (run it and be notified in a channel when it has finished) or synchronously (wait for it to complete, then move on). You call &#39;urlsnapper&#39; by sending a request to gnatsd and it load balances the request to an available &#39;urlsnapper&#39; backend - so you don&#39;t need to worry yourself about the IP addresses of your microservices. Note that this is just my personal preference - you might use Kubernetes for service discovery/routing, or Consul, or something else. I suppose one can consider <a href="" rel="nofollow">gRPC</a> the de-facto <a href="" rel="nofollow">RPC framework</a> for webservices in Go (but you&#39;ll still need something else for the service discovery part - knowing where to send your request to).</p> <p>Note that in Node, if you run a synchronous operation (like reading from a local file) or some heavy computation, your whole server freezes because you block the event loop. In Go, that doesn&#39;t happen - if you read from a local file, that goroutine gets automatically suspended and the processor does something else until that file read finishes. When the read finishes, the go runtime wakes up the goroutine and your code keeps on running from where it left off - so there&#39;s no need for callbacks, the runtime manages the illusion of serial code execution for you (by scheduling and rescheduling goroutines for you behind the scenes). You get concurrency and parallelism for free, without having to think hard about it (and no callback hell).</p></pre>bonekeeper: <pre><blockquote> <p>Is it common to use a reverse proxy for HTTP heavylifting, or handling it in Go itself? I see a lot of references using go&#39;s http library directly. Is it advisable to use it straight rather than frameworks as in most of the other languages (not very common)?</p> </blockquote> <p>Already explained.</p> <blockquote> <p>How does go take advantage of multiple cores? (natively, not containers).</p> </blockquote> <p>If you write a <a href="">simple hello world web server</a> in Go, it will already be concurrent and parallel: whenever a new client connection comes, a new goroutine will be spawned to handle it and run on an available core (paralellism). Suppose you have something more complex than a &#39;hello world&#39;, something that queries a MySQL server, for example: after you write the query request to the network, your application will wait for an answer from MySQL. At this point, your &#39;goroutine&#39; is suspended and another goroutine that is ready to do some work is awakened, and it&#39;s execution resumes from where it left off.</p> <p>Once MySQL performs the query and replies with data, the event loop that runs inside the Go runtime will notice that there is data available to be read in the socket and will wake up your goroutine for you to deal with that data, and you&#39;ll work with the query rows. No need for callbacks or explicit async/await or futures, the runtime manages it for you. As far as your code is concerned, everything happened without interruption.</p> <p>A lot of things will take advantage of multiple cores automatically for you: the webserver will spawn new goroutines to handle new clients (you don&#39;t need to do that yourself), when sorting integers from an array, the sorting implementation is already &#34;multithreaded&#34;, etc.</p> <p>To take advantage of multiple cores explicitly in your code, you basically just need to spawn goroutines to do the work that needs to be done and have a way to communicate with your goroutines (channels, usually) and a way to know when those goroutines are done with their work (sync.WaitGroup). There are a ton of great articles and talks on this subject:</p> <ul> <li><a href="">A Tour of Go - Goroutines</a></li> <li><a href="">Learn Concurrency</a></li> <li><a href="">Denis Uraganov - Concurrency: goroutines and channels</a></li> <li><a href="">Evan Huus - Complex Concurrency Patterns with Go</a></li> <li><a href="">Sameer Ajmani - Advanced Go Concurrency Patterns</a></li> <li><a href="">Rob Pike - Concurrency Is Not Parallelism</a></li> <li><a href="">Fun with Channels in Golang</a></li> <li><a href="">Golang Concurrency</a></li> </ul> <blockquote> <p>Testing. Are test runners common, or go has something built in for unit tests. Is the ecosystem mature?</p> </blockquote> <p>Yes, tests are common and a first class citizen in Go - Go has a built-in testing/benchmarking framework. Some talks on it:</p> <ul> <li><a href="">Testing Techniques</a></li> <li><a href="">Mitchell Hashimoto - Advanced Testing with Go</a></li> </ul> <blockquote> <p>C bindings? (JNI/swig in other languages)</p> </blockquote> <p>Yes, it&#39;s possible, but too complicated to explain here and I don&#39;t have a lot of experience with it. Basically the Go compiler also includes a small C compiler (!) to interface with C at compile-time (you can include C header files inside your Go code, or embed C code, for example). One thing to note is that because the Go ABI differs from the C ABI, there&#39;s a small extra overhead when making foreign calls. Some references:</p> <ul> <li><a href="">C? Go? Cgo!</a></li> <li><a href="">cgo</a></li> <li><a href="">How to include C code in your Go package</a></li> <li><a href="">Go Bindings for Various External APIs</a></li> </ul> <blockquote> <p>Anything else you&#39;d like me to know?</p> </blockquote> <ul> <li>the official <a href="">Tour of Go</a> is a great place to start.</li> <li><a href="">The Go Programming Language</a> is a great book to read</li> </ul> <p>When writing Go applications:</p> <ul> <li>Logging is very important, I recommend <a href="">logrus</a> logging to a local JSON file that gets forwarded to a centralized logging stack like <a href="">ELK</a>.</li> <li><a href="">metrics</a> will give you observability on the state, performance and health of your systems</li> <li>I use <a href="">spew</a> a lot for debugging too (think Python&#39;s <code>pprint</code> or PHP&#39;s <code>print_r</code>).</li> <li><a href="">gorm</a> makes working with databases easier.</li> <li>so does <a href="">upper</a></li> <li>learning to <a href="">profile</a> and <a href="">optimize</a> your applications will come in handy</li> <li>so will learning about how the <a href="">Go race detector</a> and optionally, <a href="">how it works</a></li> <li>and learning how to detect <a href="">memory leaks</a></li> <li>Go comes with great tooling by default, <a href="">learn about it</a></li> <li>working with weird/non-standard JSON can be a pain sometimes - I like to <a href="">gjson</a> and <a href="">sjson</a> when it makes sense</li> <li><a href="">GopherCon 2014 Best Practices for Production Environments by Peter Bourgon</a></li> </ul></pre>_youtubot_: <pre><p>Videos linked by <a href="/u/bonekeeper" rel="nofollow">/u/bonekeeper</a>:</p> <table><thead> <tr> <th align="center">Title</th> <th align="center">Channel</th> <th align="center">Published</th> <th align="center">Duration</th> <th align="center">Likes</th> <th align="center">Total Views</th> </tr> </thead><tbody> <tr> <td align="center"><a href="" rel="nofollow">Golang UK Conference 2015 - Evan Huus - Complex Concurrency Patterns with Go</a></td> <td align="center">Golang UK Conference</td> <td align="center">2015-09-10</td> <td align="center">0:53:46</td> <td align="center">53+ (94%)</td> <td align="center">7,941</td> </tr> <tr> <td align="center"><a href="" rel="nofollow">Google I/O 2013 - Advanced Go Concurrency Patterns</a></td> <td align="center">Google Developers</td> <td align="center">2013-05-18</td> <td align="center">0:34:11</td> <td align="center">552+ (99%)</td> <td align="center">83,305</td> </tr> <tr> <td align="center"><a href="" rel="nofollow">Rob Pike - &#39;Concurrency Is Not Parallelism&#39;</a></td> <td align="center">afriza na</td> <td align="center">2013-10-20</td> <td align="center">0:31:22</td> <td align="center">1,361+ (99%)</td> <td align="center">99,978</td> </tr> <tr> <td align="center"><a href="" rel="nofollow">GopherCon 2017: Mitchell Hashimoto - Advanced Testing with Go</a></td> <td align="center">Gopher Academy</td> <td align="center">2017-07-24</td> <td align="center">0:44:59</td> <td align="center">61+ (96%)</td> <td align="center">2,897</td> </tr> <tr> <td align="center"><a href="" rel="nofollow">Golang UK Conf. 2016 - Björn Rabenstein - Grand Treatise of Modern Instrumentation and Orchestration</a></td> <td align="center">Golang UK Conference</td> <td align="center">2016-09-09</td> <td align="center">0:26:57</td> <td align="center">5+ (83%)</td> <td align="center">565</td> </tr> <tr> <td align="center"><a href="" rel="nofollow">Golang UK Conference 2016 - Dave Cheney - Seven ways to Profile Go Applications</a></td> <td align="center">Golang UK Conference</td> <td align="center">2016-09-09</td> <td align="center">0:30:04</td> <td align="center">65+ (97%)</td> <td align="center">4,413</td> </tr> <tr> <td align="center"><a href="" rel="nofollow">Profiling &amp; Optimizing in Go / Brad Fitzpatrick</a></td> <td align="center">yapcasia</td> <td align="center">2015-08-28</td> <td align="center">0:59:24</td> <td align="center">136+ (100%)</td> <td align="center">7,233</td> </tr> <tr> <td align="center"><a href="" rel="nofollow">&#34;The Go Race Detector&#34; by Robert Knight</a></td> <td align="center">Go London User Group</td> <td align="center">2014-06-17</td> <td align="center">0:34:19</td> <td align="center">1+ (100%)</td> <td align="center">127</td> </tr> <tr> <td align="center"><a href="" rel="nofollow">&#34;&#34;go test -race&#34; Under the Hood&#34; by Kavya Joshi</a></td> <td align="center">Strange Loop</td> <td align="center">2016-09-18</td> <td align="center">0:44:14</td> <td align="center">0+ (0%)</td> <td align="center">3,079</td> </tr> <tr> <td align="center"><a href="" rel="nofollow">Finding Memory Leaks in Go Programs - Oleg Shaldybin</a></td> <td align="center">Hakka Labs</td> <td align="center">2014-08-13</td> <td align="center">0:28:27</td> <td align="center">67+ (98%)</td> <td align="center">6,520</td> </tr> <tr> <td align="center"><a href="" rel="nofollow">Go Tooling in Action</a></td> <td align="center">JustForFunc: Programming in Go</td> <td align="center">2016-06-02</td> <td align="center">0:41:51</td> <td align="center">1,061+ (99%)</td> <td align="center">39,598</td> </tr> </tbody></table> <hr/> <p><a href="" rel="nofollow"><sup>Info</sup></a> <sup>|</sup> <a href=";subject=delete%20comment&amp;message=dlf2e9m%0A%0AReason%3A%20%2A%2Aplease+help+us+improve%2A%2A" rel="nofollow"><sup>/u/bonekeeper</sup> <sup>can</sup> <sup>delete</sup></a> <sup>|</sup> <sup>v1.1.3b</sup></p></pre>bonekeeper: <pre><p>FWIW, I like <a href="" rel="nofollow">vscode</a> with lukehoban&#39;s <a href="" rel="nofollow">Go extension</a> - it looks <a href="" rel="nofollow">like this</a>.</p></pre>Jelterminator: <pre><p>Copying part of my response to another post on the frontpage right now:</p> <p>There&#39;s are definitely some gotchas in go though. The main ones are all caused by go being just to simple by default in some cases. Four things stand out:</p> <ul> <li>Versioning of dependencies is not available by default. I can definitely recommend <a href="" rel="nofollow">dep</a> and also my own tool <a href="" rel="nofollow">virtualgo</a>, which adds some extra functionality to dep.</li> <li>Testing works quite well, but it&#39;s just a bit too awkward by default. I can heavily recommend <a href="" rel="nofollow">testify</a>. Especially the suite package makes testing much nicer, while keeping the go simplicity.</li> <li>Logging is also a bit to simple usually for production apps. We use <a href="" rel="nofollow">logrus</a> which I&#39;m fairly happy about.</li> <li>Error handling in go is done using values instead of using exceptions. The default error values are too simple though, because they don&#39;t have stack traces. You should really use <a href="" rel="nofollow"></a>, to all errors in your applications.</li> </ul></pre>
65 次点击  
加入收藏 微博
添加一条新回复 (您需要 登录 后才能回复 没有账号 ?)
  • 请尽量让自己的回复能够对别人有帮助
  • 支持 Markdown 格式, **粗体**、~~删除线~~、`单行代码`
  • 支持 @ 本站用户;支持表情(输入 : 提示),见 Emoji cheat sheet
  • 图片支持拖拽、截图粘贴等方式上传