<p>For example, if I write a small server that runs on port 9090 and then I want to upgrade it with minimum down time. What are my options?</p>
<p>Thanks</p>
<hr/>**评论：**<br/><br/>Taikumi: <pre><p>The basic answer is that you'll need to put it behind a load balancer (i.e. HAProxy) and have it route requests to multiple server instances. You then can do a rolling deploy to take down one instance at a time and swap it with the newer updated code. Ideally this should provide you with 100% uptime throughout the upgrade.</p></pre>DarkRye: <pre><p>Can nginx web server be used for this purpose?
What about Apache?</p></pre>rcklmbr: <pre><p>Yes, <a href="http://nginx.org/en/docs/http/load_balancing.html">nginx</a>, <a href="http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html">httpd</a>, and <a href="http://www.haproxy.org/">haproxy</a> can all be used to do that. Alternatively, if you dont want to manage your own and are using EC2, you can use <a href="http://aws.amazon.com/elasticloadbalancing/">Amazon ELB</a> or <a href="https://cloud.google.com/compute/docs/load-balancing/">Google Cloud</a> has one too.</p></pre>mwholt: <pre><p>I'd love for more people to put <a href="https://github.com/mholt/caddy" rel="nofollow">Caddy</a> through its paces. If you or anyone tries it for load balancing, do let me have your feedback. <a href="http://caddyserver.com/docs/proxy" rel="nofollow">Load balancing docs</a></p></pre>ketralnis: <pre><p>haproxy is really, really good and I'd definitely recommend giving it a shot first.</p>
<p>Whatever load balancer you use, it needs to be able to reliably determine which backends are up so it knows where it can send requests. This is usually called health checking. If nginx and apache can do that, they'll work just fine.</p></pre>BraveNewCurrency: <pre><p>One problem with HAProxy is that changing the configuration requires a restart, which is not zero downtime.</p>
<p>Nginx is zero downtime, but the tools for troubleshooting/logging have been more primitive. (The Nginx company recently did some work here, so maybe it's better.)</p></pre>BlueDragonX: <pre><p>The way you restart haproxy isn't a traditional restart - it's more of a replacement. The new haproxy instance takes over listening for connections while the old one completes requests on open connections. When those connections close the old instance shuts down. This results in no down time for haproxy.</p>
<p>I manage multiple containerized haproxy instances to load balance traffic in a production CoreOS cluster. Their configuration is entirely dynamic and they restart using this method. They never lose connections.</p>
<p>The infrastructure for doing all of this is written in Go and some of the load balanced applications are as well.</p></pre>BraveNewCurrency: <pre><blockquote>
<p>This results in no down time for haproxy.</p>
</blockquote>
<p><em>Citation needed.</em></p>
<p>It's really easy to test: Pound your server with a trivial request, and do a (soft)restart of haproxy. Count the "could not connect to server" errors.</p>
<p>If there was no problem, why does the HAProxy home page cite this extremely complicated work-around to the problem? <a href="http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html" rel="nofollow">http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html</a></p></pre>BlueDragonX: <pre><p>I don't deny that's the case, and perhaps for services that make such frequent connections you'll notice it, but for our use cases it has not been an issue. We do push quite a bit of traffic through our LB's and neither our clients or our monitoring systems indicate a problem.</p></pre>BraveNewCurrency: <pre><blockquote>
<p>neither our clients or our monitoring systems indicate a problem.</p>
</blockquote>
<p>Right. There is no error logged when HAProxy isn't listening, so how would you know?</p>
<ul>
<li>If a client gets a single error, and the site is still up immediately after, would they call you? (Really?) Even if they did, would anyone "connect the dots" to haproxy? (If it's out to lunch for 100ms and you get 100 requests per second, you will average 10 errors per restart. If your site has 300 active users at a time, they will only see one error every 30 restarts, on average. )</li>
<li>Assuming haproxy is out to lunch for 100ms, and your monitoring checks every minute, monitoring won't see the problem 599 out of every 600 restarts. If you restart haproxy every day, it will take 1-2 <em>years</em> for your monitoring system to notice the problem.</li>
<li>Most monitoring systems won't alert unless you are down for a few seconds. So it's likely that your monitoring will <em>never</em> alert you to the problem.</li>
</ul>
<p>I'm not saying you have to address the problem, only that you have to understand the trade-offs. You can restart every day and still claim "five nines" of reliability (assuming nothing else goes wrong...). But you can't ever claim 100% uptime.</p></pre>UptownFunkLyrics: <pre><blockquote>
<p>I'm too hot!</p>
</blockquote>
<p>Hot Damn!</p></pre>santicl: <pre><p>nginx works great to do it. It reloads almost instantly.</p>
<p>I have this in my nginx.conf:</p>
<pre><code> location / {
proxy_pass http://localhost:8080;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
}
</code></pre>
<p>You just have to start your new go server in a diferent port, update nginx.conf and reload.</p></pre>syf81: <pre><p>As an alternative to what's already been mentioned here, if your OS supports SO_REUSEPORT, you could just make your server bind to that port with that socket option.</p>
<p>Upgrading it with minimal downtime is then as easy as launching your new server, and gracefully shutting down the old one, it can for example listen to a signal that closes the listen socket, finish whatever requests it's processing and then exit.</p></pre>mwholt: <pre><p>This sounds really cool. Where would I go to learn how to do this in Go specifically?</p></pre>syf81: <pre><p>There's no proper implementation of it in the net library yet, but basically you have to use syscall.SetsockoptInt in the os library with the appropriate file descriptor and socket option, which is pretty much how src/pkg/net/sockopt_linux.go works.</p>
<p>However if you do want to use it but don't want to go in too deep, there's a package available that does this.</p>
<p><a href="https://github.com/jbenet/go-reuseport" rel="nofollow">https://github.com/jbenet/go-reuseport</a></p></pre>docsavage: <pre><p>You might consider using goagain:
<a href="https://github.com/rcrowley/goagain">https://github.com/rcrowley/goagain</a></p>
<p>Example of zero-downtime HTTP server restarts using the manners HTTP server:
<a href="https://github.com/cupcake/mannersagain">https://github.com/cupcake/mannersagain</a></p>
<p>I plan on using this with a goji-based HTTP server.</p></pre>arschles: <pre><p>Some people have talked about the DevOps (or whatever we're calling it now) perspective - just want to expand on it.</p>
<p>I'm assuming your app is stateless, like a frontend. If you're running a frontend (like an HTTP server) that's not stateless, try to make it stateless. It'll be a lot easier to scale that way.</p>
<p>Anyway, with that out of the way, here's a list of steps to run and upgrade a service with no downtime. It has the added benefit of redundancy and fault tolerance. The cost of doing that is real money - you have to run more than 1 server. It also features some buzz technologies, but I actually like them so here goes...</p>
<ul>
<li>Put your app in a Docker image (or your favorite container format. VMs work here too, but they're pretty heavy).</li>
<li>Run it on 2+ nodes, preferably in different "datacenters" (or if you're running in a cloud, whatever your cloud calls them) to gain some failure tolerance</li>
<li>Create a load balancer that routes to your 2+ nodes. Prefer a LB with an API like Google Cloud's or AWS's, so you can tell it to stop routing to a node before you take down that node.</li>
<li>Tell your LB to stop routing to node A.</li>
<li>Pull the new docker image with your new server to node A</li>
<li>Run the new image on node A</li>
<li>Tell the LB to start routing to node A again</li>
<li>Watch the logs/stats/anything else on node A</li>
<li>If everything looks good on node A, then do the same deploy for node B. Otherwise do the same deploy to roll back node A. Node A is called your "canary" release in this case. There are a few more gotchas to canary releases that are out of the scope of this comment.</li>
</ul>
<p>Note that if you're running a load balancer that doesn't have an API (my understanding is that Nginx and HAProxy don't), it may be ok to let node A fail out. I'm not familiar enough with those LBs to give a definitive answer.</p>
<p>Also note that there are systems to automate this process. For example, Kubernetes has rolling deploys that do a similar process, without the full watch-the-logs process I talked about. CoreOS has a facility to deploy to only nodes with specific "labels" that you define, which you can, with some added glue to deal with the LBs, use to do canary releases with no downtime. I use CoreOS and I'm trying to automate this whole no-downtime+canary process now. Right now it's all manual...</p>
<p>Ok, I've gotten a little off track but hopefully this comment shows that once you've solved your initial problem of no-downtime deploys, you can expand it to do more things to make your deploys even more reliable.</p>
<p>Good luck.</p></pre>jobenjo: <pre><p>Fwiw, after looking into this for awhile, I decided it wasn't worth the trouble (running a consumer app). During deploy, the downtime is still just momentary (<100ms), and the app is built withstand random network errors, anyway. </p></pre>perihelion9: <pre><p>You've already had load balancers and traditional answers given to you, but i've found that there are always operational problems with them. You need to know the hosts and ports of all your webservers, configuration can get out of sync, balancing strategies can get wiry, and hosting multiple (small) services multiplies the problem. You can start layering more tools on top, like Serf to have cluster awareness and automatic load balancer addition/removal, but it gets ridiculous after a time - you should only need that once you reach a scale whereby <em>you need that</em>, not for rollbacks, upgrades, or poking around.</p>
<p>I've started advocating your public-facing servers <em>never</em> go down, only hotload configurations. Those servers do nothing except forward the parsed user request to a message broker (such as Rabbit), and from there your actual worker processes (which live elsewhere) can consume the messages, process them, and put another message back on the message queue to be delivered to the webserver, which serializes it and sends it to the client.</p>
<p>This means you never functionally touch your webserver (only the route mappings it has, which are hotloaded), and you can feel free to roll forward or back on your worker processes without repercussion - they just connect to Rabbit and begin processing requests when they're up.</p>
<p>In general this increases durability of your whole system, and makes it so that you will almost never drop a user connection, regardless of what you're doing with the business logic in your worker processes.</p></pre>mioelnir: <pre><p>This is also not either/or to loadbalancers etc - those you can still put before the public facing servers. Admittedly, that is then usually a whole different ballgame with regards to scale.</p></pre>perihelion9: <pre><p>Yeah, you <em>do</em> need them for scale, but not for deployments. I added this line not long after posting:</p>
<blockquote>
<p>you should only need that once you reach a scale whereby you need that, not for rollbacks, upgrades, or poking around.</p>
</blockquote></pre>Mteigers: <pre><p>Do you find that rabbit is fast enough to essentially mimic an entire web stack? </p></pre>perihelion9: <pre><p>Rabbit is only the message broker, and it's extremely quick, so yes it's fine. Splitting off worker processes (or calls to other internal APIs) are common in most setups anyway, the extra in-rack (or on-node) round-trips to Rabbit are negligible. Rabbit just simplifies everything to a pub/sub model rather than peer discovery (or DNS) in-network.</p>
<p>And anyway, this setup dramatically increases possible throughput-per-webserver (since the webservers exist only to parse requests and serialize responses). With the right nonblocking server setup, you can have a very linear rate of resource consumption on the webservers relative to the number of connections received per second (as opposed to traditional jetty/apache/etc setups, whereby resources consumed increase closer to logarithmically to <em>open connections</em>). Meaning you can serve more clients at a faster average rate, even if individually every request takes imperceptibly longer.</p></pre>iends: <pre><p>Do you have more reading material about this kind of architecture?</p></pre>pinpinbo: <pre><p>We version every binary with SemVer number scheme, change symlink to point to the new one, and do rolling deploy.</p>
<p>Extremely simple. I would go out of limbs to say that this is one of Go's biggest strength.</p></pre>fenduru: <pre><p>What aspect of this is a strength of Go?</p></pre>ericanderton: <pre><p>If I had to guess: Probably that most applications compile to a single executable making a filesystem based cut-over like this extremely easy.</p></pre>anoobisus: <pre><p>Versus... Renaming a directory that gasp might have more than one file inside.</p>
<p>Everyone has their half passed approaches to deployment and too many people are proud of how hacky and unscaleable they are.</p>
<p>Shrug</p></pre>CapoFerro: <pre><p>That assumes you have your app's dependencies 100% vendored into that directory that you are renaming or relinking. In Ruby or Python, for example, if your dependencies change between versions and you don't have everything vendored, it's often impossible to do what you describe.</p></pre>anoobisus: <pre><p>.... What? You mean your deployment might involve randomly updating nonvendored app dependencies?</p>
<p>That thing I said about hacky deployments.</p>
<p>If you can't blow away a prod machine and rebuild it instantly. You're doing it wrong</p>
<ol>
<li>Vendor it</li>
<li>Build a redeployable vagrant, packer, ec2 disk image with deps pinned.</li>
<li>spend 5 minutes and throw it in a docker container with pinned deps.</li>
</ol>
<p>I mean... If you're deploying a bare bones app and doing an npm restore on the server... Then you're not done deploying until you've restored... At which point atomicity applies.... Which of course you can't do if you rely on global environment state installing to some global rubyenv.</p>
<p>But then again virtual ruby and python and npm environments exist for very, very, very, very, I'm typing this by thumb, very very very important reasons.</p>
<p>And of course ignored by people because Hey my hacky deployment start works fine for us! Hope we never pull a bad package on the server.</p>
<p>Sure hope, you know, GitHub doesn't go down and we lose the ability to deploy our app ( which happens and is stupidly almost proudly announced on twitter)</p></pre>
