golang program as a systemd service which needs to self-update

<p>Hello. I&#39;m new to golang, but I have been able to progress quickly. I am fighting n-th hour in a row and I simply don&#39;t know what am I doing wrong. Basically, I have a very simple service file for systemd:</p> <pre><code>[Service] Type=simple User= ... ExecStart = ... </code></pre> <p>the service does not run as root user, but rather - an unprivileged user. the service itself is a for {} with a sleep (so that I don&#39;t waste CPU power)</p> <p>Everything works great. However, I now want to self-update the service. Therefore I spawn the updater process (from within the systemd service) like so:</p> <pre><code>cmd := exec.Command(&#34;sudo&#34;, &#34;myupdater&#34;) // I need to start it with sudo I suppose in order to stop and start systemd service cmd.Start() </code></pre> <p>myupdater should do these steps:</p> <ol> <li>stop the service</li> <li>copy new binary</li> <li>start the service</li> </ol> <p>the problem is - as soon as I stop the service from the child process, the child process dies as well. This has nothing to do with sudo passwords as I have already configured nopasswd entries in sudoers. I also do a tail on the service logfile and I can see the entries indicating that the service is stopping. I have been reading <em>a lot</em> about process groups and background processes, I have tried setting attributes like Setsid, Pdeathsig: 0, adding nohup as well as ampersand sign to the exec.Command and nothing is working. I simply don&#39;t know what am I doing wrong and I would be very grateful for any advice. For the time being, I think what may be happening is that the system thinks that my updater program has become a zombie (since the systemd service which spawned it died) and kills it as well.</p> <p>Should I move this thread to learngolang instead, please tell me - I wouldn&#39;t like to create a mess on the subreddit.</p> <p>kind regards (need to sleep a bit ;))</p> <hr/>**评论：**<br/><br/>tgulacsi: <pre><ol> <li>Download the new binary to $name.new, check it&#39;s ok.</li> <li>mv $name.new $name</li> <li>Stop. Systemd should restart the service.</li> </ol></pre>wjkohnen: <pre><p>This should be the accepted answer.</p></pre>absdevops: <pre><p>I guess a symlink would also work?</p></pre>wjkohnen: <pre><p>Yes. </p> <p>To elaborate: A nice, but under-used property of POSIX filesystems is that renames (<code>mv</code> does a rename <em>if</em> source and dest are on the same fs) are <strong>atomic</strong>. I.e. at any point in time (concurrently or after a crash) the directory entry either points at the old or at the new inode. Though this is only valid for each rename. </p> <p>But you can compound the atomic switch by using directories:</p> <p>Point a canonical symlink to a versioned directory and point another symlink to a binary in the canonical subtree. The symlink to the binary is what you use in the system service spec. When upgrading create a new versioned directory, check if everything is sound, create a temp symlink to the new directory, rename temp symlink to canonical symlink, voila. </p></pre>tgulacsi: <pre><p>Symlink pointing to where? Shouldn&#39;t overwrite the old binary till we checked that the new is complete and working.</p></pre>absdevops: <pre><p>Well i was thinking you have a binaryname-releases/binary-1.0 and binaryname-releases/binary-2.0 - when you need to upgrade/downgrade, just run a command ln -snf binaryname-releases/binary-$VERSION /target/dir/binary and restart. </p> <p>We did this with our tomcat versioning pre-docker days - but I like wjkohnen&#39;s idea of the 2 symlinks, the binary and the folder and using mv to ensure an atomic operation.</p></pre>serverangels: <pre><p>This is one of the rare places where systemd shines. If I was to write a golang service, i&#39;d use systemd <a href="http://0pointer.de/blog/projects/socket-activation.html">socket activation</a>. This allows the service to stay up during upgrade and is basically inetd on steroids.</p> <p>There&#39;s example code:</p> <p><a href="https://github.com/coreos/go-systemd">https://github.com/coreos/go-systemd</a></p> <p>as well as a bit of background information.</p> <p><a href="http://0pointer.de/blog/projects/socket-activation.html">http://0pointer.de/blog/projects/socket-activation.html</a></p> <p>Thanks for reminding me, I need to have a play with this stuff!</p></pre>Subjunctive__Bot: <pre><p><strong>If I were</strong></p></pre>toudi: <pre><p>I couldn&#39;t fall asleep so I started googling again. It seems that the reason was due to systemd. It has nothing to do with go. All I had to do was to add:</p> <pre><code>KillMode=process </code></pre> <p>into service definition. systemd kills spawned children by default when the main process dies..</p> <p>Now I really need some sleep :D</p></pre>epiris: <pre><p>Are you sure that you have to update the binary, meaning there is no opportunity to abstract the update details into a schema of some kind to be loaded methodically at runtime? If so what problem are you trying to actually solve? If it is service availability I would ask a good way to solve that, there are many standard ways depending on your environment and service. Finally I would do my best not to bypass the secure distribution, package signing, dependency management, system visibility and so on of package managers for a bare metal install. </p> <p>Now if you still are set on your chosen solution, I would consider at least breaking it into two services such as a <em>service-updater</em> and <em>service</em> which depends on <em>service-updater</em>. Have <em>service</em> run /servicebindir/service which is a symlink to /servicebindir/service.v1.N. The <em>service-updater</em> could simply be sleeping at all times until a updated binary is available. Then it downloads it to the servicebindir, updates the symlink and calls sudo systemctl restart &lt;servicename&gt; and waits for a condition to consider the new version &#34;okay&#34;, with the potential to rollback/blacklist the old version if need be. Now you have 2 services that do one thing, update a thing or run a thing which may cause less pain down the road.</p></pre>nesigma: <pre><blockquote> <p>there are many standard ways depending on your environment and service</p> </blockquote> <p>Could you please elaborate on those ways?</p></pre>toudi: <pre><p>maybe epris could mean stuff like having an apt repository or a rpm repo ? I&#39;m just guessing. depending on the system, you could even write hooks which execute before system stop, after start, etc. (at least that&#39;s what I ocassionaly see on systems which update systemd services :D)</p></pre>toudi: <pre><p>thank you so much for the elaborated answer! In fact, I think that I have already made the code in a way you suggested, but due to late hour yesterday and thinking too narrow I wasn&#39;t able to spot it ;) Basically, the updater and the &#39;downloader&#39; program is the same program. The updater part is executed when you invoke the program with a -u argument. So you could look at it in a way that the thing which is executed by systemd is just the downloader - i.e. a progam which checks if there is a new version of the binary, then downloads it and executes it. So even though in my case they are <em>located</em> in the same binary, that&#39;s actually irrelevant. Because, even if in n+1 release the updater code changes, the downloader part won&#39;t. So I can fire my updater without ever stopping the downlodaer systemd. I&#39;m sorry, it probably makes no sense but that&#39;s the best way I can explain it. Bottom line is - I don&#39;t actually need to update the downloader. I thought I had to and that&#39;s what lead to this whole confusion.</p></pre>mwholt: <pre><p>What was your final exec code in conjunction with that systemd setting? Like, did you use any special process attributes in the end? Or does it work without them?</p></pre>toudi: <pre><p>I still have setsid and pdeathsig but I am quite convinced that this is unnecesary. Apart from these, it is a perfectly ordinary sudo call, without any ampersands and what have you. I&#39;ll reply again tomorrow as I turned off the machine and I want to sleep a bit (have been debugging this for at least 5 hours :D)</p></pre>mwholt: <pre><p>Cool, get some sleep and keep me posted! I&#39;m curious what you discover.</p></pre>toudi: <pre><p>Ok. I removed setsid and pdeathsig and everything still works the same ;) Although I believe that this whole thing can be solved easier - please see the first answer (About mv + systemd respawning the service)</p></pre>gxti: <pre><p>The problem with that approach is that after the original main process exits, systemd thinks your service is stopped. The reason it was killing all your processes before is that after the service stopped (due to the main process quitting), it terminated the rest of the processes in the cgroup to tidy up. So making it not kill them solves the symptom, not the underlying cause. systemd doesn&#39;t play nicely with processes that fork and re-exec, unfortunately.</p> <p>tgulacsi&#39;s solution is the simplest, so if that does what you need then I&#39;d go that route. If you need to pass a listening socket from one to the other, you can use a socket unit (man systemd.socket) that the service inherits, that way you have continuity during the restart. That&#39;s the approach that I use for simple network daemons.</p> <p>If you need both processes to stick around during the changeover for other reasons, for example so that you can prove the new process works and to rollback if it doesn&#39;t, then things get messier.</p></pre>qu33ksilver: <pre><p>Yea systemd has its quirks. Glad that you had it resolved.</p></pre>cavaliercoder: <pre><p>This might be a better question for the linux sub. The problem lies in the way processes are managed by the kernel. I would have thought reparenting the child process using setsid would fix this.</p></pre>tex0: <pre><p>Yes, you start the updater in a different process group.</p></pre>chipaca: <pre><p>The first thing that comes to mind is to not write the service as a <code>for {}</code>; write it as just one iteration, and use a systemd <code>.timer</code> unit to do the for. Then the next time the timer fires, it&#39;ll run whatever the binary is.</p> <p>My next thought is, why stop the service before replacing the binary? Surely you can do that in the other order? Then all you need to do is quit, and systemd will start you again.</p> <p>As to <em>why</em> your child process is getting killed, that might be for a number of reasons. Does it also happen when not running under systemd, for example?</p></pre>toudi: <pre><p>ah, that is a very nice tip (about timer), thank you very much! as I understand, systemd won&#39;t fire the service one more time until the first spawned process finishes?</p> <p>why stop the service before replacing the binary - well because unless I do that, linux gives me an error that &#39;text file is busy&#39;. when I stop the service, I can copy the file.</p> <p>why the child process is getting killed - this was purely due to KillMode default setting (which is control-group)</p></pre>chipaca: <pre><p>ah! you&#39;re doing it wrong: don&#39;t modify the executable <em>in place</em>. If you have a problem mid-modification (the download is interrupted, or any other thing), you&#39;ll end up with an unusable binary.</p> <p>Download it to a different file in the same directory, and then rename it. That shouldn&#39;t give you the &#39;text file busy&#39; error, and if you sync the file before the rename, and the directory after, you&#39;ll be safe from a host of issues.</p></pre>toudi: <pre><p>hmm I don&#39;t think I understand. when I was doing it manually, I downloaded the binary to /tmp (different directory from the original binary), then copied the file over. However, when the service happened to be running, I was receiving &#39;text file busy&#39;. Should I use mv rather than cp ?</p></pre>chipaca: <pre><p><code>mv</code>, and from the same filesystem otherwise <code>mv</code> is <code>cp</code>+<code>rm</code>.</p> <p>The &#39;same directory&#39; was because that way you know it&#39;s the same filesystem, also because if you&#39;re doing this from go you&#39;d have to open both directories to sync them after the mv)</p></pre>jrkkrj1: <pre><p>Something like this help? <a href="https://github.com/inconshreveable/go-update" rel="nofollow">https://github.com/inconshreveable/go-update</a></p></pre>rv77ax: <pre><p>You need &lt;servicename&gt;.path and <a href="mailto:systemctl-restart@.service" rel="nofollow">systemctl-restart@.service</a>.</p> <p>&lt;servicename&gt;.path will watch if binary changed and restart the service based on service name,</p> <p>```` [Unit] Description=&#34;Watch {{ service_bin }}&#34;</p> <p>[Path] PathChanged={{ service_bin }} Unit=systemctl-restart@%p.service</p> <p>[Install] WantedBy=multi-user.target ````</p> <p><a href="mailto:systemctl-restart@.service" rel="nofollow">systemctl-restart@.service</a>,</p> <p>```` [Unit] Description=systemctl-restart@%i</p> <p>[Service] Type=simple ExecStart=/bin/systemctl restart %i ````</p> <p>PS: on mobile, not sure how the formatting works.</p></pre>st3fan: <pre><p>You could also consider going the Docker way. I’m doing this and it is working out pretty well. More moving parts but very structured. Ask me anything. </p></pre>