Job scheduler for long-running job w/ persistence

<p>Hi!</p> <p>I&#39;m currently in need of job scheduler to distribute long-running (potentially forever) jobs with persistence options (i.e. to Redis / embedded DB), healthcheck (hearbeat), possibility to end job by manager&#39;s command.</p> <p>If worker fails for some reason (i.e. it&#39;s crashed, machine died - healthcheck failed), manager need to know that and be able to reschedule it&#39;s jobs to others.</p> <p>We have 1 manager and a lot of workers to distribute work to. We&#39;re currently doing it via RDMBS, but we want to get rid of that approach as it&#39;s not flexible enough and hard to reuse.</p> <p>Thanks.</p> <hr/>**评论：**<br/><br/>jns111: <pre><p>Here&#39;s a ton of possibilities.. <a href="https://github.com/avelino/awesome-go#distributed-systems" rel="nofollow">https://github.com/avelino/awesome-go#distributed-systems</a></p></pre>dobegor: <pre><p>I need <em>long-running</em> (maybe forever) jobs. That&#39;s why I asked this here.</p></pre>tvmaly: <pre><p>You could use the os/exec command to spawn an external sub process then call setsid on it. </p> <p>Store the data about the job and its pid in something like boltdb for persistence. </p> <p>To check on the process to see if it is still running use the os.FindProcess passing in the pid. </p></pre>dobegor: <pre><p>I need to distribute these jobs across different physical machines.</p></pre>tmornini: <pre><p>Docker + AWS ECS or Kubernetes is your friend. :-)</p></pre>dobegor: <pre><p>I&#39;m not asking about scaling infrastructure. My question is about <em>distributing work</em>, not about <em>scaling workers</em>.</p></pre>yuhatemepvp: <pre><p>Grumpy much? These are all reasonable suggestions. Ive written this a couple times. Once built on top of consul to handle events, and later a more raw implementation using grpc. OS.exec is great. Only gotcha is a long standing data race with cmd.Wait() and the stdiopipes. You can work around that by creating your own pipereader/writers though.</p></pre>ChristophBerger: <pre><p>Scaling is only one aspect of containerization and orchestration. Others are distributing work and managing failover, so Docker and friends seem a nice fit for your needs.</p></pre>