Keep state over multiple instances

<p>TL;DR Looking for a way too share state over cluster without database </p> <p>Sup. Im trying to build a highload service (its kinda pet ) and it should take about 300-500 requests per second. Its should keep in RAM about 500-700Mb stack of text lines (in bytes i guess), return one line by request and iterate to a next one. (gonna add namespaces and, interface and some other funcs, but its not very important here). I guess valyala/fasthttp could give me enough speed for one instance too keep it, but I have about 8 windows servers behind load balancer and able too use it. I think its an overkill to use big database like potgre or mysql too save line number so I very interested (not even for this project but for an my erudition) is there good way to keep state over cluster. I have only one idea. Connect incenses to a chain and send changes to a neighbors. But it seems not very reactive. Also I can request state for each iteration from neighbor and take newest but generates overhead. </p> <p>And yeah, I know, If I gonna build cluster from scratch I should handle concurentions and locks. </p> <hr/>**评论：**<br/><br/>justinisrael: <pre><p>Its not clear to me what you consider to be a database or not in this case. Are you against any type of broker service storing the data for your cluster? Are you just against a formal database like postgres? You could use something like Kafka as an event source that keeps your cluster in sync. Or something like redis if you need to treat the data like a queue. Nats.io can provide queue semantics if you have 8 servers that need to work on FIFO data. </p></pre>recurrency: <pre><p>I&#39;m a huge advocate of the Kappa architecture: <a href="http://milinda.pathirage.org/kappa-architecture.com/" rel="nofollow">http://milinda.pathirage.org/kappa-architecture.com/</a>. Tl;dr: Just use Kafka as a WAL for all your state.</p></pre>jerf: <pre><p>The right answer depends on a lot of parameters you don&#39;t give here, like how important it is for it to never give out identical lines, how distributed you need this to be, how catastrophic it is if you lose one node, etc. </p> <p>If it doesn&#39;t matter to you much what is going on, and you don&#39;t mind manual intervention on node failure, a simple socket on a centralized server that hands out numbers on demand will work just fine. But those are big &#34;ifs&#34;. If you need clusterability and automatic failover and so on and so on, you end up still coming out ahead to just use a database or something, because that will still be easier than writing something <em>correct</em> with all those properties.</p> <p>Edit: There are servers that have as a feature this &#34;counter&#34; ability... Keep an eye out for those, and <em>read the fine print</em> on exactly what they provide.</p></pre>user3961: <pre><p>Etcd</p></pre>cittatva: <pre><p>Raft</p></pre>014a: <pre><p>Keep state in the client. Send an offset or something with every request.</p> <p>If that&#39;s not an option: </p> <p>Go isn&#39;t really designed to be like Erlang/Elixir, where you can do IPC between instances of your program to communicate that state. For better or worse. There might be ways to set up some sort of consensus algorithm between your apps, but its not worth it.</p> <p>Just use a database. Etcd is probably the easiest one to set up and its blazing fast. Consul or Redis would also work.</p></pre>