Concurrent architecture help?

<p>I&#39;m wondering if anyone has any ideas to solve my problem.</p> <p>I have a Go app serving HTTP requests. I have this Go app running on several machines. I need some way to coordinate between them such that each request gets a unique, incrementing ID with no gaps. I&#39;m trying to minimize the blocking that the requests need to do, and minimize the cross-chatter between servers.</p> <p>Right now I&#39;m thinking that there is a Ticker Server that generates sequential ranges of ID&#39;s and the Go app requests a batch when it&#39;s running low, generating all the ID&#39;s in that range and pushing them onto a channel. Each request in that Go app then pulls an ID from a channel.</p> <p>Anyone have a better architecture?</p> <hr/>**评论：**<br/><br/>ultra_brite: <pre><blockquote> <p>incrementing ID with no gaps</p> </blockquote> <p>give up that requirement because you&#39;re just asking for trouble. Just use uuids just like any distributed system. If all your servers rely on a unique third party then your architecture isn&#39;t distributed. It is centralized.</p></pre>CaptaincCodeman: <pre><p>Seems like a bizarre and pointless requirement but you could just use a database with an identity column. Each request does an insert to retrieve the next number.</p> <p>It&#39;s a terrible idea.</p></pre>niosop: <pre><p>How stringent are the various requirements?</p> <p>Trying to do all 3 with solid guarantees is hard. Getting batches from a central tracker is fine, but each app will need to persist a record of the current batch and which have been used, otherwise you&#39;ll end up with gaps in the case of an application crash.</p> <p>If the &#34;no gaps&#34; rule isn&#39;t that important, then you can do away with a central tracking server and just give each application it&#39;s own prefix or suffix to use. So Server-01 would issue 000101, 000201, 000301, etc. Server-02 would issue 000102, 000202, 000302, etc. Then they just need to keep track of where they are by writing it out to file every time they use a number.</p> <p>If unique is more important than no gaps, then write out the last issued before handing it off to the requestor. You&#39;ll have a gap if the service crashes between writing the record and giving it to the requestor, or if the connection dies, or any other number of reasons. It&#39;s why exactly once delivery is difficult.</p> <p>My recommendation is to relax the no gaps rule if possible, then you&#39;ll be able to satisfy the uniqueness constraint more easily.</p></pre>danredux: <pre><p>The no-gaps rule only needs to stand if there are no crashes or issues.</p></pre>qu33ksilver: <pre><p>It would be nice if you can give further background about this requirement. I am thinking, can we come up with a composite sort of ID like server_name+timestamp which allows you to know which server it came from and at the same time indicates order.</p> <p>Or if you are just entering all of this data at a central DB, just use a identity column which will auto-generate increasing nos. and store them.</p></pre>sairamk: <pre><p>Your approach sounds reasonable.</p> <p>If your app is consuming say 10k a second, distribute 600k in a separate go routing which prepares the ids and fetches them every minute or when required.</p> <p>Central server could be a mutex to decide. If you don&#39;t want a single central server, you could trying using some consensus to elect a leader among your hosts which generates these ids and coordinates. every point of time everyone knows what the state is and can be elected leader. </p> <p>In the second approach, you need to take care of ensuring the sub-system that cannot be reached would not allocate new batches of increments among themselves.</p></pre>programfog: <pre><p>I second the suggestion by ultra_brite about using uuids.</p> <p>I found this slide deck useful when looking at a similar problem in the past</p> <p><a href="https://www.slideshare.net/mobile/davegardnerisme/unique-id-generation-in-distributed-systems" rel="nofollow">https://www.slideshare.net/mobile/davegardnerisme/unique-id-generation-in-distributed-systems</a></p></pre>Matthias247: <pre><p>If there&#39;s no need on incrementing then GUIDs are a common solution for this problem. If you need this incrementing IDs then you will also need coordination between all those machines. E.g. store the last ID in a database, redis, etc. This will come with a serious performance hit and lots of reliability questions (how can you scale it, what do you do if the central server is not there, ...)</p></pre>gizzlon: <pre><p>Hm.. What do you mean by &#34;no gaps&#34; ? Does it mean that every request-id must be in-order across all the servers? So the first request to server A gets 1 and then the next request, to server B, gets id 2? If this is actually what you require I think I would just to one global counter in Redis. It&#39;s probably fast enough to touch on each request, simple to understand and implement and gives you the strongest guaranties. Only problem is that it will be a single point of failure..</p> <p>Edit: Redis docs: <a href="http://redis.io/commands/incr" rel="nofollow">http://redis.io/commands/incr</a></p></pre>