Dealing with a cache such as redis, at scale?

<p>I am looking for pointers or examples that would point me in the right direction in using something such as redis cluster or aerospike at scale using golang. </p> <p>I have tried a bunch of methods and variations but seem to always end up with performance similar to that of using something such as PHP. </p> <p>Though minus something like redis, it far exceeds its counterpart in php.</p> <hr/>**评论：**<br/><br/>iamtherealbill: <pre><p>In general, there are some techniques/methods to avoid which might help.</p> <ol> <li>DONOT use keys command - use indexing instead.</li> <li>DO use pipelining on large sequences of Redis operations</li> <li>DO use persistent connections</li> <li>DO use hashes instead of serializing a dictionary</li> <li>DO use lists or sets instead of serializing collections.</li> <li>DO performance test specific patterns in your code.</li> </ol> <p>Much of this is using Redis in the way it is intended. Just as you don&#39;t want to write Java in Go, you don&#39;t want to use Redis in only one way. Redis is key/value, sure. But that doesn&#39;t mean everything must be a string to string data type.</p> <p>Finally, you seem to be comparing Redis to PHP. This makes no sense. Redis is a data structure server, PHP is a programming language. They are not comparable as they are entirely different things. It would be like trying to compare an elderly blue whale with a red sports car. If you&#39;ve got specific code or a specific sequence we can help better.</p></pre>nastus: <pre><p>Sorry did not mean to make a comparison between redis and PHP but rather the use of golang + redis to php + redis.</p> <p>Though it&#39;s essentially an apples to oranges comparison in some regard, they both would function to do the same thing.</p> <p>I appreciate your comment though! I will try to snippet some code later :)</p></pre>tmornini: <pre><p>Probably just means that the work you&#39;re doing is being overwhelmed by the latency and/or you&#39;re blocked on the Redis connection and need to have more available.</p> <p>If you&#39;re opening and closing the Redis connection per request, that would be bad.</p> <p>If you&#39;re serializing requests, that would also obscure the results.</p></pre>tgulacsi: <pre><p>So, serialization and networking slows down the caching. Maybe it&#39;s unnecessary? Maybe you could try something in-process, like groupcache?</p></pre>nastus: <pre><p>I was looking at groupcache, it has potential for part of my application, there are a bunch of things I&#39;d have to work out to get it to function as intended but I&#39;ll circle back and review it again.</p></pre>iamtherealbill: <pre><p>The catch with groupcache is that it is a write-once cache. Once you set something, it can&#39;t be changed. I really tried and wanted to like groupcache for Redskull but ultimately I&#39;ve decided to use Consul instead for the things that had previously gone into Groupcache. Sure, it is an external process to interact with, but it works a lot better and is enabling significant complexity reduction.</p> <p>But if write-once-never-modify is what you need, along with the peering setup of groupcache being advantageous for you, go for it.</p></pre>sethammons: <pre><p>To add to the other solid advice here: if you can, consider moving to an in memory cache (like LRU) and refresh your cache in the background out to redis. Depending on your use case and traffic patterns, this could help. You can even consider a wrapper around redis and check for requests that can be collapsed (held off because the same request is already issued an wait until a result comes back). Depending on the number of incoming nodes you have, if you are having a poor cache hit ratio, you could add some deterministic routing to your nodes to make better use of your in memory cache.</p> <p>As a side note, all of this is not possible in PHP because each request spins up its own world. That is another value-add for Go as it is long lived and you can leverage many requests together.</p> <p>(On mobile, please excuse typos)</p> <p>Cheers!</p></pre>nastus: <pre><p>Thanks for the advice! Lots I&#39;m going to have to think about and learn.</p></pre>