<p>Is there any in-memory column-store in Go?</p>
<p>I want to set my de-normalized data in a in-memory cache/db, but I have varying columns.</p>
<p>So if I have an EAV table, I'll just transform it to a column-variable cache to speed up queries (mainly sums, multiplications and grouping). </p>
<p>I thought about using Cassandra, but it's a bit too much to setup, and the application isn't that big right now, and just deploying one Go app is better.</p>
<p>Even if not in-memory, as long as it's column-store and embedable .</p>
<hr/>**评论：**<br/><br/>no1youknowz: <pre><p>I was going to suggest memsql, <a href="http://www.memsql.com/" rel="nofollow">http://www.memsql.com/</a> but I am not sure it's embedable. Feel free to join their slack channel, which devs frequent and ask there.</p></pre>Emacs24: <pre><p>Is Cassandra columnar?</p>
<p>You can try <a href="http://clickhouse.yandex" rel="nofollow">Clickhouse</a>: it is a lot easier to setup on a single machine (cluster setup is not that easy though).</p>
<p>I wrote a couple of packages to deal with data insertion in a type safe manner (and they are no slouch when it comes to the insertion speed) at my last job where we used Clickhouse:</p>
<ol>
<li><a href="https://github.com/sirkon/ch-encode" rel="nofollow">ch-encode</a> to generate serializing objects for given tables</li>
<li><a href="https://github.com/sirkon/ch-insert" rel="nofollow">ch-insert</a> provides data insertion via HTTP and buffering</li>
</ol>
<p>And there's a <a href="https://github.com/mailru/go-clickhouse" rel="nofollow">database/sql driver working via HTTP interface</a></p></pre>jerf: <pre><blockquote>
<p>Is Cassandra columnar?</p>
</blockquote>
<p><a href="https://www.quora.com/How-is-Cassandra-a-columnar-database" rel="nofollow">No</a>. And to be clear, this is in the spirit of helpfulness for all concerned, not criticism, since a short post with just "No" comes off harsh.</p></pre>drink_with_me_to_day: <pre><p>I'm not sure if Cassandra is columnar, I might have used the wrong word in the post, what I need is just dynamic columns.</p></pre>Emacs24: <pre><p>Got it. Clickhouse may not work for you indeed, I make a transition of column-store into columnar (column store is frequently used a synonym for columnar).</p>
<p>You rather need document based DB.
This may work:
<a href="https://github.com/HouzuoGuo/tiedot" rel="nofollow">https://github.com/HouzuoGuo/tiedot</a></p>
<p>But there's a chance CH still suits your needs: how "dynamic" are your dynamic columns? There might be the case you have a wide (thousands of them) pre-known set of columns, not really dynamic columns. Columnar DB will work for you in this case.</p></pre>SQLServerIO: <pre><p><a href="https://github.com/golang/groupcache" rel="nofollow">https://github.com/golang/groupcache</a> maybe? It isn't strictly a column store but an embedable cache</p></pre>jerf: <pre><p>How big is your data? Are you distributed? Because if the answers are "fits comfortably in RAM I can afford" and "no", my first thought would be to just write some code that does the thing you want. And if the answers are anything else, details will be very helpful in answering the question. A bit more characterization around what data you already have and the kinds of summaries you want would be helpful, too.</p></pre>drink_with_me_to_day: <pre><p>Not distributed, and currently not that big. The 35k rows stored in jsonb is around 8MB, very little and I shouldn't need more than 1GB at a time (already considering the future). The data should fit in ram, and at worse, just throwing more ram at it is a feasible solution. I can't see it needing more than 32GB, ever.</p>
<p>It's basically analitics data that needs to be transformed (f => x +y * 10, if x == 'certain_key'), but I need to allow on-the-fly filtering on the frontend.</p>
<p>So my basic needs are to store everything in memory for fast response times over websocket, but I also need some performant transformation abilities, like count(), sum(), group by, etc.</p>
<p>This could all be done in BI software like Tableau and the likes, but it's integrated to something else and can't be separated.</p>
<p>I could just keep a cache, but I'm not sure I'd get the same performance when doing grouping and transformations that a heavy-duty DB already has.</p></pre>ofpiyush: <pre><p>I've a similar use case coming up in a week. Would love to hear about your solution and the reasons behind it.</p></pre>ecnahc515: <pre><p>Is there any specific reason you need it embeddable? Why can't you run something like Postgres on the same machine as the app using localhost or a unix domain socket. Postgres is supported on all major platforms.</p></pre>iroflmaowtf: <pre><p>this would be my first choice: postgres + unix socket</p></pre>drink_with_me_to_day: <pre><p>I already use postgres and jsonb, but queries still take a bit too long. I need sub-500ms queries.</p></pre>addos: <pre><p>Druid</p></pre>diegobernardes: <pre><p>Take a look at BoltDB and Badger. Its 100% in Go.</p></pre>TheMerovius: <pre><p>Neither of those is in-memory <em>or</em> a column store.</p></pre>diegobernardes: <pre><p>Actually they are, both use mmap, so, if you have enough memory, the OS gonna keep everything in memory. About the column, just serialize a array and you have columns. </p></pre>TheMerovius: <pre><blockquote>
<p>Actually they are, both use mmap, so, if you have enough memory, the OS gonna keep everything in memory.</p>
</blockquote>
<p>That is not the same as an "in-memory database", but fair enough.</p>
<blockquote>
<p>About the column, just serialize a array and you have columns.</p>
</blockquote>
<p>Literally the opposite of what a column store is.</p>
<p>(not saying that you can't implement a columnar database on top of a key-value store - but what you are saying is a little bit like saying "boltdb is a perfectly fine SQL database", just because it's possible to put an SQL engine on top of a key-value store)</p></pre>
