Need a disk-based cache for tree-like structures

<p><a href="https://github.com/ChrisTrenkamp/goxpath/issues/6" rel="nofollow">Here&#39;s the backstory</a></p> <p>Let&#39;s say I&#39;m parsing a large XML file and building out the tree in memory. Eventually, it will start using up a lot of memory and it needs to be freed. However, I can&#39;t just release the memory, because if the user wants to go back up the tree, the data will be lost.</p> <p>I need some kind of disk cache where I can dump parts of the tree and free some memory, but be able to retrieve that data again if it&#39;s needed. Does anyone know of a good library to use?</p> <hr/>**评论：**<br/><br/>funny_falcon: <pre><p>First simple way is to use <a href="https://golang.org/pkg/encoding/gob/" rel="nofollow">https://golang.org/pkg/encoding/gob/</a> . It has limitation of max size (if I remember correct, 256MB, or 1GB). And it will deserialize whole tree on read.</p> <p>Second simplet way is to use key-value storage. You may use virtual node-ids instead of pointers:</p> <pre><code> &lt;xml&gt; &lt;parent name=&#34;John&#34;&gt; &lt;child sex=&#34;male&#34; name=&#34;Bill&#34;&gt; &lt;toy&gt;Car&lt;/toy&gt; &lt;/child&gt; &lt;child sex=&#34;female&#34; name=&#34;Mary&#34;&gt; &lt;toy&gt;Barbie&lt;/toy&gt; &lt;/child&gt; &lt;/parent&gt; &lt;xml&gt; </code></pre> <p>Could be stored as (simplified form):</p> <pre><code>&#39;1$&#39; : &#39;parent name=&#34;John&#34;&#39; &#39;1$cnt&#39;: &#39;2&#39; &#39;1$0001&#39;: &#39;2&#39; &#39;1$0002&#39;: &#39;4&#39; &#39;2$&#39; : &#39;child sex=&#34;male&#34; name=&#34;Bill&#34;&#39; &#39;2$cnt&#39; : &#39;1&#39; &#39;2$0001&#39; : &#39;3&#39; &#39;3$&#39; : &#39;toy&#39; &#39;3$cnt&#39;: &#39;0&#39; &#39;3$text&#39;: &#39;Car&#39; &#39;4$&#39; : &#39;child sex=&#34;female&#34; name=&#34;Mary&#34;&#39; &#39;4$cnt&#39; : &#39;1&#39; &#39;4$0001&#39; : &#39;5&#39; &#39;5$&#39; : &#39;toy&#39; &#39;5$cnt&#39;: &#39;0&#39; &#39;5$text&#39;: &#39;Barbie&#39; </code></pre> <p><a href="https://github.com/boltdb/bolt" rel="nofollow">https://github.com/boltdb/bolt</a></p> <p><a href="https://github.com/cznic/kv" rel="nofollow">https://github.com/cznic/kv</a></p> <p>BoltDB has &#34;buckets&#34;, that could be nested to form tree. But it could be inefficient in term of space.</p> <p>But if you want performance/space efficiency and laze retrieval (ie do not deserialize whole tree), then you ought to build your own library.</p></pre>barsonme: <pre><p>Here&#39;s a read-only, disk-based radix tree: </p> <p>github.com/sermodigital/drt</p> <p>regardless of the route you go, mmap is your friend.</p></pre>__sahib__: <pre><p>Maybe <a href="https://github.com/boltdb/bolt" rel="nofollow">BoltDB</a> might be applicable here too. You can build hierarchies using buckets. The database file is mmap()&#39;d to memory.</p></pre>pmjtoca: <pre><p>I have been using lmdb for this type of requirements. robust and fast. lmdb is a C lib, good binding in Go exist (the gomp by msackman is my choice).</p></pre>Mythiix: <pre><p>I had trouble finding it. Link for others <a href="https://github.com/msackman/gomdb" rel="nofollow">https://github.com/msackman/gomdb</a></p></pre>