How boltdb Write its Data?

yhyddr · · 125 次点击 · 开始浏览    置顶
# How boltdb Write its Data? <a name="s8qYY"></a> ### A-Ha! Here’re three questions during reading the source code of BoltDB. I’ll explain our testing procedure to dive into the core of the BoltDB writing mechanism. Code link: [yhyddr/quicksilver](https://github.com/yhyddr/quicksilver) <a name="uxYtc"></a> ### First At first, my mentor posts a question: “Did BoltDB have a temporary file when starting a read/write transaction”. Following the quickstart, all data is stored in a file on disk. Generally speaking, the file has a linerial structure, when inserting, updating and deleting, we have to re-arrange the space occupied by this file. A tempory file should be used in this procedure, for example, .swp file for VIM. Let’s check whether BoltDB uses this method. <a name="WCyl6"></a> ### FileSystem Notify We can use [fsnotify](https://github.com/fsnotify/fsnotify) to watch all the changes in a given directory. Here’s the code: ```go package main import ( "log" "github.com/fsnotify/fsnotify" ) func main() { hang := make(chan bool) watcher, err := fsnotify.NewWatcher() if err != nil { log.Fatal(err) } watcher.Add("./") go func() { for { select { case e := <-watcher.Events: log.Println(e.Op.String(), e.Name) case err := <-watcher.Errors: log.Println(err) } } }() <-hang } ``` Testing procedure as below: [![屏幕录制 2019-10-07 下午2.27.25.mov (2.27MB)](https://cdn.learnku.com/uploads/images/201910/09/41293/qBtwq5EwWw.jpeg!/fw/1240)](https://www.yuque.com/techcats/boltdb/wl9fhl?_lake_card=%7B%22status%22%3A%22done%22%2C%22name%22%3A%22%E5%B1%8F%E5%B9%95%E5%BD%95%E5%88%B6+2019-10-07+%E4%B8%8B%E5%8D%882.27.25.mov%22%2C%22size%22%3A2384216%2C%22percent%22%3A0%2C%22id%22%3A%22nBQZd%22%2C%22videoId%22%3A%228a4f418d1d1e4d8a8512b775c29e06dc%22%2C%22coverUrl%22%3A%22https%3A%2F%2Fcdn.nlark.com%2Fyuque%2F0%2F2019%2Fjpeg%2F176280%2F1570429731070-1b3be028-68b6-4a21-86be-8511a54684d9.jpeg%22%2C%22aliyunVideoSrc%22%3Anull%2C%22taobaoVideoId%22%3A%22238084606367%22%2C%22uploaderId%22%3A176280%2C%22authKey%22%3A%22YXBwX2tleT04MDAwMDAwMTImYXV0aF9pbmZvPXsidGltZXN0YW1wRW5jcnlwdGVkIjoiNWI0MDQ5ZDkzNjYyYTIyNTdiNDAyNjYzNDY1YTBkNmEifSZkdXJhdGlvbj0mdGltZXN0YW1wPTE1NzA1OTAzNjg%3D%22%2C%22docUrl%22%3A%22https%3A%2F%2Fwww.yuque.com%2Ftechcats%2Fboltdb%2Fwl9fhl%22%2C%22card%22%3A%22video%22%7D#nBQZd) We could found our changes have records on the terminal, for example when I create a file named asdf, program log `CREATE asdf`. <a name="4TUJS"></a> ### Update BoltDB File  then we need to write code to store some data. This way we update the BoltDB file and know if there has a tempory file to replace origin file. Because it's obvious that BoltDB as a simple kv database seems do not have access to system directory but just current directory and the database file. We Write code below to insert data : ```go package main import ( "encoding/binary" "log" "os" "time" bolt "go.etcd.io/bbolt" ) var InsertNum int = 15000 func main() { hang := make(chan bool) db, err := bolt.Open("./data.db", 0600, nil) if err != nil { log.Fatal(err) } go func() { times := time.Now() for { db.Update(func(tx *bolt.Tx) error { b, err := tx.CreateBucketIfNotExists([]byte("cats")) if err != nil { return err } num, err := b.NextSequence() log.Println(num) byteid := make([]byte, 8) binary.BigEndian.PutUint64(byteid, num) b.Put(byteid, byteid) if num == InsertNum { log.Println(time.Now().Sub(times)) os.Exit(0) } return nil }) } }() <-hang } ``` the attention I use variable InsertNum control how much data could be inserted. then we run it. (sure you need to run `fsnotify` first.)<br />![image.png](https://cdn.learnku.com/uploads/images/201910/09/41293/X1pwgWbuV8.png!/fw/1240) we inserted 900 data and it cost us 20s. find that it's just writing data.db file. It means that we seem to have an answer to the first question: No! Besides, we also could get the answer by source code. <a name="LjrPM"></a> ## Second We have known the first question's answer, mentor quickly asked me the second.<br />How much data or Which size could a BoltDB file store? we want to know how much. let's find it from source code. BoltDB declare these sizes on the head of a go file <br />![image.png](https://cdn.learnku.com/uploads/images/201910/09/41293/uQnRhnYhW2.png!/fw/1240) and we got the miniFillPercent and maxFillPercent are 0.1 and 1.0, in db.go we also know about BoltDB page size is determined by OS page size.<br />![image.png](https://cdn.learnku.com/uploads/images/201910/09/41293/My51ceBJjK.png!/fw/1240) It means a page may be used in some parts. important that maxMmanpstep = 1 <<30 is 1 GB for remapping. we have known there's no temporary file that means BoltDB use memory or called Mmap to host temporary data.<br />use Mmap means, your BoltDB File should not bigger than your assignable memory space. <a name="uhHtJ"></a> ## Third From before questions, we know a lot about BoltDB. But the mentor posts once more question: In a node, the author sets 50% capacity limit otherwise spill it to two nodes. Does our datafile just use 50% space what a BoltDB database file holds?<br />![image.png](https://cdn.learnku.com/uploads/images/201910/09/41293/3FQOKUrZIX.png!/fw/1240)<br />It's because the strategy about the node spill method to control the size of the node does not overflow OS page size.<br />that question needs we know about the disk layout of the BoltDB file. I'll detailed explanation in my next article. Of course, we know the space ratio should not over 0.5.<br />And now I just give some data for you. I first insert 100000 data which is an auto-increment key and the same value. 1 -> 100000 total 977784 b because of BoltDB store in the file as bytes.<br />and we found our data file use 8.5m space. then I insert 100000 again.<br />![image.png](https://cdn.learnku.com/uploads/images/201910/09/41293/GyVHX9UHlJ.png!/fw/1240) 100000 -> 200000 and file expand to 25.2m when I insert nearly 110000th data.<br />And at last, the file is 25.2m when we hosted 200000 data.<br />![image.png](https://cdn.learnku.com/uploads/images/201910/09/41293/rmdu7gm6mo.png!/fw/1240)

入群交流(和以上内容无关):Go中文网 QQ 交流群:798786647 或加微信入微信群:274768166 备注:入群;关注公众号:Go语言中文网

125 次点击  
加入收藏 微博
暂无回复
添加一条新回复 (您需要 登录 后才能回复 没有账号 ?)
  • 请尽量让自己的回复能够对别人有帮助
  • 支持 Markdown 格式, **粗体**、~~删除线~~、`单行代码`
  • 支持 @ 本站用户;支持表情(输入 : 提示),见 Emoji cheat sheet
  • 图片支持拖拽、截图粘贴等方式上传