Getting the system block size

<p>When manually doing big file reads or writes, you generally need to decide on the size of the read/write buffer; too large and you&#39;ll hog memory, too small and the copy will take longer. In general, you want to use the drive&#39;s block size or some integer multiple of it, or you&#39;ll end up doing extra reads/writes.</p> <p>I was trying to figure out the correct code to find this size and only just figure it out. I first tried using something like:</p> <pre><code>var stat syscall.Stat_t syscall.Stat(os.DevNull, &amp;stat) stat.Blksize </code></pre> <p>This worked fine when I tested it, but it turns out that <code>Stat_t</code> and <code>Stat</code> are undefined on Windows. It turns out that there is already a function buried in <code>syscall</code> that I somehow missed despite its straightforward name: <code>Getpagesize</code>. The code above can be reduced to a simple one-liner. It also works on Windows.</p> <p>I thought I&#39;d post this here in case anyone was having the same problem.</p> <p>EDIT: It turns out that <code>Getpagesize</code> returns the memory page size, not the block size. See <a href="/u/aaron42net">/u/aaron42net</a>&#39;s <a href="https://www.reddit.com/r/golang/comments/35gx5z/getting_the_system_block_size/cr4bpbe">helpful comment</a>.</p> <hr/>**评论：**<br/><br/>aaron42net: <pre><p>Using your own write buffer prior to calling Write() does avoid unnecessary system calls, which are expensive if you do a lot of them. However, the buffer size can be kind of arbitrary without affecting the performance much. Appending a byte to a buffer a thousand times and then calling Write() is faster than calling Write() of one byte a thousand times. But whether you use a 1k or 4k or 8k buffer doesn&#39;t change the result much.</p> <p>Os.Getpagesize refers to the size of a <a href="http://en.wikipedia.org/wiki/Page_%28computer_memory%29">virtual memory page</a> not a block of disk. Also, disk writes are generally buffered by the OS unless you use File.Sync(), so you don&#39;t really have to align them with disk blocks. It won&#39;t hurt, but it&#39;s likely not worth much extra effort.</p></pre>autowikibot: <pre><h5> </h5> <h6> </h6> <h4> </h4> <p><a href="https://en.wikipedia.org/wiki/Page%20%28computer%20memory%29" rel="nofollow"><strong>Page (computer memory)</strong></a>: <a href="#sfw" rel="nofollow"></a> </p> <hr/> <blockquote> <p>A <strong>page</strong>, <strong>memory page</strong>, or <strong>virtual page</strong> is a fixed-length contiguous block of <a href="https://en.wikipedia.org/wiki/Virtual_memory" rel="nofollow">virtual memory</a>, described by a single entry in the <a href="https://en.wikipedia.org/wiki/Page_table" rel="nofollow">page table</a>. It is the smallest unit of data for memory management in a virtual memory <a href="https://en.wikipedia.org/wiki/Operating_system" rel="nofollow">operating system</a>.</p> <p>Virtual memory allows a page that does not currently reside in main memory to be addressed and used. If a program tries to access a location in such a page, an exception called a <a href="https://en.wikipedia.org/wiki/Page_fault" rel="nofollow">page fault</a> is generated. The hardware or operating system is notified and loads the required page from the auxiliary store (hard disk) automatically. A program addressing the memory has no knowledge of a page fault or a process following it. Thus a program can address more (virtual) RAM than physically exists in the computer. Virtual memory is a scheme that gives users the illusion of working with a large block of contiguous memory space (perhaps even larger than real memory), when in actuality most of their work is on auxiliary storage (disk). Fixed-size blocks (pages) or variable-size blocks of the job are read into main memory as needed.</p> <p>A transfer of pages between main memory and an auxiliary store, such as a hard disk drive, is referred to as <a href="https://en.wikipedia.org/wiki/Paging" rel="nofollow">paging</a> or swapping. </p> </blockquote> <hr/> <p>^Interesting: <a href="https://en.wikipedia.org/wiki/Random-access_memory" rel="nofollow">^{Random-access} ^memory</a> </p> <p>^Parent ^commenter ^can <a href="/message/compose?to=autowikibot&amp;subject=AutoWikibot%20NSFW%20toggle&amp;message=%2Btoggle-nsfw+cr4bpfb" rel="nofollow">^toggle ^NSFW</a> ^{or<a href="#or" rel="nofollow"></a>} <a href="/message/compose?to=autowikibot&amp;subject=AutoWikibot%20Deletion&amp;message=%2Bdelete+cr4bpfb" rel="nofollow">^delete</a>^. ^Will ^also ^delete ^on ^comment ^score ^of ^-1 ^or ^less. ^| <a href="http://www.np.reddit.com/r/autowikibot/wiki/index" rel="nofollow">^FAQs</a> ^| <a href="http://www.np.reddit.com/r/autowikibot/comments/1x013o/for_moderators_switches_commands_and_css/" rel="nofollow">^Mods</a> ^| <a href="http://www.np.reddit.com/r/autowikibot/comments/1ux484/ask_wikibot/" rel="nofollow">^Magic ^Words</a></p></pre>will_alexander: <pre><p>That&#39;s really helpful, thanks for the info! It seems like if I need to choose some arbitrary buffer length, the page length still makes the most sense. Is this correct?</p></pre>anacrolix: <pre><p>I&#39;ve been following and examining common standard library buffer sizes for years. I&#39;ve observed that buffers are usually some factor of the size of the next layer&#39;s buffer. For example pages are often 4K. Disk clusters or logical sectors can be 8-32K or more. OS socket buffers are often 64K. Pipes, 32-128K or more depending on all manner of details. So here is the heuristic: the smallest your buffer should be is the largest amount of data the consuming end can efficiently consume in one call. There&#39;s more but that&#39;s the most important.</p></pre>