<p>Hi,</p>
<p>I'm in need of writing a tool that would be able to monitor a lot of files (tens of thousands, maybe more) and perform certain action when they are created, modified or deleted.</p>
<p>For this I thought GO might be the perfect language to develop such tool in it as at the end you would get a single executable file that's easy to distribute.</p>
<p>I've searched of what solutions are available for GO and they don't sound that promising (at least what I was able to find). Either they are not maintained for quite some time or they don't support native OS FS change APIs, like FSEvents on OSX or something like ReadDirectoryChanges on Windows. Using pooling a doing a file.stat() might not be such a good idea for such large number of files.</p>
<p>For this I found <a href="https://github.com/emcrisostomo/fswatch/tree/master/libfswatch">https://github.com/emcrisostomo/fswatch/tree/master/libfswatch</a> which seems to support native APIs for OSX and Windows (the two OSs that I also need to have the tool run on).</p>
<p>The problem is that that is a C++ (with a C wrapper) library.</p>
<p>How realistic is using this in GO? Can I use it with CGO? Would there be performance issues of using this for detecting the events and then passing the path of the files that changed to GO where I will perform the required actions? </p>
<p>Any pointers in how I can make this work with GO?</p>
<hr/>**评论：**<br/><br/>gopher1717: <pre><p>You should take a look at <a href="https://github.com/fsnotify/fsnotify">https://github.com/fsnotify/fsnotify</a></p>
<p>Cross-platform and pure Go.</p></pre>daniels0xff: <pre><p>FSEvents is not supported on OSX from what I see in their readme.</p></pre>justinisrael: <pre><p>I've used both <a href="https://github.com/rjeczalik/notify">https://github.com/rjeczalik/notify</a> and fsnotify. </p>
<p>Fsnotify is pure Go because it uses syscalls. But it only supports kqueue on osx which is less efficient for tons of files because it needs a lot of file descriptors. </p>
<p>rjeczalik/notify does support FSEvents, but it needs cgo to do that because it needs to access the osx core libraries. But you end up with less file descriptor usage when you watch a ton of directories. </p></pre>corvuscrypto: <pre><p>You can access syscalls and since OSX is Unix based shouldn't that be enough to at least get the basic notifications working? The implementation shouldn't be too hard.</p>
<p>Edit: Looking at some quick docs it looks like OSX built on top of kqueue. However it may be easier to just make a binding to one of the several C libs already out there for this. You'll have some overhead with the FFI but probably not enough to really hurt you. Or you can write to a shared socket from a C program that just watches for events so a golang program can read from that same socket to retrieve the events. Lots of options here.</p></pre>KenjiTakahashi: <pre><p>Alright, to elaborate a bit more. If you literally need to perform an action upon FS event, maybe <a href="https://github.com/cortesi/modd">https://github.com/cortesi/modd</a> will be a suitable solution for you? It builds on the aforementioned <a href="https://github.com/rjeczalik/notify">https://github.com/rjeczalik/notify</a> library and in my experience works great (at least on Linux).</p>
<p> </p>
<p>I've worked around the FS events on different systems in the past and, sadly, it's a clusterfuck.</p>
<p>Every system has it's own, macOS even has two (by historical reasons, building on BSD, it supports kqueue).</p>
<p>Kqueue is no good for many files, as it opens descriptors for every file.</p>
<p>FSEvents requires CGo and linking to IOKit, so cross-compilation is out of question. It's also not fully reliable (might skip events under heavy load, known issue, happened to me many times).</p>
<p>Windows is windows :-]. The interface is strange and obnoxious, but AFAICT it performs fine, once working.</p>
<p>Inotify is probably the best of them, but Linux only.</p>
<p>Solaris has something, that I don't even remember the details.</p>
<p> </p>
<p>Also, be aware that none of these solutions will work for user space FSes, such as NFS or SSHFS (with the exception of NFS and FSEvents, because Apple added support on both sides).</p>
<p>If you asked me, I'd just go with the good ol' polling. Anyway, good luck :-).</p>
<p>edit: Formatting</p></pre>Hexodam: <pre><p>Ah, I know exactly what you mean, the use case and purpose. Been testing a few POC's for this myself for the past year or so.</p>
<p>My focus has mostly just been with Linux and Windows but not OSX so I have not run into what you speak of.</p>
<p>I first tested fsnotify ( <a href="https://github.com/fsnotify/fsnotify" rel="nofollow">https://github.com/fsnotify/fsnotify</a> ) but did not like how it worked on Windows.</p>
<p>Then I found Notify ( <a href="https://github.com/rjeczalik/notify" rel="nofollow">https://github.com/rjeczalik/notify</a> ) which worked much better. Sadly I have not had enough free time to go further but I have big plans.</p>
<p>For Windows use cases I highly recommend you take a look at USN Journals, <a href="https://blogs.technet.microsoft.com/tip_of_the_day/2015/03/28/tip-of-the-day-reading-the-usn-journal/" rel="nofollow">https://blogs.technet.microsoft.com/tip_of_the_day/2015/03/28/tip-of-the-day-reading-the-usn-journal/</a> . It is definitely how any file monitor should interface with Windows for this task.</p>
<p>Btw, are you planning on open sourcing the tool in the end?</p></pre>daniels0xff: <pre><p>If I manage to do it, yes. Most likely it will be on GitHub.</p></pre>KenjiTakahashi: <pre><p>There's also <a href="https://github.com/rjeczalik/notify" rel="nofollow">https://github.com/rjeczalik/notify</a></p></pre>mcouturier: <pre><p>You might also use an existing tool to handle the watching and process the events in Go: <a href="https://github.com/emcrisostomo/fswatch" rel="nofollow">https://github.com/emcrisostomo/fswatch</a></p></pre>
