Concurrent read of multiple files

<p>Hi all - I&#39;m pretty new to Go (~3 weeks).</p> <p>I am attempting to concurrently read multiple files and eventually store data from the files in a map. I am aware that maps aren&#39;t thread safe and that I will have to implement a mutex. The files I am reading are blocking, so during testing, I can open a read on the file and the read (eventually) goes through when the file is unblocked. (I have no control over this.)</p> <p>My initial thought is to create goroutines for each read (about 50 files) and pass the data read to a channel that unlocks the mutex and locks it again.</p> <p>At the same time, the map is being read from which I suppose will have to synchronize with the writes as well.</p> <p>Am I approaching this issue correctly?</p> <hr/>**评论：**<br/><br/>Sythe2o0: <pre><p>You don&#39;t want to use both a channel and a mutex, you should use one or the other.</p> <p>A channel that takes in the data to put in the map, and is only accepted from in one goroutine would be a way to approach this, if you pass around pointers. So each reader goroutine would only get to send off its data when the writer goroutine was ready for the next input.</p></pre>tmornini: <pre><p>Right answer!</p> <p><a href="https://github.com/tmornini/async-file-processing-go" rel="nofollow">Here&#39;s an example I just whipped up.</a></p> <p>You may need to adjust the location of the words file on your system, and it reads the same file (words) for all readers, but it should make it clear enough how to make this fully async.</p></pre>robe_and_wizard_hat: <pre><p>You can do this without a channel. Just wrap the map in a struct that has a sync.Mutex and lock the methods to set/read the values.</p> <p>So, spin up your 50 goroutines, but have them all try to do a myStruct.Write(data ). That method would then be guarded by a mutex.</p> <p>Also yeah you&#39;ll also need to guard reads to the map with the same mutex. You could use a RWMutex here if you wanted, as well.</p></pre>tmornini: <pre><p>Or you could use a channel.</p> <p><a href="https://go-proverbs.github.io" rel="nofollow">This is a Go proverb, after all</a></p></pre>allhatenocattle: <pre><p>If program structure is:</p> <pre><code>* do all map initialization from data files * only read from map </code></pre> <p>Then you will only need to serialize access to the map during initialization. if the map will be updated after initialization, you would need locking around reads and writes.</p> <p>here is an example of what I think you are trying to do: <a href="https://play.golang.org/p/lWl9vmiy6O" rel="nofollow">https://play.golang.org/p/lWl9vmiy6O</a></p> <p>(it won&#39;t run on playground since it accesses files and has the golang.org/x/sync/errgroup package, but the code runs fine on my laptop)</p></pre>ericzhill: <pre><p>Do it the go way:</p> <ol> <li>Spawn a goroutine for each file that reads, processes it, and sends results down a result channel.</li> <li>In your main goroutine, for range over the result channel and add to a map as needed. No mutex is necessary. </li> </ol> <p>Edit: sorry for bold... On mobile.</p></pre>