[Help] Websockets, Databases and Golang

<p>Hey everyone. I&#39;m building a web app that has a notification system a la Facebook. Even though I&#39;ve been a rails dev for 6+ years, this is kind of uncharted territory for me, believe it or not.</p> <p>On the backend I have a Golang REST service using Gin and Gorm (postgres). I set up an endpoint to be a websocket. The websocket polls the DB to check for new notifications, and sends the count back to the client. I have it set up so that the client asks every five seconds. The message read and send actions are in their own goroutines.</p> <p>Functionally, it works fine. I&#39;m a little concerned about my performance here, though. Just checking my Activity Monitor (OSX), my app is using 99% CPU and 6.5MB of Memory. If I open up another browser and hit the same page, it jumps to about double CPU and 8MB. I tried using pprof to find if I had a memory leak somewhere, but nothing jumped out at me. I&#39;m worried that when I actually get this to AWS and lots of people start using it, it&#39;ll lock up the server in no time flat.</p> <p>Again, this is uncharted territory for me, so this might be normal, or I may have significantly screwed something up. Google and Stack Overflow didn&#39;t seem to provide me with much help, unfortunately. Anyone have any experience with this?</p> <p>Here&#39;s a gist of the relevant parts: <a href="https://gist.github.com/jonlaing/f00ea4b0699af5ff502c">https://gist.github.com/jonlaing/f00ea4b0699af5ff502c</a></p> <p>EDIT: This is also my first major Golang app, so there&#39;s a huge possibility I&#39;ve screwed everything up.</p> <p>EDIT 2: Also, if I take out the code on the client that hits the websocket, my usage drops down to nearly negligible amounts, so I&#39;m pretty confident this is the culprit.</p> <hr/>**评论：**<br/><br/>postman_: <pre><p>The</p> <pre><code>default: break </code></pre> <p>part makes the entire switch non-blocking so the goroutine spins in a loop, wasting cycles.</p></pre>d_rudy: <pre><p>Ah thanks! Never would have thought of that. Just habit for me to put a <code>default</code> at the end of a <code>switch</code>. I&#39;ll give it a shot.</p></pre>ctbel: <pre><p><a href="/u/postman_">/u/postman_</a> doesn&#39;t stress this but it&#39;s important that you understand the difference between switch and select. Have you done the <a href="https://tour.golang.org/list">Tour of Go</a> yet? It&#39;s a great way to get to know the language, doesn&#39;t take a lot of time, it&#39;s interactive and also deals with <a href="https://tour.golang.org/concurrency/5">select</a>.</p></pre>d_rudy: <pre><p>Truth, didn&#39;t occur to me. I did the tour maybe a year ago, but that information apparently didn&#39;t stick with me. I&#39;m much more of a &#34;have a project and build it through trial-and-error&#34; kind of learner than reading, or even the tour. I feel like I could have done that tour a million times, and that concept probably wouldn&#39;t have sunk in until I screwed something up that I needed.</p> <p>Guaranteed I won&#39;t forget it this time, though!</p> <p>Edit: And to drive the point home further, I saw that page earlier today, and clearly didn&#39;t read it carefully enough, haha!</p></pre>intermernet: <pre><p>I know this sounds like patronizing advice, but you should read (or re-read, if you already have read) <a href="https://golang.org/doc/effective_go.html">Effective Go</a>.</p> <p>I re-read it every few months and I&#39;m still finding new bits of useful information. It&#39;s <em>packed</em> with useful info which isn&#39;t often evident on the first (or <em>n</em>th) read.</p> <p>Don&#39;t try to absorb all of the details in one read, but when you come across something that doesn&#39;t seem right, or you seem to be fighting the language, Stare at the relevant sections of that doc in a zen-like fashion.</p> <p>It&#39;s helped me out so many times I&#39;ve lost count. I&#39;m actually thinking of printing it out so I have a hard-copy at hand. (I seem to absorb information better when reading real paper, but YMMV :-) )</p></pre>d_rudy: <pre><p>That one&#39;s usually my go to when something doesn&#39;t feel right. I added the channels today trying to fix the performance problems, so it didn&#39;t even occur to me that I was doing those wrong.</p></pre>ctbel: <pre><p>At first I wanted to reference Effective Go or the actual language reference but nowhere is it explained as simple as in Tour of Go. Guess I figured that you might have skipped the tour entirely because I distinctly remember it being the place where I first learned about the select statement and default selector. I have come to realize that I might be biased though. Give or take 2.5 days out of the first 3 days of using Go were spent on examining the select statement and the impact of default selector. </p> <p>My first application was losing data when it was running on multiple threads. Since my own code was very basic I figured it must be caused by the (only) third party library I was using. It was a simple library and I immediately noticed the select/default. After revisiting the Tour of Go I was confident that my bug would be no more if I surgically removed &#34;default:&#34; from the source. I spent 2.5 days trying to figure out why that didn&#39;t fix it. Turns out it did fix it but my test setup kept reporting missing data because counter++ isn&#39;t thread safe. I relate to the trial-and-error approach.</p></pre>d_rudy: <pre><p>Oh wow! Yeah that did it! Thank you so much!</p></pre>koalainthetree: <pre><p>Is there somewhere I can read more about this? Or would you mind explaining why this is the case?</p></pre>d_rudy: <pre><p>Well, now that I&#39;ve just learned this, according to the docs:</p> <blockquote> <p>The <code>select</code>statement lets a goroutine wait on multiple communication operations.</p> <p>A <code>select</code> blocks until one of its cases can run, then it executes that case. It chooses one at random if multiple are ready.</p> </blockquote> <p>So basically, the <code>for</code> loop would have been sleeping while it was waiting for something to populate a channel, but instead, because of the <code>default</code> it just kept running through the loop over and over again.</p> <p>The difference on my machine was that it went from 99% CPU to 0.1% when I removed the <code>default</code>.</p></pre>postman_: <pre><p>From the <a href="https://golang.org/ref/spec" rel="nofollow">spec</a>:</p> <blockquote> <p>If one or more of the communications can proceed, a single one that can proceed is chosen via a uniform pseudo-random selection. Otherwise, if there is a default case, that case is chosen. If there is no default case, the &#34;select&#34; statement blocks until at least one of the communications can proceed.</p> </blockquote> <p>So, in the given code fragment the select never blocks because it has a default clause.</p> <p>Regarding the break:</p> <blockquote> <p>A &#34;break&#34; statement terminates execution of the innermost &#34;for&#34;, &#34;switch&#34;, or &#34;select&#34; statement within the same function. </p> </blockquote> <p>The break is essentially no-op in this case. So what you get is a tight &#34;for&#34; loop which does nothing (aside from using CPU for jumping over the code) if there is no data in channels.</p></pre>ecmdome: <pre><p>Also I wouldnt open a new gorm instance each time... Gorm utilizes go&#39;s database/sql package which handles connection pooling very well.</p></pre>d_rudy: <pre><p>So are you saying I should have one connection for the app?</p></pre>ecmdome: <pre><p>Yes one connection you can check if its alive using the .Ping() function which returns an error if the connection is dead.</p></pre>d_rudy: <pre><p>Okay, thanks!</p></pre>