<p>Hey everyone! I've been developing an <a href="https://i.gyazo.com/a535e0fa72cdbf4cc0e079e68e30e7c7.gif">action rpg web browser game</a> utilizing websockets within nodejs.</p>
<p>I didn't know I was going to get this far into my game. But basically I am at Monster AI and synchronization. The global game loop to synchronize players and send data through the pipe can become very CPU intensive. I've done some scale testing and I've gotten around 5,000 active games with around 2-3 active players in in each game attacking mobs and moving around. I've noticed I can only utilize 1 CPU core while doing this and clustering, IPC, and in-memory databases will not remedy my particular issue because I use object references within the buffer data. (users object has properties that reference back to their websocket connection). Nodejs clustering is pointless for me because it will basically run all instances of my game code on each core, no thanks. This will hurt performance and bottleneck my app. Plus, clustering is for high concurrency and I'm not really looking for that, just looking for something to utilize the 4 or 7 other <strong>cores</strong> I bought for my server.</p>
<p>I've found a nodejs addon that shares it memory across all forked processes, that works on Linux only, which is exactly what I need. But I don't dev on linux and would need to test it out extensively. So I'm at a bit of dilemma. I've heard GO supports multiple processes with goroutines? If so, I think converting my game code (about 40k LOC thus far) to GO might be my only option. (And a great one!)</p>
<p>I cannot stand the GO syntax though, I've been in JS land for way too long and maybe it's just something I need to get used to. But I am ready to get used to it because I know GO can scale huge, and that's what I'm looking for. </p>
<p>TLDR: Fed up with being bound to only 1 core within nodejs and want to switch to GO so my entire game app can utilize multiple cores. Is GO right for me?</p>
<hr/>**评论：**<br/><br/>Bromlife: <pre><p>I can't really give you an unbiased opinion, as I loathe Node.js and its event driven madness. But I can tell you that Go's use of CPU cores is leaps & bounds better, seeing as Node isn't multicore. Not to mention Go will be quicker than Node in many other ways, too.</p>
<p>The thing you really have to ask yourself, is this worth the cost of a rewrite + learning a new language & its idioms?</p>
<p>My experience is that it was definitely worth it, but those three months of not adding value to my business was <em>hard</em> - but now that I'm on the other side of it I'm very very happy I made the change. However, I like Go's clean syntax & idioms so YMMV.</p>
<p>Maybe you should just implement <a href="https://nodejs.org/api/cluster.html">clustering</a> into your project?</p></pre>WombatScared: <pre><blockquote>
<p>Maybe you should just implement clustering into your project?</p>
</blockquote>
<p>Good idea and I tried this but had negative results. My problem with clustering is it basically multiplies my memory usage by the amount of cores I have. It runs an entire app in separated v8 isolates which basically decreases performance in my scenario because I utilize nodejs` memory buffers a lot to store user data (inventory items, players min/max damage, health, etc) in memory. That memory gets multiplied and it ends up using about nth the amount of memory usage... but it does increase performance under extreme <strong>high concurrency</strong> but not by much and definitely not worth it in my case. And having all my memory multiplied across processes just screams horror and I couldn't live with myself. </p>
<p>Plus, I believe we can only send data to the clustered forks through IPC. Which literally JSON.stringifies anything that get's passed, so there is a TON of overheard there as well. And this means we cannot pass object references through IPC as well. I believe web-workers helps with this, but the data still needs to be sent through IPC and that just seems so silly to me that I need to send data across my own pipe (which wastes cpu power) just to utilize extra cores.</p>
<p>If someone can chime in on the clustering issue maybe I am missing something but I've tested it and it does what I explained. Maybe my app just isn't designed well for it.</p>
<p>Edit: The only thing that fixes this is the ability to share memory across multiple processes (that means my object references and memory can be utilized by any core or child forker I create). See <a href="http://blog.varunajayasiri.com/shared-memory-with-nodejs" rel="nofollow">here</a> for that, it's linux only and I dev on windows and haven't tested it at all, but it seems like exactly what I need. I would rather dev on windows inside GO though.</p></pre>Philodoxx: <pre><blockquote>
<p>And having all my memory multiplied across processes just screams horror and I couldn't live with myself.</p>
</blockquote>
<p>Why though? Does your server have enough ram to handle n node processes running? If so, it's not a big deal. Sure it might make you feel dirty deep down but if you're not memory bound starting up extra processes is a lot easier than a complete rewrite.</p></pre>synalx: <pre><p>Also, memory for executable code, such as node and its shared libraries, should be shared across all instances of the process.</p></pre>synalx: <pre><p>I think clustering is the right answer, if you really care about how well your game will scale. No matter how well you utilize multiple cores, eventually you will hit the limits of what you can do on a single machine, and you will have to shard/cluster/parallelize your code. Better to do that now than to figure out how to split up a larger codebase with deep assumptions about memory sharing.</p>
<p>Just treat your multiple cores as multiple machines, and design for that architecture. Yes, this means running multiple copies of your server on the same machine. No, that's not a bad thing. You shouldn't be storing the same data in memory in each process, you should be sharding your workload along some sensible boundary (it sounds like each game is independent? If so, you can just spin up new instances when you need to handle more games).</p></pre>Fwippy: <pre><p>You have 5000 games, that's 5000 logical pieces you could break the work into. Obviously you don't want 5000 copies of node running on one machine, but you should be able to chunk the data up so that, say, you have 5 processes each with 1000 games, and each should take about 20% of the memory.</p>
<p>The hard part of concurrency/parallelism is identifying the ways you can break up the work, and then cleanly separating it.</p></pre>lofties: <pre><p>If I read this correctly, you want concurrency, but you currently store the game state in memory that is only available to the process itself?</p>
<blockquote>
<p>in-memory databases will not remedy my particular issue because I use object references within the buffer data</p>
</blockquote>
<p>Go will not solve your issue. I would recommend you to consider refactoring your code in a way that an (in-memory) DB can be of service here.</p></pre>WombatScared: <pre><blockquote>
<p>If I read this correctly, you want concurrency, but you currently store the game state in memory that is only available to the process itself?</p>
</blockquote>
<p>Could this be node's design fault or mine? (Most likely mine), but just out of curiosity if you store data in the memory buffer (for example, let's say a basic c++ console application), would that application's memory be bound to only that process (or core?) Other cores should be able to use that applications memory data right? (I don't think this is possible within node because of the 'V8' isolates [separate instances]). (Without being hacky, for example, that memory add-on) </p>
<p>Edit: I don't mean to be rude by saying 'design fault', but maybe the design philosophy would be more appropriate.</p></pre>dazzford: <pre><p>In C++ you could access that other memory directly but you really shouldn't. It's chock full of hazards.</p>
<p>There are reasons RPC boundaries exist. A single in memory DB is the way to go; redis, mongo, etc.</p>
<p>It handles the isolation for you so you don't have to worry about those hazards.</p></pre>ptman: <pre><p>Just yesterday there was an article by someone who switched from node.js to go: <a href="http://www.philipotoole.com/400-days-of-go/">http://www.philipotoole.com/400-days-of-go/</a></p>
<p>I tried to look carefully at both node.js and Go before making the choice for my current project and went with Go. It has been working well so far. One think where node.js seems to be superior is parsing JSON, as Go uses a lot of reflection to achieve that. But as you point out, Go is much better at concurrency. Have you thought about moving the state/cache to some other database, like redis?</p></pre>egonelbre: <pre><blockquote>
<p>One think where node.js seems to be superior is parsing JSON...</p>
</blockquote>
<p>When you want performance, you probably shouldn't be using JSON in the first place. i.e. Cap'n Proto or FlatBuffers would be probably more appropriate.</p></pre>ptman: <pre><p>True. But I assume there's javascript in the browser. And it might take a while to switch to different serialization, if it even makes sense.</p></pre>WombatScared: <pre><blockquote>
<p>Go is much better at concurrency. Have you thought about moving the state/cache to some other database, like redis?</p>
</blockquote>
<p>Hey, thanks for that article that was a good read for me.</p>
<p>I think @lofties might be up to something, I think the design of my code is wrong... I'm going to take another look at an in-db memory storage and post up some questions at nodejs` github to maybe see if I'm truly fucked.</p>
<p>If I am, I am definitely going to switch to GO. Maybe I won't make dumb design decisions when re-writing everything in it. It will take a couple weeks though, I'm so slow at learning new syntax, but It will be worth it.</p></pre>dazzford: <pre><p>Honestly, you will make dumb decisions. We all do, and that's how we learn to write better code.</p>
<p>The fact you are trying, seeking advice, and considering others opinions means you are on the right track and going to do great.</p></pre>google_you: <pre><p>It looks like you're doing things wrong. Changing programming language wouldn't fix that.</p>
<p>Wrong parts:</p>
<blockquote>
<p>The global game loop to synchronize players and send data through the pipe can become very CPU intensive.</p>
</blockquote>
<p>This is usually network IO bound, not CPU.</p>
<blockquote>
<p>I use object references within the buffer data.</p>
</blockquote>
<p>Share-nothing(tm) is what you want if you want maintainable high throughput concurrency and parallelism.</p>
<blockquote>
<p>I cannot stand the GO syntax though, I've been in JS land for way too long and maybe it's just something I need to get used to.</p>
</blockquote>
<p>If you're nagging about syntax, good luck. Coding is just coding. I'd seriously look at your architecture and algorithms first.</p></pre>WatchDogx: <pre><p>Why not just balance the games between node processes/cores.<br/>
All the players would still be on the same process.<br/>
I mean go should work, so will c, c++, java, Scala, python, etc.
You could probably rethink your app architecture and get it to work on node too. </p></pre>Bromlife: <pre><p>OP wants concurrency, I think that out of the box Go offers better concurrency support than the languages mentioned.</p>
<p>Node doesn't support CPU core concurrency without clustering.</p></pre>timrichard: <pre><p>Hi,
Just another approach that might work for you...</p>
<p>You could keep your Node workers load balanced behind a tool like PM2 ( <a href="https://keymetrics.io/2015/03/26/pm2-clustering-made-easy/" rel="nofollow">https://keymetrics.io/2015/03/26/pm2-clustering-made-easy/</a> ), with one process per core. The data can be shared between workers using an in-memory key/value datastore like Memcached or Redis.
You can avoid Websocket Session Affinity/StickySession issues by outsourcing your Websocket handling to a service like Fanout.io [ <a href="https://fanout.io/" rel="nofollow">https://fanout.io/</a> ]. </p>
<p>Each Websocket connection between a player and your cluster of workers is a 'channel'. You can share the details of the active channels between your workers via your shared memory DB (Memcached or Redis). The players maintain their WebSocket connections to the outsourced WebSocket proxy (fanout). When you want to transmit to a player on a channel, one of your workers can use the fanout API to get fanout to send the message via the WebSocket connection they have open. When a player wants to send a message back, it can use standard http to talk to your load balancer, which passes on the message to any worker to deal with. I've prototyped a simple demo of this out via a NodeJS/Express backend and a few mobile clients using a frontend app I developed in Ionic. Seemed to work okay... </p>
<p>If you wanted to scale your worker pool out, you could go for something like AWS ElasticBeanstalk to act as a load balancer, with as many EC2 workers as you like. Elasticache would be your Memcached/Redis component in that scenario.</p>
<p>If you really want to get the maximum concurrency bang for the buck, it's also worth looking at Elixir (Erlang with Ruby-ish syntax) and the Phoenix framework.</p></pre>jerf: <pre><p>Don't use the Erlang VM for anything with a heavy compute load. If you game has trivial logic and is mostly routing communication, it's fine, but if you start in with serious AI, it'll rapidly become too slow.</p></pre>koffiezet: <pre><p>Reading the thread here, you already seem to have decided to rewrite the thing in Go, but as some people already suggested, I'm not sure that would solve all your problems. It seems like not knowing the limitations of the platform and not designing for such scale is what got you. With a redesign this could probably be done in Node too. I personally wouldn't do it, but that's just my preference.</p>
<p>You mentioning 5000 separate games seems like you could even easily do this with just running multiple Node.js instances on the same server, each fixed to a different core, and running a reverse proxy in front of it and some sort of tagging so the same client of a game always ends up on the same instance. But in the end, your application is not designed to scale. There are many ways to do that, and if your game does not need some global state shared between different games, it might be pretty easy.</p>
<p>Now some people propose using things like Redis - which can partially solve things for you, but be sure you know what you're getting into - Redis and other nosql/in-mem db's also have (serious) pitfalls. Redis for example will become tricky once you would have to go beyond one server, and on top of that, if for some reason your Redis server is restarted - you lose all data.</p>
<p>Another thing to mention is, if that's going to be your first application in Go, it will end up not being the best Go code ever written. You'll not be familiar with the language, it's best practices, it's do and dont's - and you'll make a lot of mistakes initially, which might end up in your core design and bite you in the ass afterwards - but you sure would learn a lot :)</p></pre>mc_hammerd: <pre><p>sounds like a good use case</p>
<p>unfortunately one of the things node does every time you call an array func like <code>map</code> is wrap the source array in an object {}, so its definitely slower than using a typed language.</p>
<p>40k lines of node is probably 60k lines of go... but porting goes pretty fast once u get the datatypes done.</p>
<p>the syntax is way more terse... one guy made <a href="https://github.com/DAddYE/igo" rel="nofollow">iGo</a> that has a coffeescript like syntax. you can try it but last it was updated to go 1.3, so i cant recommend it idk if it still works..</p>
<p>one thing you can do is profile the cpu usage of your node app... i like this quote the pareto principle, 80% of the bugs are gonna be in 20% of the code, (or... 80% of the bottlenecks are in 20% of the code)... i find its true and i find that its even better, sometimes 95% of the bottlenecks are in only 10 lines of code.</p>
<p>or set a task for each core, one for managing users, one for maps, one for chat, one for http, and 3-4 to the rooms (especially pathing i guess)... should help and not be that hard.</p>
<p>and last you could try compiling your game to ASMjs... it replaces all the V8 builtins with tighter ASM JS implementations (ie no {} objects)... its possible you can get 2-300% gains from that.</p></pre>egonelbre: <pre><p>Replace "GO" with "Go".</p>
<blockquote>
<p>users object has properties that reference back to their websocket connection</p>
</blockquote>
<p>I'm not clear on the why/how... but it sounds like a bad idea.</p>
<blockquote>
<p>I cannot stand the Go syntax though...</p>
</blockquote>
<p>You'll probably get used to it.</p>
<p>I think Go will be a better option than Node. Of course whether it's worth the cost to rewrite is for you to decide.</p>
<p>Regarding the memory usage. AFAIR if you use a memory mapped file it will be shared between processes, which can be used to prevent loading the data to memory multiple times (of course it's only efficient if it's read only).</p></pre>WombatScared: <pre><blockquote>
<p>I'm not clear on the why/how... but it sounds like a bad idea.</p>
</blockquote>
<p>Yeah, <a href="https://github.com/nodejs/node/issues/2874" rel="nofollow">just created a question</a> about that. Thanks for your concern! However, I am starting to lean closer and closer to Go :) I'm getting a bit frustrated with nodejs` single core bound / clustering thingabob.</p></pre>synalx: <pre><p>There's nothing really wrong with node's approach. Single threaded event driven servers are quite common in the UNIX world, and scaling <em>out</em> instead of <em>up</em> is industry best practice these days.</p></pre>Jamo008: <pre><p>Node developer for about 4 years, swapped to Go last year. Will never go back
