<p>Hi there, I'm a game designer/programmer who's working on a new game. I am the man responsible for the mess of spaghetti PHP 3 code that was Neopets in the early 2000s and want to do things the right way this time.</p>
<p>I'm creating a REST API using GoBuffalo and gocql (Cassandra) for my game. I've only been using Go for a week but really like it so far, but I think I've come upon an issue that's insurmountable.</p>
<p>My issue is that I have a huge amount of database tables which grow and change rapidly. Things like the Cooking system in the game could be represented by about 10 tables alone (it's a complicated game.)</p>
<p>The REST API that I'm making first is for our admin section. This would consist of about 300+ different pages for entering data into the game. I'm trying to make a generic endpoint so I don't have to create 300 different handlers duplicating the same code, so :</p>
<ul>
<li>/api/admin/model/monster/list</li>
<li>/api/admin/model/cooking_recipe/list</li>
<li>/api/admin/model/cooking_recipe/show/32</li>
<li>etc.</li>
</ul>
<p>this is the controller/handler code that I've written so far</p>
<pre><code>/*
Show - reads in a single row by ID and outputs the marshalled JSON
*/
func (ar AdminResource) Show(c buffalo.Context) error {
model := c.Param("model")
id := c.Param("id")
// Get database connection
dbm := c.Value("dbm").(*dbManager)
conn, err := dbm.GetConnection("cassandra")
if err != nil {
log.Fatal(err)
}
sess, err := conn.GetHandle()
if err != nil {
log.Fatal(err)
}
// Query
zone := new(models.Monster)
iter := sess.Query(`SELECT id, created, name FROM monster WHERE id = ? LIMIT 1`, id).Iter()
sm, err := iter.SliceMap()
if len(sm) != 1 {
errs := Error{123, fmt.Sprintf("Unable to read %s resource with ID %s", model, id)}
return c.Render(403, r.JSON(errs.toList()))
}
return c.Render(200, r.JSON(sm[0]))
}
</code></pre>
<p>This is just with the 'Monster' table though, and it works Every attempt I've had making it a generic interface has failed. I've tried :</p>
<ul>
<li><p>Factory : models.ModelFactory("Monster") - this returns the correct *Monster struct but the compiler thinks it's an interface{} so I can't do anything with it.</p></li>
<li><p>Reflection : Just can't get this to work. It seems I can return a rtype but there's no way to convert that into a normal type.</p></li>
<li><p>Type Assertion : models.ModelFactory("Monster").(*models.Monster) - this doesn't work because I need to assert the type as a variable string and I have the same issue as reflection above.</p></li>
<li><p>gocassa : This makes the problem even worse because you need a &struct to pass in. I went back to raw gocql after fighting with in for 2 days.</p></li>
</ul>
<p>I sort of understand why this may be the case. How could the compiler know what type a variable is if it's instantiated within some black box code it doesn't understand. I just wondered if there's any sneaky way I can get around it. This is really the big one, the rest of the API is simple.</p>
<p>So my only option seems to be to duplicate the above code<em>6</em>300+ :( I really don't want to do that.</p>
<p>Thanks! Any help would be appreciated. Hopefully the answer is not "wait for Go 2.0!"</p>
<p><strong>EDIT</strong> - Thanks for all the help. I'm going to do whatever it takes to stick with Go for now. All your replies gave me some ideas!</p>
<hr/>**评论：**<br/><br/>Redundancy_: <pre><p>It looks like you're putting what I call "static data" into a database - by static data I mean stuff that's frequently authored by game designers and doesn't typically change in a production run of a game server.</p>
<p>This dodges the question of "how do I do this in Go?" a bit, but I'd recommend taking a step back and asking yourself if you really want to do it like that at all - how are you versioning data changes and promoting them, how can you sync back to an old version? Is a RESTful API efficient enough for data you're likely to to want to bulk load? Sometimes the desire to store data in a database comes from a place of just being the default assumption.</p>
<p>I'd recommend just reading through <a href="https://medium.com/@offbyone/file-static-data-e2af8f8e8c0a">https://medium.com/@offbyone/file-static-data-e2af8f8e8c0a</a> as another point of view.</p>
<p>Like I said though, this doesn't solve the question you asked though - it sounds a lot like you want an orm, so maybe <a href="https://github.com/scylladb/gocqlx">https://github.com/scylladb/gocqlx</a> might be something to look at?</p></pre>borovan: <pre><p>Yup, sounds about right, it's a huge amount of static data. I want to do it this way because this entire thing I'm building is somewhat of a prototype. At some point I'm going to hand it off to a team that can build code that's going to stand up in a production environment.</p>
<p>Things like versioning changes is somewhat out of the scope of what I'm doing but I'd definitely consider adding it in if I could. Creating this system is more of an aid to game design than something that's going to be in a live game, but the closer I can get the less work we have to do later.</p></pre>Redundancy_: <pre><p>So for a prototype, why bother? just do a pass-through file system to json files in source control. </p></pre>borovan: <pre><p>Well the closer I can get to something that's useable in production the better, plus I'm learning a lot as I work on this project.</p>
<p>Also there's somebody working on a front-end that's going to allow me to input data soon.</p></pre>Redundancy_: <pre><p>I think you'll find yourself stuck in the assumption of a database more quickly than you might think. </p>
<p>It's a very common cycle to do a prototype on the assumption that it'll be replaced by something better, then people just keep adding to it because it's there and it's less work than doing something better, and before long that editing UI is a big project that relies on it and you're stuck (excepting a large effort to move away).</p>
<p>I don't think you need to use Go for this, but you can also move away from using a database earlier rather than later and use something like Python or Node as a standalone interface to the data editing on disk. Separating your tools and your engine can be a very good architectural firewall, even though it can also cause some pain. </p>
<p>I think that you would have hit issues with cassandra anyway, because it's designed to enable a very specific high performance way of accessing data and writing it by giving you some quite limited access patterns that frequently require you to duplicate data (hence things like materialized views) - it's probably a pretty terrible fit for an editor, imo. </p>
<p>$0.02 Do yourself a long term favour and ensure that if you do have an API to your data, you ensure you're not just mirroring the constraints of the underlying data model.</p></pre>kaldonis: <pre><p>I'd argue that the value of just getting a prototype done faster outweighs the benefit of having production ready code. Having a working prototype could uncover issues that lead you to completely reconsider your data model, etc. </p></pre>mcandre: <pre><p>As a quick hack, you can store the static data as a single binary blob in the database, and later think about providing this information separately from the database.</p></pre>icholy: <pre><p>That's fine, choose the right tool for the job. Sounds like a good use-case for an ORM which Go isn't the best at. Code gen might be something to look into if you decide to stick with it.</p></pre>birkbork: <pre><p>You need to use an orm as other has suggested. You will have the same troubles with any programming language and a massive database. So go learn to use an ORM, database migrations and such.</p>
<p>For golang, Gorm is a good one!
Myself, I picked up using an ORM while in PHP and kept doing it because it's just superior to raw sql.</p>
<p>I'm no good at python, but if you are most at home there maybe spend some days in python learning to use an ORM, then come back to go?</p>
<p>Here's a decent article showing you how to work with gorm: <a href="https://medium.com/@cgrant/developing-a-simple-crud-api-with-go-gin-and-gorm-df87d98e6ed1">https://medium.com/@cgrant/developing-a-simple-crud-api-with-go-gin-and-gorm-df87d98e6ed1</a></p>
<p>For static game driven data, it does not belong in a database.</p>
<p>Instead put it in text files or such.</p>
<p>Here's an example how I use yaml data to drive my rpg game written in go:</p>
<p>package game</p>
<pre><code> import (
"fmt"
"log"
"github.com/ghodss/yaml"
)
// Item ...
type Item struct {
baseName string // base bame = "apple juice". object.Name = "fresh apple juice" (current name)
object
Effect *Effect
Qualities []ItemQuality
}
// ItemQuality ...
type ItemQuality struct {
Name string
Effect *Effect
Condition uint64
}
type itemDefinition struct {
Name string
Kind string
Weight float64 // in grams
Energy uint
Sprite string
Tiles int // number of tiles covered by this item, defaults to 1
Qualities []itemQualityDef
}
type itemQualityDef struct {
Name string
Effect string
Condition uint64
}
var (
itemDefs = []itemDefinition{}
)
func init() {
data := DataFile("data/items.yml")
if err := yaml.Unmarshal(data, &itemDefs); err != nil {
panic(err)
}
}
func getItemDefinition(name string) (*itemDefinition, error) {
for _, def := range itemDefs {
if name == def.Name {
return &def, nil
}
}
return nil, fmt.Errorf("Item name dont exist: " + name)
}
// NewItem creates a item from items.yml definition
func (w *World) NewItem(name string) *Item {
def, err := getItemDefinition(name)
if err != nil {
panic(err)
}
item := &Item{
object: object{
GID: w.spawnCount.Inc(),
Name: def.Name,
Kind: def.Kind,
Weight: def.Weight,
Tiles: def.Tiles,
},
baseName: def.Name,
}
for _, qual := range def.Qualities {
item.Qualities = append(item.Qualities, ItemQuality{
Name: qual.Name,
Effect: NewEffect(qual.Effect),
Condition: qual.Condition})
}
if item.Tiles == 0 {
item.Tiles = 1
}
item.Sprite = def.Kind + "/" + def.Sprite
item.updateQuality()
return item
}
</code></pre>
<p>And the items.yml looks like this:</p>
<pre><code> - name : tiny stone
kind : STONE
sprite: stone_tiny
weight: 2500
- name : small stone
kind : STONE
sprite: stone_small
weight: 5000
</code></pre></pre>borovan: <pre><p>Ok thanks! I really do need the data in a database though... think World of Warcraft type complexity... well probably even more complex. I couldn't track it all otherwise.</p></pre>classhero: <pre><blockquote>
<p>think World of Warcraft type complexity... well probably even more complex.</p>
</blockquote>
<p>Your game might be complex, but that doesn't mean your data model has to be :). In fact, I could make the argument that software engineering involves a lot of making a simple (maintainable, operable) backend act as a lever for very advanced features that <em>seem</em> to the customer like they'd need an advanced backend. You want to avoid complication here, not strive for it.</p>
<p>Some of your posts speak to wanting this to be as close to production-ready as possible. It's worth thinking about what it will be like to operate your system, as well as develop it. You pay for every dependency (including the database) one way or another - do you really want to be in charge for tuning, optimizing, and responding to failures for 600+ database tables? </p>
<p>Try approaching the problem from a different angle. Start with "What's the absolute cheapest way to get the end-user experience I want", and then critically evaluate if the cheapest solution is "good enough" or if you need something more. A lot of people have suggested storing this data statically (which I'd +1 as well); why won't this work for you? For example, one counterpoint is needing to do some sort of deployment whenever static data changes (this doesn't imply downtime, btw, but you will need to "move bits to the boxes" as it were); but is that actually a big enough problem you'd toss the solution in favor of a database? </p></pre>borovan: <pre><p>I'm not striving for complexity I just want to have no ceiling on how complicated the game can become.</p>
<p>Storing the data statically vs. a database is really the same to me, just the latter is way more flexible. Plus since it's not SQL and you don't have foreign keys I'm going to run loads of scripts and tests on the data just to make sure the game design still makes sense.</p>
<p>You can always just have a caching layer in front of the database to give it the same performance characteristics as static data anyway.</p></pre>birkbork: <pre><p>Feel free to implement it as you please, but saying you must have it in a database is not correct at all and a bad starting point for getting any kind of ok performance out of a larger application.
If you notice, the code I posted is read on application start as it is not changing. (The "func init()" is special), and thus always live in memory. Compare this to reading items/other static stuff from database every time it is needed, you will quickly realize the overhead is enormous.</p>
<p>My game is a "mmo" / "world builder" cross-over, this design works perfectly well.</p>
<p>Either way, having this data in a text file is meaningful even if you insist on adding it in the database, since you easily version the text file along with your source code, and would then seed the database with the file content. (Personally I keep an "instance" of an item in a persistent database, but this is acctually item definitions. They are fully static).</p>
<p>I'm just trying to suggest ways for you to allow for better scalability and maintainability as your project grows. Maybe you need to try it your way and run up against the wall before you understand you need other ways. (That's what I had to do to come to this conclusion, lol).</p>
<p>Feel free to pm me if you want to discuss randomly or see more code snippets, I'm bored and glad to help! </p></pre>borovan: <pre><p>Hah mmo/world builder, mine too :)</p>
<p>Ok will have a play around, thanks for the advice. It won't be going live for years so I'm not too bothered about scalability at this point.</p></pre>birkbork: <pre><p>I wish you all the luck! Please don't hesitate to ask about anything</p></pre>hell_0n_wheel: <pre><blockquote>
<p>It won't be going live for years so I'm not too bothered about scalability at this point.</p>
</blockquote>
<p>Scalability isn't something that you can fix at the end of a project. It's something you have to design for at the outset. Especially when using a database... a bad schema can reduce even the most efficient application to a crawl (lock contention). You said in another post that you want to write something prod-ready... I don't see it happening here.</p></pre>birkbork: <pre><p>EDIT: updated my reply below</p></pre>seweso: <pre><p>This might not be the answer you were looking for, but maybe you have way too many tables? Furthermore, you might be doing OO wrong. As you don't need to map all types to specific structs in your language. ( Just like composition is more power-full than inheritance ). </p>
<p>So you might want type to be a property instead of using reflection or type assertions. And you want to put different type of objects in one table, with a column indicating the type. </p>
<p>Just my two cents. </p></pre>borovan: <pre><p>I don't really know another way of doing it. These tables are fully normalised and all have different properties. Cassandra doesn't work best with fully normalised tables but I'm going to cross that bridge when I come to it.</p>
<p>300 tables is just the definition of the objects within the game world, I haven't got to the live data yet.</p></pre>seweso: <pre><blockquote>
<p>I don't really know another way of doing it</p>
</blockquote>
<p>The other way would be what I said, make type a field/property. That way you can also query on it, and you can get all objects in one query, instead of having to query per type. And even then you can later map the 'object has a type' to 'object is a specific struct' later, if that makes the domain logic easier to implement. </p>
<p>If you need to query 'what objects are in this [room/bag/whatever]' then you really want one table with objects of multiple types. </p>
<p>If tables share a lot of fields (they are mostly the same), then you also want those objects to be in one table. It makes everything so much easier and faster. </p>
<p>I mean, if you have a Cat database, would you threat the specific cat breed as a column or would you create a table for every breed? If you EVER have multiple tables with identical field definitions, you need to scratch your head and think about what you are doing.</p>
<p>:)</p></pre>borovan: <pre><p>Ok cool, will think about it. The tables do have different columns and are fully normalised but I'm probably still thinking in the 2000s mySQL mindset!</p></pre>seweso: <pre><p>And if you don't need to query/index certain fields you can store data in Json for instance. That's a more document based design. No need to go with No-SQL entirely. </p>
<p>The thing which needs to click in your brain is that you are automating the automation process itself. Any project with lots of meta data, lots of types requires you to do that or else you need to write a lot of boiler plat code. </p>
<p>The hard part is finding balance between readable/maintainable code and a fast/small codebase. These things are at odds with each other. </p>
<p>If readability is important (vs speed), normalising is still the way to go and reflection is a good way to deal with type info. </p></pre>neotecha: <pre><blockquote>
<p>No need to go with No-SQL entirely</p>
</blockquote>
<p>One of No-SQL's strengths is rapid prototyping, and sounds right up the direction that <a href="/u/borovan" rel="nofollow">/u/borovan</a> is trying to go :-)</p></pre>borovan: <pre><p>Also the ability to save a 100K blob of player data millions of times a second. Can't do that with mySQL.</p></pre>bvierra: <pre><p>This is actually a misconception, you can do this with mysql... however if you are saving player data as a blob without incrementals, well you are doing it wrong. However mysql can handle millions of writes per second when done correctly.</p></pre>Asdayasman: <pre><p>Take a look at the way <a href="https://github.com/django-polymorphic/django-polymorphic" rel="nofollow">django-polymorphic</a> stores its data for a more flexible idea.</p></pre>slashclee: <pre><p>I’m not super familiar with Cassandra, but in SQL you could just create a <code>properties</code> table and store all of the key/value pairs there instead of hardcoding them into the tables for your 300+ object types.</p></pre>kardianos: <pre><p>I have a small line-of-business application written in Go with just over 100 tables.</p>
<p>Key for any app + database:</p>
<p>Metadata, Metadata, Metadata.</p>
<p>Let's make this super simple for this post (you can do fancier things too.
You can start with a map that maps API calls to requests such as:</p>
<p>map[string]struct{SQL string, vars []string}{"monster": {"select * from monster where ID = {{.ID|int}};, []string{"ID"}}</p>
<p>Then you just define a single endpoint http endpoint starting with"/api/admin", strip that prefix, and match from the map. When executing it the above syntax is from "text/template" with special functions designed to escape and validate data types. You could also do something more fancy to allow you to pass the values into the SQL as parameters (which is what I do).</p>
<p>But really, you only need / want a single HTTP endpoint, then define metadata, either from the database schema itself (which can get a bit fancy) or do something dirt simple and define the SQL only.</p></pre>elingeniero: <pre><p>I think you're missing a <code>"</code> in your code example. </p>
<p>I think the problem the op is having is not the query generation, it's the functions that handle the objects from the queries - i.e. you have a <code>func getMonsterA(db *sql.DB, id int) *models.MonsterA, error { ... }</code> and similarly a <code>func getMonsterB</code> etc. as well as all the functions that deal in similar ways with each.</p></pre>maus80: <pre><p>Relevant discussion on SO: <a href="https://stackoverflow.com/questions/6390585/in-golang-is-it-possible-to-get-reflect-type-from-the-type-itself-from-name-as" rel="nofollow">https://stackoverflow.com/questions/6390585/in-golang-is-it-possible-to-get-reflect-type-from-the-type-itself-from-name-as</a></p></pre>borovan: <pre><p>Yeah... fine, I don't think what I'm trying to do is possible. I'll make my own Go... with blackjack and hookers!</p></pre>elingeniero: <pre><p>The only solution that comes to my mind is code generation. I've never had to use it myself so I can't give you any tips about it but I know at least 1 company that does code gen in production so it's not totally untested. </p>
<p>I appreciate you don't want to make your tables less pure like another commenter suggested, and I agree with you that you should keep to best database practices.</p>
<p>Personally, I do use Python (Django specifically) for services that do large amounts of simple database work, as I think this is still a use case where Python is better than Go for this exact reason that dealing with lots of database objects is very tedious in Go.</p>
<p>Generally my projects have a central Django-based core that handles all the data storage, and all processing tasks are microservices written in Go (or Rust/JavaScript/Elixir) which I communicate with using gRPC.</p>
<p>So my conclusion is that you're not insane for feeling like you want to use Python here - Go isn't universally better and I think this is a situation where it definitely isn't the best choice.</p></pre>borovan: <pre><p>Thanks for the reply! I'd rather not go for the code generation route if I can help it, just seems a messy workaround for something that needs probably a few lines of Python (and definitely one line of PHP but let's not go there)</p>
<p>As I've never used Python I'm thinking of starting with a barebones framework like Falcon and then learning it from there. I find it easier to learn in a small framework and then when I get frustrated that X doesn't exist, look for solutions that provide it.</p>
<p>This project doesn't have to be perfect, it just needs to let me iterate the game design rapidly and have data that my Unity programmer can access. It's only the two of us right now but a few years out it'll be a large team.</p></pre>elingeniero: <pre><p>Ok, Python is my first true programming love so obviously I think you should use it. </p>
<p>I've not tried falcon but it looks cool. My experience of building several products in flask is that even though you start off with good intentions of keeping it minimal, invariably the projects end up using an orm, user management, an admin interface, etc... At that point you've just cobbled together a less integrated and less well documented version of Django so I now just start all projects in Django. It's not even particularly harder to get a new project set up with, and comes with everything you need (properly tested, integrated, and documented too).</p>
<p>The point of the above is that if you are considering learning a Python framework, I would absolutely recommend going full Django straight away.</p>
<p>On the other hand, I don't think Django is particular easy for programmers new to Python so, the choice is yours.</p></pre>asdonair: <pre><p>Python is OK for REST APIs of this kind. If you feel possible to look non-REST API, take a look at Twirp (<a href="https://twitchtv.github.io/twirp/docs/routing.html" rel="nofollow">https://twitchtv.github.io/twirp/docs/routing.html</a>). </p></pre>hockeyhippie: <pre><p>I've written code that does exactly what you describe by defining an interface type that all my table structures implement. The REST handlers then just work against the interface. </p></pre>borovan: <pre><p>So i guess the issue with that is you wouldn't get an instance of the struct that is a *Monster, you'd have a ModelInterface instead. Let me play around with that.</p></pre>hockeyhippie: <pre><p>I can't share the exact code but I'll write up some notes on the approach when I get time. </p></pre>borovan: <pre><p>Awesome thanks. Yeah I hit another roadblock. I want to use the struct *Zone but if it's a generic interface then I don't have access to all the hints and fields. I'm going to give it a few more tries though.</p></pre>hockeyhippie: <pre><p>There's a trick to that, it has to do with putting your database objects in a library (package) and having a factory method for the library to return an object for whatever table you want. Each db object that implements the interface is in a file in that library. The file has an init() method that registers the object to the library and this adds the name of the object + it's field metadata to a lookup table in the library. Go calls each init() method at startup so all the objects are registered.</p>
<pre><code>type MyDbObject interface {
GetTableName() string
GetKeyField() string
New() MyDbObject
}
</code></pre></pre>troyk: <pre><p>Exactly this, wish it could be sent to the top. The signal-noise ratio for this topic shows how many reach for ORM's and libs to solve, what should be a simple problem, if you didn't look to ORM type solutions first.</p>
<p>Either A:
Construct the sql from the params and return the result of the query without abstracting to a model first</p>
<p>or B:
Use a common interface to your model, like the one above, that satisfies the needs of the code in the single handler.</p></pre>very-little-gravitas: <pre><p>The trick is to get each of your types to instanciate themselves, so have a function monster.Find() or monster.Create() that queries the dB and returns a <strong>Monster or []</strong>Monster. Don’t use interface{}. </p>
<p>If you have too many types, maybe you don’t need that many (for example all monsters can be the same go type with different attributes). 300 sounds excessive. </p>
<p>Be careful not to try to directly translate PHP to Go as it will just be frustrating.</p></pre>Justinsaccount: <pre><blockquote>
<p>this returns the correct *Monster struct but the compiler thinks it's an interface{} so I can't do anything with it.</p>
</blockquote>
<p>What would you do with it? Are you always just serializing it and returning it to the response as json?</p>
<p>If you just make every object have a common interface like MarshalJSON, then it doesn't matter what type you have.</p></pre>borovan: <pre><p>I'd like to validate the fields based on annotated/hinted methods. Basically, read the struct metadata and do a load of validation, filtering and logging.</p></pre>badtuple: <pre><p>Sounds like you might be able to define an interface there so that you don't have to handle that logic outside the struct? That way you can handle internal mutation/validation/model-specific-stuff within the struct. Everything outside of the struct would just be treated as a Monster or other generic form. If you run into a rare case where you <em>need</em> to get them back to their non-interface form (which never seems to happen for me in APIs?) just use a switch and assert.</p>
<pre><code>type Monster interface {
Validate() error
Filter()
GetType() string // Literally just returns type as a string
}
</code></pre>
<p>Skimmed the thread so I might have missed it, but surprised this wasn't the top solution.</p></pre>caique_cp: <pre><p>Oh man, I loved neopets...</p></pre>borovan: <pre><p>:P</p></pre>jmatosp: <pre><p>It's not a technology limit, you need to go back to basics and read a bit about software design.</p>
<p>The fact that you are interfacing an API to tables means that something is wrong, software provides more than just raw data storage.</p>
<p>If you only need to provide access to tables, there are plenty of tools that have solved that problem.</p>
<p>On a superficial look (OOP perspective):</p>
<ul>
<li>Models are much more then data struct, they can/(prob)should have behaviour</li>
<li>Cassandra, Factories, Reflection are just tools to make you software work, you should be able to design/solve your software with basic language constructs</li>
<li>If you consider tables as your problem, then you should represent Table struct and work from there</li>
</ul></pre>borovan: <pre><p>Sorry, a bit more context. I just want the API to reference tables directly as an aid to game design. Instead of making a GDD I prefer to map out the game as a data model, throw test data in there and the play with it on the Unity client. Systems just seem to make more sense to me that way.</p>
<p>Initially it needs to be quite flexible and fluid, so I'd create definitions like CookingRecipe CookingRecipeItem etc., create the tables, insert a few rows and then send that data to Unity. As the game design solidifies then it the models would take more shape Cooking->addRecipe() for example, providing a layer between the API and the raw database tables.</p>
<p>This is a small project with just two people but eventually we're going to hire up to 20-30 people. Doesn't have to be perfect for now, but production-ready stuff can come later. Thankfully we don't have much of a deadline so we can move slow and break stuff :)</p></pre>jmatosp: <pre><p>the models aren't a layer between the API and the database, the models ARE the software.</p>
<p>The database is just a storage engine, chose any, even simple files would work for a prototype.
The API is an interface for your game "behaviour", chose one or many, REST, RPC, ...</p>
<p>if it is a proxy (REST API -> DATA) to a database then just code generic without any references to your "models", they are just tables with items, so represent them like that</p></pre>phlatphrog: <pre><p>I haven't used GoBuffalo, but I'm gonna assume it wants to do the typical ORM approach of mapping structs to tables, because that's what most programmers want, a simple representation of their tables in native (& simple) constructs within the working language (go in this case).</p>
<p>But don't forget that that's all written on top of more basic stuff that just does SQL underneath.</p>
<p>You can write your own abstraction layer, and completely ignore the ORM approach. If want you want is "generic" REST interfaces to your tables, rather than making structs for everything, take a metadata approach, where you have definitions of tables with columns, and information about what type of data is stored in the columns. You can then write a single REST service point which accepts an "object" (table) name and list of "attributes" (columns), and automatically translate that into a query, retrieve results, create a JSON response and return it, all based on your metadata, and not a matching struct anywhere.</p>
<p>At that point, rather than having 300 various structs that mostly/hopefully align with your tables, and a zillion methods dealing with each kind of struct, you can have a single handler that will serve data from any table, as long as you provide a metadata definition. (So, you'll have a set of metadata somewhere that describes your 300 tables.)</p>
<p>If you don't want a single REST point, you can use the metadata to automatically register your 300 different service points by writing functions that return functions (with closures to capture the metadata per service point).</p>
<p>If you really wanna get crazy, you can use the metadata from the database itself. Every database has a way to query "what tables exist?", "what columns in this table? and their types?" You can take a half-way approach, where you hard code the list of tables to be handled and then get the metadata from the db. You can write a program that queries the database, generates your metadata, which you can then edit/tweak. Whatever.</p>
<p>One of the things that I <em>really like</em> about go is how easy it is to handle functions as first class objects. With this, you can create hooks in your metadata framework for "business logic". For <em>this</em> set of objects/tables, do func A before serving a response, for <em>that</em> set, do func B to canonicalize attribute X, or just about anything you can dream up. (I'm assuming you'd probably pass in data rows as <code>map[string]interface{}</code> or something.)</p>
<p>Bottom line(s):</p>
<ol>
<li>don't be constrained by a certain framework. If it's not working, use something else, or even cook up your own.</li>
<li>just because go doesn't have "generics" doesn't mean you cannot write amazingly smart/flexible software with it.</li>
</ol>
<p>Here are a couple functions that I used in a program I wrote that just takes whatever query/raw sql, and processes all values returned. This are very quick & dirty, but demonstrate that you can do just about anything you want. If I passed in a list of types, I could scan to particular types, rather than to all strings (or all interface{}).</p>
<pre><code>func ScanToStrings(row *sql.Rows, numColumns int) []sql.NullString {
strings := make([]sql.NullString, numColumns, numColumns)
scanArgs := make([]interface{}, numColumns, numColumns)
for i := 0; i < numColumns; i++ {
scanArgs[i] = &strings[i]
}
err := row.Scan(scanArgs...)
MustNot("scanning to strings", err)
return strings
}
func ScanAllValues(rows *sql.Rows, nCols int) []interface{} {
vals := make([]interface{}, nCols, nCols)
args := make([]interface{}, nCols, nCols)
for i := 0; i < nCols; i++ {
args[i] = &vals[i]
}
err := rows.Scan(args...)
MustNot("scanning all values", err)
return vals
}
</code></pre></pre>hockeyhippie: <pre><p>I found this package very helpful for things like that: <a href="http://jmoiron.github.io/sqlx/" rel="nofollow">http://jmoiron.github.io/sqlx/</a></p></pre>borovan: <pre><p>Awesome thanks for the reply. I guess that if I'm just dealing with an array I lose lots of functionality that could be provided by the struct and the tags (json, db, etc.) What I'd love to have is something like :</p>
<pre><code>model := Factory.("zone")
model.LoadByID(id)
model.Set("name", "peter")
model.Save()
</code></pre>
<p>or something like that. Even if that's not the final code that's used in production it would help so much in the short term. Yes it could be done with just interface{} slices/arrays but I'm not sure that's the best approach.</p></pre>phlatphrog: <pre><p>Because I'm a crazy nut, and I'll take any excuse to write a bunch of go code, I created a fuller example here: <a href="https://github.com/pdk/genquery" rel="nofollow">https://github.com/pdk/genquery</a></p>
<p>This has example implementation of <code>GetString(colName string)</code>, <code>GetNumeric(colName string)</code>, etc. and a metadata driven scanner that figures out the right type variables to scan based on metadata types (<code>ScanRow</code> in datacontainer.go).</p>
<p>(What I was doing with the code from my previous example didn't require dealing with handling different types. Just getting everything as strings was sufficient.)</p>
<p>The point being that you can create whatever framework you want, at the level of abstraction that most closely matches what you want to accomplish. If you really want to provide a base with a bunch of structs so that other programmers can easily build on top of specific types, you can use an ORM approach. If you just want to build a way to (almost generically) get/retrieve data in a database via REST calls, then a bunch of structs is just noise.</p>
<p>Getting back to the "thinking of switching to python" issue: What I'm trying to show is that you can build stuff that is just as flexible as stuff that people might think python is better at, but the biggest difference is that go is a lot more forceful about encouraging the programmer to check for errors and handle types correctly. In python it's easy to write code that <em>mostly works</em>. What I've done is a very quick and dirty example, and I've glossed over a lot of error handling, but some might still complain about it being too verbose:</p>
<pre><code> nullIsGood, err := data.GetBool("is_good")
FatalIfErr("retrieve is_good", err)
isGood, err := nullIsGood.Value()
FatalIfErr("retrieve is_good value", err)
log.Printf("column is_good has value %t", isGood)
</code></pre>
<p>In python there will be the same failure conditions: is the column actually in the result? is it null? is it the data type I think it is? A difference is that in python the tendency is to just code as if everything will be just fine, and that your assumptions are valid.</p>
<p>And then, of course, on a completely different trajectory, there's the generators for everything path. :) There are tools floating around that will look at your data base and generate go structs. That can provide a basis for just generating structs for all your 300 tables, and all the associated code you want for every type.</p></pre>caseynashvegas: <pre><p>I'd look into sqlboiler. It's an orm/model generator that is quite idiomatic go. I'd link to the project but I'm on mobile at the moment. I've used it on a couple projects and never had an issue. To me, it's the only sane way to handle a relational DB in go. </p>
<p>One thing I like about it is it is DB first as opposed to code first, which allows you to use DB modeling tools. </p>
<p>That being said, maybe you don't need a relational DB for a game engine, maybe a document store? </p></pre>shovelpost: <pre><p>Please also post your question on <a href="https://groups.google.com/forum/#!forum/golang-nuts" rel="nofollow">https://groups.google.com/forum/#!forum/golang-nuts</a></p></pre>borovan: <pre><p>I'm shy</p></pre>shovelpost: <pre><p>That's understandable but judging from the comments of the thread the problem is tough. Posting in golang-nuts does not guarantee you will get a good answer but a lot of distinguished people read the mailing list so there's a chance you'll get some insights that you won't find here.</p></pre>borovan: <pre><p>Ok cool. Yeah things have changed so much since I posted this so not exactly relevant any more, but will definitely post a follow up. Thinking Go is a bit of pain but I'd love to make it work!</p></pre>shovelpost: <pre><blockquote>
<p>but will definitely post a follow up.</p>
</blockquote>
<p>Please do. I'd love to read the follow up or how you made it work.</p></pre>burnaftertweeting: <pre><p>Use interfaces to create a 'generic' type. You can use the interface to fetch a specific type and fill in relevant data from your db. You'll need to identify the type of resource when the record is saved to the db, so that you can switch on the resource type later. See <a href="https://play.golang.org/p/bv652AOhouT" rel="nofollow">https://play.golang.org/p/bv652AOhouT</a> for an example.</p>
<p>P.S. I played a lot of neopets growing up, its one of the reasons I got into web development :)</p></pre>borovan: <pre><p>Well that was probably my wife's fault... with her HTML guide!</p>
<p>Ok, I think my issue is that I was trying to access fields of a generic interface{} type. So for instance, if my func ModelFactory(string name) *Model {} returned a *Zone, then I'd want to get it's ID, but I couldn't.</p>
<p>Appreciate the reply - think I'm on the right track now.</p></pre>burnaftertweeting: <pre><p>In part it was! But the site itself led to me wondering how data got saved in between pages. I tried to rebuild the site with just HTML but that didn't go so well xD</p>
<p>I had similar issues with the generic interface{} for months. I tried using reflection to get around it too...and so did all my coworkers. Seems to be a recurring theme with interfaces.</p>
<p>Anyways, glad you got it figured out!</p></pre>user3961: <pre><p>My experiences using gRPC have been promising. If you write your “structs” in .proto files, you can end up using different languages for different parts of the game. It’s supposed to be faster than http and more rigid than arbitrary json. It’s more work up front, but for large scale projects (I’m hoping) it will pay off in performance, maintenance, and scalability. </p></pre>oscooter: <pre><p>I think you've had some good recommendations for ORMs etc, so I won't touch on that, but I do want to touch on your database choice.</p>
<p>I'd strongly recommend rethinking your usage of Cassandra if you are dealing with relational data, which it sounds like you are since you've mentioned you're normalizing all your data. You will wind up hating your life going down that road.</p>
<p>You'll be saving yourself a lot of headaches in the future if you take a step back and really analyze what Cassandra is good at and think about if your use case really fits the bill.</p>
<p>In my job we were using Cassandra to deal with a mass number of streaming, non-relational, geo-spatial temporal data. Cassandra was the right tool for this use case. It's write throughput was good for the streaming nature of the data and while we were duplicating data we were able to setup good indexes to support the queries we needed to do. However, this lead some developers to say "well we're using Cassandra as our database, so let's put all our data there" and we wound up using it as our main data store, even for our relational data.</p>
<p>I urge you, <em>do not</em> do this. It was painful for us and we wound up migrating all of our relational data to PostgreSQL. If you do stick with Cassandra, at least format your tables in a Cassandra way.</p>
<p>Example if you have the <code>Monster</code> type that can have many <code>Items</code> on it's body, instead of having a mapping table that maps <code>MonsterID</code>s to <code>ItemID</code>s as you would in a normalized database, make a <a href="https://docs.datastax.com/en/cql/3.1/cql/cql_using/cqlUseUDT.html" rel="nofollow">user-defined type</a> for your <code>Item</code> and have the <code>Monster</code> column family have an array of <code>Item</code>s. </p>
<p>Also look into <a href="https://blogs.infosupport.com/static-columns-in-cassandra-and-their-benefits/" rel="nofollow">static columns</a>. They help in denormalizing one-to-many relationships.</p>
<p>Again, I urge you not to normalize your data if you use Cassandra. Eventually you will hit a point where you need/want to do something with your relational data that you cannot because you are fighting your tools. Materialized views help, but won't cure all your problems.</p></pre>borovan: <pre><p>Thanks, I sort of get it, have done a lot of research about Cassandra. I know it's not the best for relational data but I didn't want to have to manage two database systems. Plus I just wanted to learn the API and see if it could actually do what I wanted it to.</p>
<p>I'm 100% sure that we need a NoSQL solution for persisting live player data but the game design really lends itself to SQL. The concept of having an ID in one DBMS that referred to a foreign ID in another kept me up at night... but perhaps that's the way to go.</p>
<p>I thought it was OK because we'd just store the normalised data (static game definitions) in Cassandra but never really have to query it much other than sending JSON blobs to the client.</p>
<p>Anyway going to GDC tomorrow so will have a lot of people to bug about these questions! :)</p></pre>Pancakepalpatine: <pre><p>Can you help me get my old login?? My brother hanged the pwd, but I had ~500 baby Neopets paintbrushes. xD</p>
<p>In order to actually contribute to the conversation, I've had great experiences with Gorm for this type of problem. Go doesn't play nicely with ORMs like I was used to with Ruby or Python, but they're still better than writing your own solution.</p></pre>borovan: <pre><p>I stopped doing that loooonnng ago sorry! Baby PBs though, nice! Assume you got them fair and square? :)</p></pre>Pancakepalpatine: <pre><p>:)</p></pre>Improvotter: <pre><p>I agree with almost all people here, but just want to add that you should look at some better ways of implementing REST APIs using HTTP. Don’t add ‘/list’ or ‘/show’ to your URLs. This is just a nitpick. But I overall agree that you should look at some more software design and take a step back.</p></pre>egonelbre: <pre><p>One approach with these kinds of problems helps is working your way in reverse... Imagine that you have all the things you need then what would be the minimal code that would be sufficient to write the code that you need.</p>
<p>As an example:</p>
<pre><code>mux.Register("/admin/monster/{monster.ID}", admin.Manager(&Monsters{}))
</code></pre>
<p>Then you write all those different bits and pieces of domain-logic that you need and work on the next level of abstraction, adjusting the first level as necessary. When you can implement some level there with an existing piece (such as database), great, you managed avoid some work. Eventually, you will have the full-stack implemented.</p>
<p>By taking your starting point of making different folders such as <code>admin/model</code> you are forcing your design into a solution that may not make sense for you, which leads to accidental complexity and things that don't fit together.</p>
<p>Of course, there's a downside to this approach... it requires more thinking up-front than the "framework style" throwing things together.</p></pre>jillesme: <pre><p>If you have these complex tables you will really benefit from using Django (and REST framework). Its ORM is unmatched! </p></pre>
