What is the best way to pass a db to web handlers in a large project?

<p>I know this is a topic that has been discussed a few times on here already, but it seems there is a lot of contradiction from post to post. To my knowledge, there are three popular ways to share a database with <code>http.HandlerFunc</code> type functions:</p> <ol> <li>Global database variable in the package. This is obviously a bad choice for large programs.</li> <li>Dependency injection: <ul> <li>Using middleware to pass the database to handlers. Of course the signature of the handlers (<code>func something(db *DB, w http.ResponseWriter, r *http.Request)</code>) now breaks the http.HandlerFunc signature, and is therefore often not recommended.</li> <li>Creating a struct which contains a pointer to the database, and attaching functions with the <code>http.HandlerFunc</code> to that struct. This allows us to access the struct, and therefore the database.</li> </ul></li> <li>Using the <code>context</code> package. Simply add the database pointer to the root context / use middleware to add the database to the root context of requests that require it.</li> </ol> <p>From what I understand, using global variables is bad, and using the context package is wrong, because we loose oversight about which handlers require what.</p> <p>This leaves Dependency Injection, but if we do not want to break the <em>holy</em> <code>http.HandlerFunc</code>signature, we are left with attaching hundreds of functions to a struct which we first have to initialize with the database. What do we do if we have many packages that require access to a database? Do we simple create a struct per package, add functions to that, and initialize them all in the main package?</p> <hr/>**评论：**<br/><br/>kpurdon: <pre><blockquote> <p>Creating a struct which contains a pointer to the database, and attaching functions with the http.HandlerFunc to that struct. This allows us to access the struct, and therefore the database.</p> </blockquote> <p>Kind-of ... I usually like to keep direct database access out of http handlers, opting to have an internal package that contains all of the database access layer. This internal package exposes a struct which has the actual sql.DB (or conn pool/...). I pass this struct (often an interface, not the actual struct) to my http handler struct. </p> <p>Here is a specific example from a project I hack on in my free time:</p> <p><a href="https://github.com/kpurdon/beercellar/blob/master/internal/beercellardb/beercellardb.go#L30-L78" rel="nofollow">https://github.com/kpurdon/beercellar/blob/master/internal/beercellardb/beercellardb.go#L30-L78</a> is the internal database access interface/client</p> <p><a href="https://github.com/kpurdon/beercellar/blob/master/handlers/beercellar/beercellar.go#L10" rel="nofollow">https://github.com/kpurdon/beercellar/blob/master/handlers/beercellar/beercellar.go#L10</a> is the struct where I take the database (Datastore) interface. All the http handlers are methods on this.</p> <p><a href="https://github.com/kpurdon/beercellar/blob/master/main.go#L44-L52" rel="nofollow">https://github.com/kpurdon/beercellar/blob/master/main.go#L44-L52</a> is where this all gets initialized.</p></pre>heraclmene: <pre><p>I too, do this. You can see an example here: <a href="https://github.com/bunsenapp/bunsen/blob/master/services/user/http/user_handler.go" rel="nofollow">https://github.com/bunsenapp/bunsen/blob/master/services/user/http/user_handler.go</a></p> <p>It works extremely well and makes testing super easy. </p></pre>kpurdon: <pre><p>Simplicity during testing is one of the huge advantages for sure.</p></pre>stdiopt: <pre><p>That&#39;s a big interface, I know its kind of unrelated but.., I&#39;ve been trying to idealize something which at first I took the same approach but I read somewhere that golang interfaces should be simple as having one method, the golang official comments says something about interface naming like its good to suffix one method interfaces with &#39;-er&#39; (Writer,Reader) and people also claim that we should not obligate the dev to implement everything but just what he wants, I came up with a different solution that i&#39;m not sure if it is idiomatic<br/> package usersvc</p> <pre><code>type DataStore struct { Create func(*User) error FindByName func(string, *User) error FindByEmail func(string, *User) error } func NewDataStore() *DataStore { return &amp;DataStore{ Create: func(*User) error { return ErrNotImplemented }, FindByName: func(string, *User) error { return ErrNotImplemented }, FindByEmail: func(string, *User) error { return ErrNotImplemented }, } } </code></pre> <p>The constructor here is more like an helper than a need</p> <p>the mainly disadvantage i see on this is that on a missed implementation it will fail on runtime rather on compile time as interface, but honestly obligating the dev to implementing every single method of interface will lead to him having some empty ones for the sake of compiling and cause the same effect</p></pre>kpurdon: <pre><p>For a public package, with an often implemented interface I totally agree. Here, the utility of the package/interface is the ability to mock out the database layer for unit testing. I do this using vektra/mockery and testify/mock. Less concerned about the size since I don&#39;t expect anybody else to implement this interface.</p></pre>stdiopt: <pre><p>I usually try to keep internal implementations like they were public, it will keep the consistency and one day later I&#39;ll be my public. I understand your point, I&#39;m still studying around the mockery is indeed a plus, and still not sure if the struct way is &#39;good&#39; or pleasant for third parties</p></pre>murzark: <pre><p>New Go developer here. I have been thinking about this matter too as I&#39;m sure many new Go developers do.</p> <p>Just want to say thank you for the clear examples.</p></pre>jonnydoeboy: <pre><p>Yeah, so this is basically Dependency Injection by attaching functions to a handler struct. I agree that it&#39;s a good idea to separate the handler code from the actual database queries.</p> <p>I guess if we have too many actions on a database, we can instead attach these functions to the models - passing a database or similar.</p> <p>I really appreciate the clear examples!</p></pre>kpurdon: <pre><p>I tend to shy away from thick models, but yes that is also an option.</p></pre>thechilts: <pre><p>You don&#39;t have to break the <em>holy</em> http.HandlerFunc signature. Here&#39;s what I sometimes do. Let&#39;s call this option (4) which you didn&#39;t have in your original list:</p> <pre><code>func homeHandler(db *DB) func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) { // here, you have access to the db // ... } } </code></pre> <p>To get your handler, you would of course call it with the db as follows:</p> <pre><code>homeHandler(db) </code></pre> <p>So this does a few things that you talk about:</p> <ul> <li>it isn&#39;t a global variable since it is passed from wherever you choose to use it</li> <li>the <em>holy</em> http.HandlerFunc is not broken</li> <li>you can do this for any handler anywhere, in different packages</li> </ul> <p>Note: using the context for a database connection is not recommended. The context should generally only be for things that are related to that request. Of course, not everyone agrees with this, though I think most generally stick to it.</p> <p>Your point about the struct which contains the db and attaching functions to it is also valid. I&#39;m not saying the above replaces that since I agree that that is also a good solution, but this helps with a few of the other negative points you mentioned about the other solutions. :)</p></pre>Shonucic: <pre><p>I have no problem breaking the HandleFunc signature. That is what the http.Handler interface is for.</p> <p>Create a struct with the stuff you need on it and a ServeHTTP method. </p> <p>Instantiate one with your database connection, the actual handler (which can now have whatever signature you like), and whatever else and pass it to your router for each route.</p> <p>ServeHTTP passes the db connection to the <em>actual</em> handlers.</p></pre>daveddev: <pre><p>So, 2b, or not 2b. That is the question?</p> <p>...</p> <p>Ok, really... yes. Include the db connection pool as a pointer in the relevant types. If it&#39;s a helper function in custom sub-package which does not have a relevant &#34;contextual&#34; type, then the db would likely be the first arg. My main funcs are often simply setting up needed types and passing dependencies into them, then calling some sort of &#34;run&#34; func(s).</p></pre>ArghusSquare: <pre><blockquote> <p>This is obviously a bad choice for large programs.</p> </blockquote> <p>Is it? I&#39;m asking because I have seen it in many stack overflow answers. Like a package &#34;database&#34; with &#34;var DBCon *sql.DB&#34;</p> <p>And every other packages access it with database.DBCon. Is it really that bad?</p></pre>jonnydoeboy: <pre><p>If you are working alone, probably not. If a team is working on different parts, you may want to make it explicit that the handlers require a database connection.</p> <p>While the *sql.DB is safe for concurrent access, it&#39;s more of a best practice.</p></pre>ArghusSquare: <pre><p>Hm, I have also seen it in nodejs projects with every route having the var db = require(&#39;mydb&#39;).db() code. I mean at the end if you need to access the database in the handler, you&#39;ll use it. </p></pre>colezlaw: <pre><p>I think the approach you&#39;ll see most often is DI, where you make a closure to hold the DB connection:</p> <pre><code>func MakeDBHandler(fn func(db *DB, w http.ResponseWriter, r *http.Request) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { db := GetDB() fn(db, r, w) } } ... http.HandleFunc(&#34;/foo/&#34;, MakeDBHandler(myDBHandler)) </code></pre> <p>Or you could make MakeDBHandler take the DB as an argument.</p> <p>You really don&#39;t want to put the DB connection on the Context (IMO) because you can&#39;t tell by API that it needs to be there. <a href="https://dave.cheney.net/2017/01/26/context-is-for-cancelation" rel="nofollow">https://dave.cheney.net/2017/01/26/context-is-for-cancelation</a> has good arguments against putting most types of values on the context.</p></pre>Tikiatua: <pre><p>Hi there,</p> <p>We have built our fare share of complex projects and went with using closures to pass in a database interface after trying almost every other option.</p> <p>If you really want to have code that is easy to understand and to test, you could even write specific interfaces for the handler functions and pass in a database struct that will satisfy this specific interface.</p> <p>Although this might sound like overkill, it will make it very easy for you to write mock database structs to test specific handlers.</p> <p>Please note that we are using echo as web-framework.</p> <pre><code>// Define the interface your handler will require type dbInterface { Get(string) string } // GetSample will return the sample value to the given id func GetSample(db dbInterface) echo.HandlerFunc { return func(ctx echo.Context) error { sampleID := ctx.Param(&#34;id&#34;) if sampleID == &#34;&#34; { return echo.NewHTTPError(http.StatusBadRequest, &#34;please provide an id&#34;) } sampleValue := db.Get(sampleID) return ctx.String(http.StatusOK, sampleValue) } } </code></pre></pre>cocotyty: <pre><p>Maybe you can use summer. <a href="https://github.com/cocotyty/summer" rel="nofollow">https://github.com/cocotyty/summer</a></p> <p><a href="https://github.com/cocotyty/summer-web-application-example" rel="nofollow">Example</a></p></pre>ArghusSquare: <pre><p>Does anyone know what works best for gorilla/rpc? </p></pre>phlatphrog: <pre><p>[go noob here, so take this with a block of salt.]</p> <p>Based on the end of <a href="https://blog.golang.org/error-handling-and-go" rel="nofollow">the blog post about error handling</a>, where it is demonstrated how to wrap your controllers for consistent error handling, I&#39;m gonna vote for the &#34;middleware&#34; approach. I don&#39;t think having your own flavor of controllers (ie, diff sig than &#34;standard&#34;) is any sin. It seems to me to be A Right Thing.</p></pre>jonnydoeboy: <pre><p>I totally agree that Middleware the way to go in regards to logging, error handling, making sure the user is authorized, etc. I wish I could provide sources, but I am on mobile right now, but I do keep reading that <code>http.HandlerFunc</code> is &#39;holy&#39;, and changing the signature means it looses utility. </p> <p>The link you provided however does (kind of) change the function signature, in that it also returns an error. I can see that being acceptable, as the parameters do not change. I wonder if the common consensus is to just create a custom function signature that allows the database to be passed. </p></pre>nhooyr: <pre><p>Fairly certain that is the common consensus.</p> <p><a href="https://blog.golang.org/error-handling-and-go" rel="nofollow">https://blog.golang.org/error-handling-and-go</a></p></pre>jonnydoeboy: <pre><p>That&#39;s the link I was talking about. It seems to be acceptable to return something, but not change the actual calling signature.</p></pre>nhooyr: <pre><p>oh my, sorry, didn&#39;t read the original comment.</p> <p>It&#39;s far more useful imo to have the handler return an error because it simplifies repetitive error handling so much. imo, it should just be a standard part of net/http, it is much more elegant.</p> <p>caddy does it too in their httpserver package.</p> <p>Though, really, once you make it return something, it doesn&#39;t matter. You&#39;ve lost compatibility with standard net/http, you can do whatever you want. It&#39;s just not really that necessary to change the signature because you can make the handler a method on a struct that contains your dependencies.</p></pre>throwlikepollock: <pre><p>Fwiw, i&#39;d choose middleware but <em>not</em> to change the signature of your http func.</p> <p>You can do this with closures. Passing in your db to a function that returns a http handler. That http handler will now always have access to the db through closures.</p> <p>Not sure if there are any significant costs to closures for high performance requirements like dbs might have, so keep that in mind. Always perf :)</p></pre>jonnydoeboy: <pre><p>As long as I define my actual handler inside of the closure, this works. But if I just want to pass a function to a middleware function, I have no way of accessing the database from the handler function if I do not change the signature.</p></pre>