Best practices for managing configuration for a "fat" app?

<p>I&#39;m writing <a href="https://leanpub.com/12fa-docker-golang">12 Factor Applications with Docker &amp; Go</a> and I&#39;m currently working on code samples for Go-related chapters. I already covered <a href="https://github.com/namsral/flag">namsral/flag</a>, added a footnote about <a href="https://github.com/spf13/pflag">sfp13/pflag</a>, added an example of using <a href="https://github.com/spf13/viper">spf13/viper</a> and while I&#39;m happy with those, there are some caveats as to how I would use this from the code in a more &#34;component&#34; fashion.</p> <p>For example - in a SaaS app, you would need to have two database connections available. One would be for your data, and one would be for the read-only administration database (providing invoices and account details for example).</p> <p>What would you do in this case?</p> <ol> <li>Just create an object factory which would provide something like <code>GetDB</code> and <code>GetAdminDB</code> (or GetDB(&#34;name&#34;)), and make sure that you read configs for both databases. Use the factory from your app logic.</li> <li>Read config for both connections in main() and create DB objects, passing those to your app via context or some kind of factory model?</li> <li>Read config for both connections, your app logic would directly create <code>*sqlx.DB</code> objects with appropriate logic?</li> <li>Something like <a href="https://github.com/codegangsta/inject">codegangsta/inject</a> that would &#34;transparently&#34; provide the required/requested objects to your app functions. It&#39;s not idiomatic and there&#39;s issues to be worked out (ie, should you map <code>*sqlx.DB</code> into your typed <code>*DB</code> and <code>*AdminDB</code> to avoid type collisions &amp; have some distinction as to <em>which</em> database connection you want to use).</li> </ol> <p>I&#39;m trying to flesh out a simple way how to read a somewhat massive config, and provide it progressively to the consumers (ie, application logic). Ideally, I wouldn&#39;t provide the config to the &#34;leaf&#34;, but only the object(s) that can be constructed from it.</p> <p>In case you do something similar, which pattern worked best for you? If you saw something cool that&#39;s open source, I&#39;d also appreciate some GH links and your input as to what you like about some approach. If you want to look at my current way of doing things, there&#39;s <a href="https://github.com/titpetric/books/blob/master/12fa-docker-golang/chapter4/mysql/main.go">this example on GH from the book</a> - Ideally I&#39;d like listDatabases be only something like 5 lines:</p> <pre><code>db := factory.GetDatabase(&#34;default/admin&#34;) defer db.Close() result := []databaseName{} err := db.Select(&amp;result, &#34;show databases&#34;) return err </code></pre> <p>The obvious issue I have with the above example is that it needs to handle database connection errors (I don&#39;t want to resort to panic/recover). I can connect to the database beforehand, but I don&#39;t want to use a single connection across the whole stack (there&#39;s benefits to pooling, obviously). I&#39;m just looking to make it a bit less verbose if possible. How would you do it?</p> <p>P.s. I&#39;ll gladly give a few copies for insightful replies. Just DM me and let me know if you&#39;d like one ;)</p> <hr/>**评论：**<br/><br/>peterbourgon: <pre><p>I recall having this conversation with you before, talking about dependency injection frameworks. The method of using flags for each dimension of configuration, constructing objects in func main, and then passing them explicitly to components that need them as dependencies doesn&#39;t break down when you reach some number of dependencies. It continues to work fine for even the fattest of apps.</p> <pre><code>func main() { var ( dataDSN = flag.String(&#34;data-dsn&#34;, &#34;&#34;, &#34;DSN for data DB (rw)&#34;) adminDSN = flag.String(&#34;admin-dsn&#34;, &#34;&#34;, &#34;DSN for admin DB (ro)&#34;) // ... ) flag.Parse() dataDB, err := sql.Open(&#34;mysql&#34;, *dataDSN) if err != nil { // ... } adminDB, err := sql.Open(&#34;mysql&#34;, *adminDSN) if err != nil { // ... } // ... databases, err := listDatabases(adminDB) if err != nil { // ... } // ... } </code></pre> <p>No need for frameworks or abstractions to confuse future readers in order to save you some typing. No need for a dependency injection package or an object factory to hide deps in a magic black box. Definitely don&#39;t put them into a context object — <em>every</em> tutorial or blog post on context tells you never to use it in this way.</p></pre>titpetric: <pre><p>If i understand it correctly, there&#39;s only one connection being made in this case, and used for the whole app. It&#39;s the &#39;tricky&#39; part where you want to manage a connection pool. Concretely with mysql one connection will be limited to one cpu core. While I agree with you, it&#39;s quite a pitfall. Also if you&#39;re working with transactions it might cause some issues, but that needs testing ;)</p></pre>peterbourgon: <pre><blockquote> <p>If i understand it correctly, there&#39;s only one connection being made in this case,</p> </blockquote> <p>Negative.</p> <blockquote> <p>The returned DB is safe for concurrent use by multiple goroutines <strong>and maintains its own pool of idle connections.</strong></p> </blockquote> <p><a href="https://golang.org/pkg/database/sql/#Open">https://golang.org/pkg/database/sql/#Open</a></p></pre>titpetric: <pre><p>But if you call Open once, the logical conclusion is that it will only make one connection, and additional connections will be made only on additional calls to Open (and released on Close, ie, returned to the pool). I&#39;ll verify :)</p></pre>peterbourgon: <pre><blockquote> <p>But if you call Open once, the logical conclusion is that it will only make one connection, and additional connections will be made only on additional calls to Open (and released on Close, ie, returned to the pool).</p> </blockquote> <p>tl;dr: no, not at all. But dig in: <a href="https://github.com/golang/go/blob/d40bb738ff59bada723fe5f834d41531391b532a/src/database/sql/sql.go#L568">func database/sql.Open</a> launches the <a href="https://github.com/golang/go/blob/d40bb738ff59bada723fe5f834d41531391b532a/src/database/sql/sql.go#L582">connectionOpener goroutine</a>, which <a href="https://github.com/golang/go/blob/d40bb738ff59bada723fe5f834d41531391b532a/src/database/sql/sql.go#L838">opens a new connection</a> by delegating to the <a href="https://github.com/golang/go/blob/d40bb738ff59bada723fe5f834d41531391b532a/src/database/sql/sql.go#L847">underlying driver.Open function</a> as necessary. The driver performs the mechanics of opening and closing connections on request; the database/sql.DB maintains the pool.</p></pre>titpetric: <pre><p>I&#39;ll trust you on your word, however it deserves some verification, not because you might be wrong, but because sqlx is just one of the possible clients that might keep an open connection and manage a connection pool transparently. If you&#39;d create a new kind of client (websocket or something else that&#39;s keep-alive), you might need to create a pool of those connections or move this logic individual app functions. Again, totally valid for sql, but might not be with redigo, websocket, etc. where you&#39;re dealing with a single connection.</p></pre>titpetric: <pre><p>Also, your talks on Go best practices are awesome! :)</p></pre>materialdesigner: <pre><p>2, 2, 2, 2, only 2. Please God only 2.</p> <p>Please make main() your outermost shell of your application. </p> <p>Main() responsibility is to pull as much information from the environment to bootstrap the running program into working correctly. This means both physically pulling information from the environment (e.g. Reading binary flags, reading a configuration file, reading ENV variables) and also configuring the various subsystems and plugging them into each other, to make sure it will work when run. And <em>then</em> it <strong>tells</strong> the fully configured and initialized program to run. That means all of the rest of your code can be assumed to be configured correctly. It means the rest of your program can use the power of the type system. </p></pre>titpetric: <pre><p>tl;dr: upvoted; There&#39;s where the distinction between a microservice and a &#34;fat&#34; app would come in. I agree with you but I still need some pattern that would introduce as little friction/verbosity as possible ;)</p> <p>There&#39;s a lot of bad practice going around, including defining flags (with flag stdlib) inside init functions for submodules. Singletons are definitely something I&#39;d like to avoid. And it&#39;s usually not a trade-off with using the power of the type system, just how to <em>elegantly</em> do this without increasing verbosity :)</p></pre>materialdesigner: <pre><p>What&#39;s the problem with increasing verbosity? What makes you think it&#39;s going to increase verbosity? Where&#39;s the friction?</p> <p>And yeah, no defining things in init functions, period.</p></pre>