<p>Hey all, </p>
<p>I was wondering if there is anyone on this board that has experience with API / Database testing either with Go specifically or more generically any server side testing. I am looking for someone to challenge the following drawbacks I have found with API / database testing.</p>
<p>So far to date I have worked on several API servers in a variety of different languages / frameworks / and database backends; some with no testing in place, others with several layers of unit/integration tests.</p>
<p>These are the thoughts that I have on testing API servers but I would love to be challenged and shown better ways to approach this.</p>
<ol>
<li><p>Unit tests can be quite useless. Even if all unit tests work individually, you still have no idea if all the separate components will work together. They also tend to (but not always) test behaviors of the server side language / database back-end that you can pretty safely take for granted rather than the specific edge cases that will bite you. Obviously the best unit testers consider this while writing tests, but regardless you still don't know how the full system works together.</p></li>
<li><p>Integration tests are fragile and take a long time to set up properly. It can often take more code and time to write a good integration test that walks through an entire workflow of related APIs and measures the effects on the database, than to write the actual API code itself. Furthermore if the API or Workflow changes at all, you must revisit all the affected integration tests to update their usage, provided dummy data, and expected results / errors. Furthermore these types of tests tend to provide a false sense of security as they still allow edge cases in the system to go untested, especially concurrent usage.</p></li>
<li><p>No testing frameworks / setups I have ever encountered test the server under actual production conditions. Obviously some APIs cannot be safely run on a production database, but by using mocks, stubs, testing databases, and even separate testing machines, you are actually testing something different than your production environment. Just because a test passes on the test server doesn't mean your production code is sound.</p></li>
<li><p>There are benefits to testing, but usually the amount of work to get a good test suite outweighs the provided value and you need people that REALLY understand good testing practices to get these benefits. Without careful test writing you can get incomplete coverage, false negatives, false positives and a false sense of security. In the end it really doesn't matter if my test cases say I pass or fail, the real reality is, does the system work in production for the end user, using real data. It seems impossible to write integration tests that can replicate the usage of a typical real world user in real world production conditions efficiently. Poorly written / documented test suites are more of a hindrance than benefit altogether so unless you plan to dedicate lots of time and energy to writing and maintaining them, why bother.</p></li>
<li><p>My personal experience is that on the projects with no formal code testing, we ended up performing more real world testing with sample clients which provided more useful / real-world feedback to the system. Since we didn't have the security net of code testing, we were suspicious of everything and debugged APIs line by line when issues were uncovered or we prepared for releases. This forced us to review algorithms directly as the end user actually uses them and we would often uncover unrelated bugs or optimizations by performing this line by line debugging. Since we weren't forced to write / maintain and troubleshoot test code, we had more time to focus on the code that actually ran the system and could make changes to the API workflow more easily. We also dedicated no brain power to how our tests were set up, how our testing framework worked, and the catches and gotchas that are present in any testing suite.</p></li>
</ol>
<p>If anyone would like to counter my opinions/experiences with possible solutions I welcome the discussion. Despite all of my opinions / experiences, I do think testing can be useful when properly done and I look to see what others have done to combat the issues I have had previously. In particular, my manager is hesitant to dedicate any resources to API testing and I am trying to find a happy medium where I can still have some test coverage without breaking the project schedule to get the benefits.</p>
<hr/>**评论：**<br/><br/>jerf: <pre><p>I like to break my "web APIs" into three steps: Marshaling a request into a struct, calling some appropriate method on that struct, and marshaling the result back out to the user. I do not always break these into separate functions in my smaller handlers, but I mentally at least keep track of the line between those things.</p>
<p>This does not address everything you mention here on its own, but it helps, because generally the marshaling and unmarshaling code gets to a place where you don't really need to test it much anymore. That reduces the problem to testing the functionality of a particular object, which looks more like conventional testing.</p>
<p>However, when testing needs to hit databases to work, well, that's a general problem, isn't it? When I really care, I tend to wrap the database behind an interface and swap out a test implementation of that database, which allows me to test whatever it is I'm trying to do. This is generally useful to me because rather a lot of my code involves making a DB call, then doing a lot of non-trivial logic to it before I present it to the user. If you're basically taking stuff out of the DB and shoveling it to the user, then there really isn't much to test here on an API-by-API basis.</p>
<p>So if your unmarshaling into a struct is trivial, and your code is just querying a DB and spitting out the results to the user, which trivially unmarshal back out to the user (<code>json.NewEncoder(rw).Encode(result)</code>), there isn't much there there to actually test, no. However, when that is not the case, unit tests can still be valuable.</p>
<p>And if you clearly delineate between "marshaling this into an object that represents the request", "performing the request", and "unmarshaling the answer", you might find it arranges your code in a way that either makes it obvious some things can't really be tested, but other things are now both in need of testing, and <em>can</em> be tested.</p>
<p>This isn't actually a Go answer; it's something I've developed over the 20 years of web I've been doing. HTTP is big and icky and anything that tries to incorporate it into a test suite is asking for trouble because it's just so darned <em>big</em>. The first thing you want to do with a request is clearly and sanely boil it down to what truly matters, and hand it off via some clearly delineated interface to the "real processing" code, so that it is clear how to test that code, and it is clear exactly what that "real" code is taking as input and generating as output.</p>
<p>I like the <code>http.Handler</code> interface for how well net/http works, but I think it encourages an antipattern of jamming too many isolated concerns into one big lump in the ServeHTTP function.</p></pre>tgulacsi: <pre><p>Just as github.com/go-ki/kit separates its Endpoints...</p></pre>Virtual-Go: <pre><p>Thanks for your thoughts. </p>
<p>I especially like your idea of breaking handlers down mentally into smaller subsections, and focusing testing on the aspects likely to break or be complex and ignoring the aspects that "Just Work" for the sake of unit testing. The fewer tests you have, the less there is to maintain, the faster the test cases finish running, and the less likely you are to have false positives / negatives hidden in the test code.</p></pre>tgulacsi: <pre><p>Just as github.com/go-ki/kit separates its Endpoints...</p></pre>everdev: <pre><p>I'm wondering why there's a disconnected between the types of tests you write and the "real world" testing you're doing. It's common to uncover edge cases in production, but then ideally you can integrate those edge cases back into your tests to make your suite more robust.</p>
<p>For me, writing tests (unit & integration) is a way of automating my task of testing the application. Passing tests doesn't mean your application is perfect, but it does mean that your app works perfectly perfect for the tests you've written, which is often better than nothing.</p>
<p>There is a cost/benefit to it though and you don't need to test everything, just the mission critical stuff or things that are too time consuming to check manually.</p></pre>Virtual-Go: <pre><p>Yeah you are right, production bugs can be converted into tests, however in my personal opinion, it would be nice to have tests that would help catch these types of bugs before release.</p>
<p>You can't discount the value of regression testing but essentially the real value here would be to identify the types of areas of the software that lead to bugs and then try to prevent this class of error proactively.</p></pre>Virtual-Go: <pre><p>Also, some examples of "Tester" testing versus "Real World" testing would be something like the following:</p>
<p>1.
In "Tester" testing, a programmer tries to submit something like a 3000 character string to your API and determines that the API doesn't handle the string gracefully.</p>
<p>You could argue this is valuable information because the application should be able to gracefully handle any input.</p>
<p>But if in the real world your users never submit more than 10 characters for that field, is there value in bulletproofing your API to unrealistic data?</p>
<p>I would say pragmatically, you probably do want to make your API as intelligent as possible but in the real world where resources are constrained would it be better to focus on the data your users will realistically provide and then worry about 3000 character strings if the users ever need to submit them?</p>
<p>2.
In "Tester" testing, your testers will likely use the specification to determine what values are valid / invalid to submit to the system. If the specification happens to be inaccurate / incomplete which happens quite frequently on my current project, then you won't become aware of this until the end users get the functionality and start submitting the values and complain about unexpected behavior. Now you have to update the specification, documentation, and tests to reflect the newly redefined scope.</p>
<p>In "Real World" testing, the end users will try to submit the data as intended and identify these specification issues immediately, even if you do eventually write code tests after this phase, this ensures you write them the intended way the first time.</p></pre>hell_0n_wheel: <pre><blockquote>
<p>Integration tests are fragile and take a long time to set up properly</p>
</blockquote>
<p>The integration tests I'm running right now cover everything from API to a local mysql instance with a mock DB... including setup and teardown... in about 5 sec. About 250 lines written for setup / teardown / mocking, and now you can ensure every line of SQL in your code actually does what you expect.</p>
<p>Somewhere in Code Complete they cite a study that found bugs take something of an exponentially increasing amount of time to fix, the longer in the design / code / test / release process it lingers. If you think you're spending too much time testing, you're in for a bigger surprise when it's time to chase down bugs in production.</p>
<blockquote>
<p>No testing frameworks... test the server under actual production conditions</p>
</blockquote>
<p>I don't test under actual production conditions, I test 10x production levels. Plenty of tools available to do this, it's a problem that's been around for well over a decade. Want to hand-roll something quick and dirty, Ansible + a small python script, and you can test with as many TCP stacks as you can launch instances. </p>
<blockquote>
<p>you are actually testing something different than your production environment</p>
</blockquote>
<p>If your environment is defined in code (SaltStack, Chef, etc), you wouldn't be able to tell the difference between test and production. There are plenty of tricks one can play besides this, as well...</p>
<blockquote>
<p>Just because a test passes on the test server doesn't mean your production code is sound.</p>
</blockquote>
<p>From here to the end of your post, roughly half of it, all I can read are the frustrations of someone who isn't skilled at testing, and has no confidence in his existing tests. Now that I think of it, most of your post reads like this. Signal to noise ratio is kinda low...</p>
<p>I can't really speak to that any more than to encourage you to see how other folks are testing their codeand learn a bit about testing methodologies. <a href="http://artoftesting.com/software-testing-tutorial.html" rel="nofollow">The Art of Software Testing</a> may be a bit outdated, but it gives a good foundation of methods to consider that will make your testing more effective.</p>
<p>If you wanna skip the academics, check out <a href="https://testing.googleblog.com/" rel="nofollow">Google's testing blog</a>, or Netflix, or Spotify, or Pinterest, <a href="https://code.facebook.com/posts/testing/" rel="nofollow">or Facebook</a>... they're all solid resources for technical insights gained while running at scale.</p></pre>Virtual-Go: <pre><p>I definitely would not consider myself a testing expert, so you could very well be right that my displeasure of testing come from inexperience.</p>
<p>That being said, I also know that when tests are well done they do lend value and that's why I am trying to find helpful solutions to some of the discomforts I have dealt with previously. I am looking for some good points to help discount my manager's requests that I cut testing hours from my project estimates. Normally I would argue more fervently against it if we were writing more testable code but often my API's are simply querying a database and converting the response set to JSON, or accepting JSON and writing the values back to the DB. What I would argue would be more helpful would be some level of data validation so that the JSON object coming into the server must live up to a specific specification. Maybe someone has experience with something like this and I will figure out a better way to test these APIs.</p></pre>hell_0n_wheel: <pre><blockquote>
<p>I am looking for some good points to help discount my manager's requests that I cut testing hours from my project estimates.</p>
</blockquote>
<p>That's a whole different ball game than what you asked in your original post. Good luck with that...</p></pre>Virtual-Go: <pre><p>Its actually not if you read my full post. Note the last sentence, probably missed it.</p>
<p>In some regards I am playing devil's advocate on this post because often the best advice you get is a rebuttal to a perceived weakness or flaw. As I mentioned, I do think testing has a value and a place, but on JSON APIs I struggle to find real meat worthy of testing. The only tests I think have any value on these APIs right now would be full integration tests because while each individual API is simple, they must work together properly to facilitate complex work flows.</p>
<p>I am looking for suggestions that would help me to quickly put together tests suites that validate these complex workflows without having to pump inordinate amounts of time to set them up and maintain them as the specification changes.</p>
<p>Perhaps someone more experienced than me can share the wisdom of how they would accomplish this on a large scale project with shifting specifications.</p></pre>Virtual-Go: <pre><p>"I don't test under actual production conditions, I test 10x production levels. Plenty of tools available to do this, it's a problem that's been around for well over a decade. Want to hand-roll something quick and dirty, Ansible + a small python script, and you can test with as many TCP stacks as you can launch instances."</p>
<p>I agree you can script this to blast the hell out of your test database well beyond the levels of production. My point centered on the fact that its difficult to manage this when you end up with 3 or more development environments with different schema because by very nature production, testing, and development will never all be in the same state at the same time on large scale projects. For any record you have in your testing database, your expected test results will have to reflect the results of that record being included. This often means you are either using dummy test data which is not reflective at all of real data, or you are testing a subset of data conditions that could exist in your application but likely not that 14 million record legacy production table you are forced to support and sometimes goes down because of 'network issues'. This doesn't even remotely include issues that could be the result of tests failing simply because they are on the test server and the test server has a hardware issue that is not present on the production server. </p>
<p>Also you mentioned you use mocking which by its very nature means you are not testing your production code. You are testing your mocking implementation. What happens if a developer on the team writes a mock that doesn't properly represent the feature it is mocking? What happens if using the mock in your test is actually hiding a bug you would have found by using the production implementation directly? In both cases you are being impeded in your flow, and the end result is the code compiled in your production binary is not tested. What if you overlooked one thing or somebody set an environment flag differently between your production and testing environment and the binary is compiled slightly differently on one setup.</p>
<p>Yes that is a pessimistic view, but the reality is if you don't consider things like this you can get bit by them as I have on previous projects where the lead didn't consider these types of eventualities. Hope you don't mind my challenges, as I would like to learn from you if you genuinely think these are solved problems.</p></pre>masterwujiang: <pre><blockquote>
<ol>
<li>No testing frameworks / setups I have ever encountered test the server under actual production conditions.</li>
</ol>
</blockquote>
<p>With help of containerization, you can construct your testing environment as same as production, like master-slave setup or memcache as cache layer. </p></pre>Redundancy_: <pre><p>I've done this, and it's still slow and nasty.</p>
<p>Typically, you want some sort of versioning scheme for database updates, and running that to provision a database and do all the updates from scratch can add minutes to your DB initialization. </p>
<p>I won't say you shouldn't make it possible, but maybe it's something more for running more occasionally, perhaps with something like <a href="http://dredd.readthedocs.io/en/latest/" rel="nofollow">http://dredd.readthedocs.io/en/latest/</a></p></pre>Virtual-Go: <pre><p>I do actually have a mirrored testing and production database and server.</p>
<p>I still think this results in several issues from a consistency standpoint. Until I know of a better way it is the way I personally test APIs.</p>
<p>You will always have to maintain a specific set of data in the testing database so you can generate expected results and errors, however the production database is constantly filling up with unique combinations of data that could result in specific edge cases in the larger system.</p></pre>burnaftertweeting: <pre><p>Im a bit of a scrub when it comes to testing, but I set up a simple - though far from through set of tests via Postman. </p>
<p>By running your api against a test db and using Postman to make requests you can verify that most of your routes work as expected. Takes about 5-10 mins a route once you get the hang of it.</p>
<p>No its not ideal, and no you arent getting full code coverage. But if youre looking for just good enough coverage, you can set up basic integration across an api in a few hours.</p></pre>Virtual-Go: <pre><p>That sounds interesting.</p>
<p>Do you use input files to Postman to setup the testing?</p></pre>burnaftertweeting: <pre><p>You can use Postman to set up collections and environments suitable for different projects, then export those files and use Newman (command line Postman) to run your tests dynamically. So I built all the tests by hand first, exported to JSON and now I'm working on getting the input from my go app. From there I can run tests or a build script that verifies tests pass automatically before build.</p></pre>Virtual-Go: <pre><p>How long have you been using this kind of setup? Are their aspects of it that you don't like or find cumbersome?</p>
<p>I have been using Postman directly to inspect and test my APIs as I build them but this seems like a pretty cool solution to get some automated testing in place.</p></pre>thelamacmdr: <pre><p>I had a much longer post written up but decided to scrap it because I wasn't quite sure how much value it would have added. A lot of the posts I'm reading it seems -- and this might be a wild assumption on my part -- are from the perspective of developers that need to test their own code.</p>
<p>As a dedicated QA tester, writing test automation and designing tests is all I do so this stood out to me (emphasis mine) :</p>
<blockquote>
<p>My personal experience is that on the projects <strong>with no formal code testing</strong>
This is because each of the points (except 1) you present are engineering problems in and of themselves. Once you've solved the problem once, the time invested could potentially pay off. Of course whether it truly is worth it is probably something you'd have to decide. I could probably finish that full write up with my own experiences if it does apply to you in some way, sleep be damned.</p>
</blockquote></pre>Virtual-Go: <pre><p>I agree, that is why I am trying to see how people have approached these issues in a general sense. When my manager is using the above points to argue that we don't have time to write tests or maintain them in a resource constrained project, I want to have some good suggestions to counter his points and find a happy medium where I can get a few extra hours per sprint to account for some testing.</p></pre>SeerUD: <pre><p>Test what's worth testing. If you find a bug, add a test that identifies it to make sure it doesn't happen again, whether that be unit, or functional.</p></pre>Virtual-Go: <pre><p>That has tended to be my approach. I would argue the usefulness of the test is highest when you don't know there is a bug in the first place, but regression testing is definitely nice to have when refactoring. Cheers.</p></pre>
