Advice on building a load testing tool in Go

<p>I have an interesting project that I&#39;ve been maintaining for a while, it consists of two separate tools: a load generator and a data generator. The data generator outputs an arbitrary and configurable set of Google protocol buffer data, the load generator reads that data and sends it to various backend endpoints (note: this is a distributed backend system not a traditional web app).</p> <p>The load tool is written in Python, the data tool is written in Java. Both tools could use a major overhaul/refactor. I feel like the two tools could be re-written and combined into one tool and I think Go would be perfect for this. I could enumerate the reasons but they would probably be obvious to everyone here in <a href="/r/golang">r/golang</a>.</p> <p>I&#39;m very new to Golang and haven&#39;t done a lot more than write some very small scripts/tools. I was wondering if I could get some feedback here on how to set up the data structures and concurrency model for this tool. </p> <p>I want to spawn individual clients (i.e. users who will be sending protobuf data to our endpoints) and then will need to periodically check to see if a run has been cancelled. Actually, each individual client can send 2 to 3 types of data to various servers which equates to 2 to 3 threads in the Python code. As a future enhancement, I would like to generate and send data on the fly instead of generating data with the Java tool, writing it to disk and moving it to a VM/drive where the Python scripts can access it.</p> <p>Basically, I have a lot of concurrent I/O, writing to disk, http requests and I&#39;ll probably be sending some data/results to S3 bucket at some point too.</p> <p>What I&#39;m looking for is ideas on how to set up the channels, goroutines primarily. I might use an external queue for storing data temporarily as well, either RabbitMQ or SQS, or just do it in Go if that would make more sense - the case I&#39;m picturing there would be where I&#39;ve generated data quickly enough that my http goroutines aren&#39;t ready to process it, also because this is a load tool, there are cases where I&#39;ll want to intentionally stagger when each client sends up data.</p> <p>Thanks in advance.</p> <p>BTW, I may just do most/all of this on my own time. I&#39;d like to learn Go and I&#39;d like to replace these tools. My ultimate goal would be to come back to my team and blow them away with something much better than what we have now. </p> <hr/>**评论：**<br/><br/>VivaceNaaris: <pre><p>Checkout <a href="https://github.com/gocraft/work" rel="nofollow">gowork/work</a>. It ships with a web-based GUI for your work, and has a nifty framework and way of handling the job process. Writing middlewares to record metrics will be a breeze for you, and you won&#39;t have to worry (as much) about goroutine managment. I&#39;m not sure how well it would handle with the desired request rate, but I&#39;ve personally used it to handle ingesting data from web endpoints at rates in excess of 600/second (tiny to your requirements but notable). Also, it&#39;s worth mentioning that it uses Redis, which is outside of your requirements. I&#39;m not sure how flexible you are on RabbitMQ.</p> <p>That would take care of a good chunk of the structuring, metrics, and you&#39;ll have no issue generating a large number of workers.</p> <p>As for generating data on the fly... that won&#39;t be an issue in my opinion. Not entirely sure about the nature of the data, but creating constraints for values and writing a pseudo-random generator wouldn&#39;t be hard at all. I might suggest that before you overhaul both systems simultaneously, you adjust your Java too to send the generated test data to your testing service(s) via HTTP. That way, you don&#39;t need to worry about two programs breaking while you try to fix one.</p></pre>tech_tuna: <pre><p>This looks excellent, thanks! Sure Redis is fine, RabbitMQ was just an idea really. </p> <blockquote> <p>I might suggest that before you overhaul both systems simultaneously, you adjust your Java too to send the generated test data to your testing service(s) via HTTP. That way, you don&#39;t need to worry about two programs breaking while you try to fix one.</p> </blockquote> <p>Yes, I&#39;d most likely replace the Python tool with Go first but I really want to replace both of them at some point with something written in one language.</p></pre>