<p>Here in the office we read data from csv files, but someone mentioned that we could use json files because it was faster. I just want to know if it is worth the time.</p>
<p>Or any other file you might think would fit better.</p>
<p>Thanks!</p>
<hr/>**评论：**<br/><br/>metamatic: <pre><p>It's worth pointing out that CSV is a lot less standardized than JSON. While there's <a href="https://tools.ietf.org/html/rfc4180" rel="nofollow">RFC 4180</a>, in the real world you'll find a lot of variations from that.</p>
<p>I've dealt with reading and writing files containing millions of records of data as CSV, and I very much doubt speed of CSV vs JSON parsing will be a bottleneck for anything you're doing.</p></pre>gopherdiesthrowaway: <pre><p>This. The whole question sounds like premature optimization.</p></pre>Kipzy: <pre><p>We get a primary csv file which we need to work on. I've been doing just fine with the csvs, but someone from the team(2guys working in python) said we could read the csv and write the same thing to a new JSON file so they could work "faster".</p></pre>barsonme: <pre><p>This entirely depends what your data looks like and what you're using it for.</p></pre>xrstf: <pre><p>Depends on your data.</p>
<p>CSV can usually by parsed and processed incrementally, so your memory footprint stays relatively constant. For a JSON file, you usually parse it as a whole (like you'd do with a DOM parser for XML). I had a case where I wanted to load JSON 100MB JSON files and the memory usage exploded in my face (in PHP and Node.js).</p>
<p>You <em>can</em> incrementally parse JSON (like with a SAX parser for XML), but then you have to keep track of your state and such.</p>
<p>What you gain from using JSON is stricter string handling, so if you data contains "bad" characters for CSV (line breaks, commas, quotes), JSON can help you handle that.</p>
<p>Some people went with the best of both words: A file with one JSON document per line, so you can still incrementally walk through the file, but also have the niceness of JSON parsing. There's a dedicated name for these JSON-line files, but I forgot it.</p>
<pre><code>{"id":1,"foo":2}
{"id":1,"foo":2}
{"id":1,"foo":2}
</code></pre>
<p>You can also be a bit more strict and create a valid JSON <em>file</em> like this:</p>
<pre><code>[
{"id":1,"foo":2},
{"id":1,"foo":2},
{"id":1,"foo":2}
]
</code></pre>
<p>Just skip the first and last line and trim the trailing commas. Or read the file as a whole. If you are the only producer and consumer of the JSON files, this might be a choice; otherwise I'd stay away from this, as it leads to weird "we output JSON, but we promise to format it in a special way so you can read it line by line and do this magic." statements in documentations.</p></pre>cathalgarvey: <pre><p>Instead of serialising a list of objects, one JSON-object per line gives you the best of both worlds: incremental encoding/decoding and the cleanliness of JSON.</p></pre>sbinet: <pre><p>incrementally parsing JSON is relatively easy though:
<a href="https://godoc.org/encoding/json#example-Decoder-Token" rel="nofollow">https://godoc.org/encoding/json#example-Decoder-Token</a></p>
<p>see for example my solution to the advent-of-code day#12 (which involved counting numbers inside deeply nested JSON objects):
<a href="https://github.com/sbinet/advent-of-code/blob/master/day-12/main.go" rel="nofollow">https://github.com/sbinet/advent-of-code/blob/master/day-12/main.go</a></p>
<p>modifying it to hand over users a stream of JSON objects lines should be relatively straightforward...</p></pre>schoenobates: <pre><p>Could this be the json-per-line format?: <a href="http://jsonlines.org" rel="nofollow">http://jsonlines.org</a></p></pre>xrstf: <pre><p>Yep, that was it!</p></pre>Kipzy: <pre><p>I won't be the only one working with this, we are a team of 4 people, 2 in go and 2 in python. </p>
<p>We get a primary csv file which we need to work on. I've been doing just fine with the csvs, but someone from the team(the 2 guys from python) said we could read the csv and write the same thing to a new JSON file so they could work "faster".</p></pre>mc_hammerd: <pre><p>csv for tables of data</p>
<p>json for key/value store and supports arrays/objects as values</p></pre>cathalgarvey: <pre><p>CSV has interchange value: you can dump from and import to spreadsheets. If you don't need this, use JSON.</p>
<p>If you're dumping lots of data, instead of dumping a slice of objects, instead dump each object on a new line: all the benefits of CSV's linewise parsing, with all the type-hinting and structure of JSON.</p></pre>Justinsaccount: <pre><p>Loading in a json file may be faster development wise, but it is definitely not faster from a file size and performance standpoint.</p>
<p>This is not likely to matter unless you are working with multi gigabyte data files. I think the last time I benchmarked it, extracting some columns from a csv file was a bit faster than doing the same thing for the same file written using the "json lines" method. Over 20G of log files, this adds up.</p>
<p>If all you are doing is sending around a few megs of data, use whatever is easiest to work with.</p></pre>RobLoach: <pre><p>JSON.... More control.</p></pre>Fwippy: <pre><p>It seems unlikely that the speed of reading CSV files is meaningfully slower than JSON in your application.</p>
<p>Have you profiled your code and seen that a lot of time is taken in the CSV reader? If so, create a benchmark with CSV and JSON readers, and compare. If JSON comes out a ways ahead, then you can consider switching to it for speed reasons.</p>
<p>But if you haven't done that, this really smells like a very premature optimization.</p></pre>Kipzy: <pre><p>I haven't done that, but i'm doing fine with the csv file. </p>
<p>We get a primary csv file which we need to work on, but someone from the team(2guys working in python) said we could read the csv and write the same thing to a new JSON file so they could work "faster".</p></pre>Fwippy: <pre><p>Oh, like faster for coding time, easier for them to work with? That's definitely a valid reason, I thought you meant execution time of the program.</p></pre>Kipzy: <pre><p>and it is for execution time of the program, plus we report with Csvs files, so i really want to know if it is a waste of time doing that csv2json thing</p></pre>earthboundkid: <pre><p>Gob. </p></pre>Kipzy: <pre><p>it won't be Go exclusive</p></pre>lethalman: <pre><p>There's no reason you should use csv. Write a json object per each line, and you will save your life. It's not about being fast, it's about being robust.</p>
<p>By the time you really need a csv because your tool doesn't support json, then convert json to csv.</p>
<p>Two main reasons compared to csv:</p>
<p><strong>You can store one JSON object in one text line</strong></p>
<p>That means reading and parsing json is a lot easier to parallelize (as long as your final algorithm is parallelizable, of course).
While for a csv, if strings are encoded within quotes, a single csv record may span multiple lines, which makes it harder to parallelize.</p>
<p><strong>JSON has data types: null, bool, number, strings</strong></p>
<p>Csv has just strings without null, unless you have some kind of protocol (like the first row in the csv describes the types). That means you are losing information when exporting to csv: when importing you may need to interpret a "true" string as true boolean, a "number" string as a number, and an empty "" string as null, which may or may not be correct unless you have a clear schema of your csv.</p></pre>Justinsaccount: <pre><blockquote>
<p>JSON has data types: null, bool, number, strings</p>
</blockquote>
<p>You're missing the bigger ones: JSON natively supports nested arrays and hashes in a standard way.</p></pre>
