<p>Hi all,</p>
<p>Yesterday I did a presentation about Go to our dev teams and one of the major valid concerns raised was limiting which libraries developers are allowed to use (due to license review).</p>
<p>In a corporate environment, where we ship on-prem software to our customers, every single library we use (down to the exact version number) has to be reviewed by our architecture and legal tean (e.g. GPL avoidace) before it can be utilized in our commercial products.</p>
<p>In the Java world, we use Artifactory to create a repo of only the approved libraries (and versions) that have been reviewed and then all of our build tools only pull the libraries from this internal repo. Our build servers are actually shut off from the internet, so no one can pull in a library from Java Maven Central and bypass this process.</p>
<p>This is a legal requirement and we would be exposed to liability from our customers if we didn't do this.</p>
<p>As you may guess, this badly falls apart for Go where every import has the URL of the repo directly in the package name (e.g. github.com/user/package, etc). Hence it is not possible to mirror it on a different, internal URL and limit just to the libraries and versions that have been approved.</p>
<p>Just wondering how are other companies with similar legal requirements handling this when doing Go development?</p>
<p>This was actually identified as a major blocker before we could look at Go adoption.</p>
<p>Thank you for any suggestions.</p>
<hr/>**评论：**<br/><br/>dgryski: <pre><p>At Booking, we use git submodules for vendoring. However, the submodules point at copies of the repositories on our internal git server, instead of at the actual upstream repo on github or wherever. When we add a submodule, it goes in the place in GOPATH where the library <em>should</em> be. We have a monorepo containing our entire GOPATH.</p>
<p>Adding a dependency then is reviewing the license, cloning to our internal server, then adding a submodule pointing at our copy in the correct place.</p>
<p>If you're not using submodules, it becomes even easier. Just check in the vendored source wholesale into your GOPATH (assuming the entire thing is under version control.)</p></pre>jfurmankiewicz: <pre><p>but that means you cannot use newer tools like govendor, right?</p>
<p>You basically have to check out everything from your local repo into $GOPATH.</p>
<p>Seems kinda clunky and probably easy to bypass by a lazy developer who doesn't want to go through the hassle of license approval (and we've had a few of those)....</p></pre>dgryski: <pre><p>You might also find <a href="https://github.com/pmezard/licenses">https://github.com/pmezard/licenses</a> useful</p></pre>aerth0x1A4: <pre><p>In case someone wanted to check their whole go path... </p>
<p><code>go get -v github.com/pmezard/licenses</code></p>
<p><code>cd $GOPATH && licenses $(go list ./...)</code></p></pre>jfurmankiewicz: <pre><p>+1 good stuff</p></pre>adampresley: <pre><p>That's really handy</p></pre>dgryski: <pre><p>Correct; our dependencies are managed by our own tools.</p>
<p>Git submodules handles cloning all the repositories.The <em>entire</em> GOPATH is our repository. So, clone the repository, cd into it, and then you can start working (after pulling submodules, of course).</p>
<p>Our build server also checks out the repository (including submodules) and builds from that. If a dev has added a package to their local GOPATH, the CI will fail. You can have a separate audit script that checks the list of submodules on your internal git server.</p></pre>dgryski: <pre><p>Note that whatever solution you chose, you won't be able to run <code>go get</code> anymore.</p></pre>intermernet: <pre><p>Considering the dev machines aren't internet connected, they could probably hack around this by modifying host files or internal DNS to point the common repo hostnames to their internal git server.</p>
<p>It's a horrible hack, but it would probably work.</p></pre>weberc2: <pre><p>You can create an internal repository of approved software using git, hg, etc and <em><strong>vendor</strong></em> from those repos.</p>
<p>EDIT: Added emphasis for downvoters who misread. ;)</p></pre>jfurmankiewicz: <pre><p>Yes, but the URL changes for all the imports.</p>
<p>It would not be anymore, e.g.</p>
<pre><code>github.com/gin-gonic/gin
</code></pre>
<p>but</p>
<pre><code>git.mycompany.com/gin-tonic/gin
</code></pre>
<p>in all of our Go code. Any library that gets pulled in and references the original URL of any of its own dependencies would fail to compie, as it would still refer to the original github URL.</p>
<p>So this would not work</p></pre>ItsNotMineISwear: <pre><p>I'm pretty sure you can vendor the dependencies under a "github.com" folder and then the import won't change.</p></pre>jfurmankiewicz: <pre><p>OK, so I am looking at the vendor/vendor.json file and let's say for my gin library:</p>
<pre><code> {
"checksumSHA1": "XxSDNaJf0LpjvMapDmmhNIlf+MQ=",
"path": "github.com/gin-gonic/gin",
"revision": "5caaac4c5c712a9e7a7de29e6c24ef46c753017f",
"revisionTime": "2016-04-14T23:39:28Z"
},
</code></pre>
<p>how would I vendor it to say "check it out from git.mycompany.com but put it under github.com"?</p></pre>HectorJ: <pre><p>Most vendoring tools allow you to specify an import path, but to use another VCS source.</p>
<p>Example with govendor: <a href="https://github.com/kardianos/govendor/blob/master/doc/faq.md#q-ive-forked-a-package-and-i-havent-upstreamed-the-changes-yet-what-should-i-do">https://github.com/kardianos/govendor/blob/master/doc/faq.md#q-ive-forked-a-package-and-i-havent-upstreamed-the-changes-yet-what-should-i-do</a></p>
<p>Example with Glide: <a href="https://github.com/Masterminds/glide#glideyaml">https://github.com/Masterminds/glide#glideyaml</a> (see the repo field)</p></pre>jfurmankiewicz: <pre><p>YES!</p>
<pre><code>Q: I've forked a package and I haven't upstreamed the changes yet. What should I do?
A: Assuming you've pushed your changes to a accessable repsoitory, run
govendor fetch github.com/normal/pkg::github.com/myfork/pkg.
This will fetch from "myfork" but place package in "normal".
</code></pre>
<p>This will do very well. I think we can definitely live with this.</p>
<p>All give gold to HectorJ above.</p></pre>weberc2: <pre><p>This is why I suggested you vendor your dependencies, so you don't have the URL business to worry about. The repo is mostly for reference, so you and your colleagues know which libs are safe to vendor into your projects. I think this should suffice for your use case?</p>
<p>The big takeaway is to use vendoring, not <code>go get</code>.</p></pre>jfurmankiewicz: <pre><p>Sorry, I must be missing something.</p>
<p>Even if we use vendoring, it still fetches it (using govendor or a similar tool) from the main github URL, not from our internal repo.</p>
<p>So there really seems to be no way to lock this down.</p>
<p>Please correct me if I am missing something, my knowledge of Go vendoring is still limited at this time to basically using govendor.</p></pre>weberc2: <pre><p>I don't know much about <code>govendor</code>, but I don't see anything that's stopping you from copy/pasting from your repository into a project's /vendor directory and dropping in a <code>VERSION</code> file. This could be easily automated.</p></pre>int32_t: <pre><p>You can use <a href="https://getgb.io/"><em>gb</em></a>.</p>
<p>Basic usage:</p>
<pre><code>$ mkdir -p myproject/src/helloworld
$ cp main.go myproject/src/helloworld/ # Put the source there
$ gb vendor fetch github.com/spf13/cobra # As an example of dependent library.
$ gb build
</code></pre>
<p>Note that you <em>have to</em> fetch <em>all</em> third-party libraries (the ones other than the built-in standard libraries) for a gb project.</p>
<p>Not sure what's still missing for your requirements though.</p>
<p>Edit:
There is a manifest file (which is literally a JSON file) stored in the vendor/ directory under the project directory.
It records both the <code>importpath</code> and the <code>repository</code> URL for each package independently. I suppose one can change the <code>repository</code> URL to map to elsewhere, but I have not tried it.</p>
<p>Edit2: The manifest file is employed by <code>gb vendor restore</code>.</p></pre>caseynashvegas: <pre><p>My suggestion is to take a trust but verify approach. Using the built in parser package you could easily walk your repositories and verify imports against a white list. Having said that, I also encourage people to minimize dependencies. With Go I find very little need to use 3rd party packages often due to the completeness of the standard library. I'm not suggesting that you always reinvent the wheel, but I find myself thinking twice before pulling in 3rd party code.</p></pre>martingx: <pre><p>We chose to licence our code under similar open source licences (GPL compatible) so it doesn't matter so much.</p></pre>jfurmankiewicz: <pre><p>Well, about 0.001% of commercial companies would agree to that...so not really an option.</p></pre>martingzz: <pre><p>It wasn't an option at my company until pretty recently, now it's the default for many of us in the company. Huge numbers of companies release their stuff openly these days.</p>
<p>Since things like github, circleci, dockerhub etc are all free to use for open source, it's a great way to save money on running similar infrastructure yourself and that all helps convince companies that writing as open source is a worthwhile tradeoff.</p>
<p>Of course it all depends on the nature of your work. I understand that for certain types of company it might not make as much commercial sense still.</p></pre>Veonik: <pre><p>IMO, companies should use permissive licenses like Apache, BSD, or MIT. Releasing code under GPL makes it a pain in the ass to use in further commercial products, which means it probably just wont get used in commercial products.</p>
<p>I would also argue that GPL is not "open", but we'll save that for another day.</p></pre>
