<p>I should be specific - this is only one of two ideas I have for what I need to do. Now, I might be able to recompile the program in the background whenever something new is added and it may be possible to do so quickly as long as I'm only adding a few new classes or even objects of an already existing class, but..... Basically, I need a way to do the equivalent of loading a new class in a VM and then calling it (but only after I've made sure its loaded and all that jazz) or else a way to make an object from text which actually has code in it based upon the text (in fact the text it would be getting would just be the code to put in 3 separate functions if that was possible). The problem is that while I'll have a program I hope to have always-on which transfers data and holds the other data I'm worried about, I need to be able to load arbitrary other data wrapped in a semi-arbitrary api.... hence why I need to be able to make an object with custom code in it or load already compiled code int eh minute when that other data is actively loaded into my program (because in reality that 'data' can be a full other program, a file, a data structure, basically anything). Like, maybe it would be possible for me to set it up so a bunch of the API functions get compiled at load time but then have them as basically singletons or closures or whatever (I'm super new to Golang..... just started looking into it today for a project that ideally needs to be able to keep track of potentially several hundred to tens of thousands of separate little objects of which a large amount can be actively wanting to run code at the same time) and then have a genericish class that just takes say ("function name", @function) pairs and then let's you call a method within it called say method(string, array) which then called the function which matches string with the array as inputs? </p>
<p>This might be silly, but I'm new to GO and used to running with languages that can do stuff dynamically and live if only through some old features of the JVM or because that's hwo the language was built (e.g. Common Lisp), but for this project I was really hoping to use GO, but a core idea is that it has other programs run as part of it then adds additional features to help do... things (sorry can't go into details), but Go is very good with the additional features bit and even the general wrapping, EXCEPT that I'm not sure I can get a program wrapping to work very well.... and I do have a backup plan (just let programs write their own API wrappers in whatever language and just have them be ready to accept me sending them a message in probably JSON or else a custom format which tells them whether we want a function to be run or are asking for a raw value then the relevant data depending upon the type of request).</p>
<p>EDIT: As part of this project I feel like we might end up making the equivalent of an interpreter-esque environment ot run GO code in in which case I guess our problem is solved, but I was hoping to make that a very minimal design (e.g. basically a program that just takes very elementary input in a DSL and returns an answer).</p>
<p>2nd EDIT: Would basically 'pickling' the relevant state data, recompiling entirely, then reloading that data into the new compiled version be fast enough to be viable? I do know GO is supposed to be pretty fast at this stuff so I'm legit considering that as an alternative option.</p>
<hr/>**评论：**<br/><br/>akarl: <pre><p>You could use the plugin feature since Go 1.8. Check out <a href="https://golang.org/pkg/plugin/" rel="nofollow">https://golang.org/pkg/plugin/</a></p></pre>Xeoncross: <pre><p>Anyone have links to real-world usage of this package? (github, bitbucket, gitlab, etc..)</p></pre>greut: <pre><p>Terraform uses its own tool for providers, <a href="https://github.com/hashicorp/go-plugin" rel="nofollow">https://github.com/hashicorp/go-plugin</a> probably because it predates the 1.8 release.</p></pre>zanotam: <pre><p>Well, it looks like I'm likely to use it as half-APi half-customFFI (the code it may be interfacing with and actually talking to may or may not be Go code, but from the outside it shouldn't matter just like any well written API shouldn't care much about the underlying implementation).</p></pre>epiris: <pre><p>Someone asked something similar a couple months ago, <a href="https://www.reddit.com/r/golang/comments/7qb74r/can_golang_package_source_with_no_imports_be/" rel="nofollow">my answer</a> is the same here: arbitrary code execution is never the right solution, ever. </p>
<p>Consider that the entire Go tool chain fetches libraries from the internet and compiles them to architecture specific executables for your os, without executing arbitrary code. Ignoring <a href="https://nvd.nist.gov/vuln/detail/CVE-2018-6574" rel="nofollow">CVE-2018-6574</a> of course since it was fixed, because arbitrary code execution is a security risk too high to accept. I would be glad to nudge you in a better direction, but so far you have only described the potential solutions so you would need to define your problem.</p></pre>zanotam: <pre><p>Unfortunately, arbitrary code execution is actually what I need.... technically I just need arbitrary API's and then I let other programs run their own code and spit back out what I need, but I either need a very complicated thing to restart without problems while loading in new API's or, more realistically, the ability to load in arbitrary code (well not truly arbitrary, but.... a decent amount of freedom in its form) is important. I can't really go into more detail, but it's not intended to be truly arbitrary code execution, at least.</p></pre>joushou: <pre><p>But again, you are focusing on the solution that you have thought up, and are asking only about how to implement this solution.</p>
<p>What we need to know is the <em>problem</em> that your stated solution is trying to solve. Knowing that, we would be able to suggest alternative solutions. You speak of "arbitrary data" and "arbitrary API", but this conveys no meaning. Are you building a database that others can manipulate? If so, what does it contain?</p>
<p>The solution you are suggesting sounds flat out conceptually <em>wrong</em>, and thus we would love to guide you in a better direction.</p></pre>zanotam: <pre><p>I'm about to submit a proposal to Google via official channels for funding and I'd rather not talk about stuff too much. In this case 'arbitrary' means 'decided by an arbitrary program as given permission to run by the user' so I feel 'arbitrary' is pretty accurate. I'm considering the possibility of switching to the second language of choice and transferring requests with JSON then switching to JS as needed.... but then I need to know how to wrap it as much as possible in Golang so as to provide the greatest degree of concurrency possible.</p></pre>titpetric: <pre><p>There are projects that provide an interpreter layer, to which you can expose a subset of what go provides. For example, if you wanted to expose only your APIs, you could do that, without giving access to packages like <code>os</code> or whatever else that could cause damage.</p>
<p><a href="https://github.com/mattn/anko" rel="nofollow">https://github.com/mattn/anko</a></p>
<p>It obviously has other caveats. The performance will not be great as there's a "virtual machine" running and interpreting this code. The second caveat is that the syntax will not exactly be the same as Go, but from what I see, this project provides at least somewhat compatible syntax overall. There are also projects that give you a javascript VM (a lot more complex than this), and there are other language interpreters available also.</p>
<p>I find that this can be a good tradeof between arbitrary code, vs. code that is basically sandboxed to access only the objects that you pass to it. Disclaimer: I haven't used the particular GH project in production, because I rarely have so high demands for arbitrary logic injection, to warrant inclusion of a VM.</p></pre>zanotam: <pre><p>Unfortunately, I may be able to choose a JS VM instead..... but..... the problem is that while things are meant to be sandboxed from each other, they're not meant to be sandboxed from danger like the os package, unfortunately. In fact, they might REQUIRE such access.... not something I have much of a choice in but I thank you for alternative options! </p></pre>epiris: <pre><p>No, it isn't what you need, it's what you have misguidedly chosen against the better interest of your place of employment. There is no way to "secure" a system designed to take arbitrary input, this is provable because arbitrary is undefined by definition. You may not check if undefined is well formed, therefore you may not sanitize your input.</p>
<p>Any other assumptions you may make to justify your poorly engineered solution may be easily rebuked, "only internal customers", "legally bound 3rd parties", <insert all other weak and impossible to audit counter measures here>. You are going to put your business at risk by implementing this system. Since fellow redditors and I have made you aware of this you have officially transitioned from ignorant to negligent so I hope you reconsider. Best of luck.</p></pre>zanotam: <pre><p>IN 2 or so months when implementation starts, I'm going to have as good as I can people working with me to make sure data is never shared in an undesired way so the real danger is going to be from how far we get in the project (there will be one period where our later security via custom VM shenanigans and our earlier security via only having a few known programs running alongside)...... But to a certain extent safety is not a concern here. Or, rather, the safety is going to be as much as myself and a few google engineers can come up with, but in the end the danger is going to be the end-user primarily and not us which is..... something that can never be avoided, but will be done as well as I can, I guess.</p></pre>epiris: <pre><blockquote>
<p>IN 2 or so months when implementation starts, I'm going to have as good as I can people working with me to make sure data is never shared in an undesired way so the real danger is going to be from how far we get in the project (there will be one period where our later security via custom VM shenanigans and our earlier security via only having a few known programs running alongside)...... But to a certain extent safety is not a concern here. Or, rather, the safety is going to be as much as myself and a few google engineers can come up with, but in the end the danger is going to be the end-user primarily and not us which is..... something that can never be avoided, but will be done as well as I can, I guess.</p>
</blockquote>
<p>Oh okay, you will be protected early in the project with "<strong>Very Good People®</strong>", and later on implement "<strong>Shenanigan Guard: Secure VM™</strong>" for a runtime container that is configured to use "<strong>Running Alongside Constraint: LIMITER INFUSION™</strong>" - or as most security experts in my field call it: RACLI. It's like Brocoli but it also tastes good to an unrefined palate. Good choices, good luck.</p></pre>zanotam: <pre><p>We aren't going to be perfectly safe. When allowing arbitrary code to run you simply can't guarantee safety..... think of the simple fact that viruses exist for OS's and this project is basically going to be implementing a lot of similar features but at a higher level while ignoring the low level concerns. Then I hope with our own VM we can fully isolate things so that the only way to cheat out of the sandbox is to go talk to the OS and ask for stuff which we cannot control. We aren't goign to be completely safe though and that was never the intent - the goal is to make it so that we have lots of go bits who are sandboxed and do not share any data they do not want to.... just like in Go itself where you choose what to expose to the public. </p></pre>Killing_Spark: <pre><p>The plugins will have to be other go binaries. But did you consider just executing the given binaries as subprocesses? You could then communicate over stdin/stdout with the subprocesses. That communication is of course slower than using a plugin but you would be completly language agnostic. </p></pre>zanotam: <pre><p>Yeah, I get that. I meant the GO binaries may end up going outside the boundaries of Go code.... it's going to be an annoying real world necessity for my case.</p></pre>zanotam: <pre><p>HOpefully that does get more support including Windows.... but it looks like it does work as I'd hope something would and is one possible fix for me. If it's not too complicated then my group can have someone work out how to make that work in cases it currently doesn't if we don't just use reflection instead. But it sounds like it kinda serves different purposes actually - we'd use reflection for one thing and plugins for the other.</p></pre>fnordstar: <pre><p>That was a hard read.</p></pre>zanotam: <pre><p>I'm.... not used to discussing technical problems.....</p></pre>dlsniper: <pre><p>It's unclear what problem you are actually trying to solve. You describe how you want to solve it but not what's the issue that leads to it.</p></pre>mrunkel: <pre><p>Yes, OP has fallen into the xyproblem. </p>
<p><a href="http://xyproblem.info/" rel="nofollow">http://xyproblem.info/</a></p></pre>zanotam: <pre><p>I wish I could and I understand why you would ask that, but..... not at the moment?</p></pre>dlsniper: <pre><p>You wrote a lot of text about how you want to solve the problem, but not what problem you have to solve.</p>
<p>As others have pointed out, to receive actual, useful, feedback/help, it would be better if you could describe the problem you are trying to solve.</p>
<p>For example, you can write something like this: I would like to create an application which has to process data from AWS Kinesis, Kafka, AMQP, S3, etc. This application needs to handle all messages, without losing anything. The message type is not known ahead of time, and I would like to be able to add new functionality to process new message types in a manner that allows the application not to skip messages.</p>
<p>At the moment, this is what I understand from your description, but I have to admit, I got bored of actually reading/paying attention to your question after the first 6-7 lines. Because of that, I actually cannot give you any advice on the issue, because not only am I guessing what problem you are facing, but also I'm having to deal with the additional complexity of trying to figure out if what I imagine to be your use-case is correct or not.</p>
<p>Maybe not everyone got bored as I did and someone gave you helpful advice; others may have just sent their thoughts on random guesses. That's why having a good, clear question helps others understand the problem you are trying to solve. Additionally, being forced to explain this concisely will force you to (re)think the problem and maybe even give you insight into it that you previously did not realize.</p>
<p>An additional problem with the description above is that you seem to already have a solution in mind, something like loading a plugin while running, which while possible in Go, it may not be a (good) solution at all for the problem. For example, with Kinesis/Kafka you can have checkpoints to be able to resume reading from them. With SQS/AMQP you can make sure that you are not processing any more messages as you shut down the application to replace it with the newer version and additionally you can increase the queue capacity for storing messages, etc. All of which can be possible solutions that don't necessarily involve the solution you present.</p>
<p>Hope this helps.</p></pre>zanotam: <pre><p>Thank you, but if you really want to see then go down to the bottom of the thread where I've started a subconversation where I explain more than I feel safe doing so, but whatever. Random redditors on <a href="/r/golang" rel="nofollow">/r/golang</a> are probably cool people I can mostly trust, yes? I hope so, because, well, my future is running on this problem and I can't afford to have it scooped from under me, but well, you can go to the bottom of the comments to the last reply and my reply to that to see more.</p></pre>dlsniper: <pre><p>You can explain a problem without having to actually explain the business details behind it. It requires a bit of effort but you should be the one that puts that effort in, as opposed to those that are trying to help you.</p></pre>zanotam: <pre><p>THis is a relatively new problem and it's tied pretty tightly to my business logic. To be honest I've gotten a lot out of just the first 2 or 3 answers already so I'm pretty good on things for the moment and can always ask again later.</p></pre>joushou: <pre><p>A few bits and pieces about writing more readable question posts:</p>
<ol>
<li>Focus on first describing the problem you are trying to solve as exact as you can, without any detours. <em>Then</em> describe the solution your are attempting, and <em>then</em> describe the why you are having trouble with that solution. </li>
<li>Avoid ellipsis ("..."). It has no place in a post like this. Period, comma, semicolon or em dash.</li>
<li>Use paragraphs whenever the subject changes slightly. That is, two newlines.</li>
</ol>
<p>Example, based roughly on your post (notice how short it is!):</p>
<p>I am building a product that should contain a datastore (explain data!) and some base functionality (explain functionality!), which has the capability to be <em>extended</em> with additional functionality (explain example additional functionality!), preferably at runtime.</p>
<p>How would one go about doing this in Go? I was imagining something similar to loading plugins with additional functionality. Is this doable, or are there alternative methods more suitable for this task?</p></pre>lw6c: <pre><p><a href="https://github.com/dearplain/goloader" rel="nofollow">https://github.com/dearplain/goloader</a>, but has some limits.
<a href="https://golang.org/pkg/text/template/" rel="nofollow">https://golang.org/pkg/text/template/</a>, not just a template, but also a little script engine.</p></pre>zanotam: <pre><blockquote>
<p><a href="https://github.com/dearplain/goloader" rel="nofollow">https://github.com/dearplain/goloader</a></p>
</blockquote>
<p>Interesting. We might have a use for that with our plan to use html5 for graphical stuff plus some javascript code mixed in potentially at points (possibly just JSON). Thank you!</p></pre>: <pre><p>[deleted]</p></pre>zanotam: <pre><p>Can I create a struct filled with functions or closures (yes, I get that a closure can probably create a function, at least in languages I'm used to that's my usual usage for it) at runtime? Since functions are just a type that a variable can hold, yeah? Like, I'm guessing you could tell I've thought about this problem in other languages which often uses reflection so I think I get what you're trying to do, but.... well, I might end up using this more often than I'd like as part of some sort of pickling system as well (or does such a thing already exist for Golang? Like is it possible for me to basically save basically every relevant pointer+type of that pointer then keep the data from being trash collected for the time it takes for me to read off the raw data into a slice I guess it's called where each part of the slice is of variable bytes holding the actual data and then later basicalily dump the data back into memory using the slice to dictionary array to recreate the pointers to the data which would be more or les sunchanged excpet in cases where there's an already existing function to pickle or backup or whatever an object and then recreate it?) I don't even know if that's a logical thing to do or remotely possible, but I THINK I need a way to basically... Okay, basically temporarily create two more or less perfect versions of the data which are both running an outdated version of my program and restart one then load the data back in and restart the other and load the data back in if at all reasonably possible. I know this is a separate question, but.... Well, basically if I'm given a bunch of objects is there a reasonable way to save them then reload them from there saved point if I keep track of what I saved at least as far as the type of what I saved and maybe tag or whatever I saved? I guess type changes are kinda confusing me along with whether I can just save an object as a value and then later resurrect it using just the value's data (seems like I should be able to, but who knows?) as long as I know its type so that I can take the value (which I can keep in only one version being updated at a time and send to the other as a value
) of everything relevant then reboot things with all the values turned back into the right type and working as if not rebooted (assuming the type doesn't get changed). Like I wouldn't do this with all data, just a certain subset, but it would be a hugely helpful to be able to update while 'live' by updating two synched versions (well, one by one the two versions which synch CERTAIN data in full and not other data which I won't reset when I reset anyways).</p></pre>: <pre><p>[deleted]</p></pre>joushou: <pre><p>I think that OP should be cautious about the dark path that they're heading down regardless of language. The suggested solution is likely a bad idea regardless, it's just <em>also</em> cumbersome in Go.</p></pre>snowywind: <pre><p>Is Go a hard requirement? (I.e. does a hard fought business contract go into breach if you don't use it?)</p>
<p>If yes: Add an interpreter for some scripting language to your Go binary. You can use a line like "We built this on the highly robust Go foundation and extended its capabilities using a highly tuned Lua engine." That's your "arbitrary code" running in an environment where you can control access permissions from Go.</p>
<p>Dynamic data structures aren't really a core strength in Go; not impossible, but against the idiomatic grain. Worst case, just transmit, store and manipulate your data structures in JSON, XML or some other serialized format.</p>
<p>If no: Go may not be the best tool if you really need eval() to solve your problem.</p>
<p>Yes or No: I will be another voice imploring you to take a few steps back. Take a few more steps back. Now tell us the actual business need you're trying to fulfill before you end up with a Rube Goldberg machine. We are all saying this because we've all seen this opera play out; some of us even starred in it. There be dragons where you're headed.</p></pre>zanotam: <pre><p>I really don't need a full eval(). Like, when the data request is actually being bounced back and forth there will be a set 'here ya go', 'go away', or 'does not compute' type of reply set....... but honestly I want to thank you for one for being one of the better answers. By the time we get further in the program the plan is to have more resources so we'll probably be working directly to help build a VM aimed primarily at our product, but designed for anyone to use Go with along with porting things so that Go can be run on mobile platforms. Basically, we're looking at an... enhanced Desktop Environment.... with special channels to allow programs to directly but safely interact with each other via data exchange and potentially method calls while providing additional.... and some additional stuff. God even saying this much is dangerous for me as this is kinda my key to a job and possibly a decent amount of money on top of that, but basically there will at least be a server which helps keep track of connections and what's running without modeling much data exchange definitely written in Go so it can hold as many copies as possible reasonably, a client doing the heavy lifting, and then a graphical layer. As you can tell it's an ambitious project if we don't get shut down early at like the IDE stage or whatever, but we'll see.... the need to provide communication between programs which can be written in arbitrary languages.... the way I plan on doing things is ala OOP where every program is like a separate package with an API but also some allowed access directly to the equivalent of fields.... and so I need an interface between an arbitrary program and my program so I'm thinking of wrapping everything in a go API type looking thing then having those api's make dispatches in JSON to the program who will do what is necessary to spit out JSON which they will be given the chance to potentially convert in their api and then shoot out in some pre-designed messaging format. This message will probably be deleted in a few days, but this thread should be less popular now so I can safely discuss it temporarily at least I guess.</p></pre>lonahex: <pre><blockquote>
<p>Basically, we're looking at an... enhanced Desktop Environment.... with special channels to allow programs to directly but safely interact with each other via data exchange and potentially method calls while providing additional.... and some additional stuff.</p>
</blockquote>
<p>DBus? <a href="https://www.freedesktop.org/wiki/Software/dbus/" rel="nofollow">https://www.freedesktop.org/wiki/Software/dbus/</a></p>
<p>If all you need it to allow program to talk to each other, why not have some sort of IPC between them instead of loading foreign code in a running program (that's what it sounds like to me). </p></pre>zanotam: <pre><p>I do have an IPC basically planned and that's what I thought I was basically talking about.... but if someone installs a new program I need a way to handle it. ideally I would load as little as possible so anytime a new program gets started up or loaded up from a frozen state. Basically, the problem is that the goal is to allow people to write fan-made wrappers for program communication and the like so a non-GO+JSON based IPC is..... unlikely to fulfill the requirements of my program. Basically, it's like an IPC but technically it allows people to programmatically connect even things that weren't intended to connect.</p>
<p>EDIT: I may be an idiot sorry I should have read your link. One of hte few people who knows about the project and is starting to become more invested just mentioned dbus.</p></pre>
