A tour of Go

A Tour of Go(GO之旅)

Posted on Thursday, June 21, 2012.（投递于2012年6月21号星期四）
Last week, I gave a talk about Go at the Boston Google Developers Group meeting. There were some problems with the recording, so I have rerecorded the talk as a screencast and posted it on YouTube.(上星期我在波士顿谷歌开发者组织会议作了一次演讲，下面是提问记录，我把它放在了YouTube上)

Here are the answers to questions asked at the end of the talk.（下面是演讲后提问的答案）

Q. How does Go work with debuggers?（go是怎样和调试器工作的）

To start, both Go toolchains include debugging information that gdb can read in the final binaries, so basic gdb functionality works on Go programs just as it does on C programs.
(gdb像在c语言中一样为go提供调试信息)

We’ve talked for a while about a custom Go debugger, but there isn’t one yet.

Many of the programs we want to debug are live, running programs. The net/http/pprof package provides debugging information like goroutine stacks(协程栈), memory profiling（内存优化分析）, and cpu profiling（CPU优化分析） in response to special HTTP requests.（HTTP请求）

Q. If a goroutine is stuck（卡住） reading from a channel with no other references, does the goroutine get garbage collected（垃圾回收）?

No. From the garbage collection point of view, both sides of the channel are represented by the same pointer, so it can’t distinguish the receive and send sides. Even if we could detect this situation, we’ve found that it’s very useful to keep these goroutines around, because the program is probably heading for a deadlock. When a Go program deadlocks, it prints all its goroutine stacks and then exits. If we garbage collected the goroutines as they got stuck, the deadlock handler wouldn’t have anything useful to print except "your entire program has been garbage collected".

Q. Can a C++ program call into Go?（C++调用Go?）

We wrote a tool called cgo so that Go programs can call into C, and we’ve implemented support for Go in SWIG, so that Go programs can call into C++. In those programs, the C or C++ can in turn call back into Go. But we don’t have support for a C or C++ program—one that starts execution in the C or C++ world instead of the Go world—to call into Go.

The hardest part of the cross-language calls(跨语言调用) is converting between the C calling convention(调用惯例) and the Go calling convention, specifically with the regard to the implementation of segmented stacks(分段栈的实现). But that’s been done and works.

Making the assumption that these mixed-language binaries start in Go has simplified a number of parts of the implementation. I don’t anticipate(预期，期望) any technical surprises involved in removing these assumptions. It’s just work.

Q. What are the areas that you specifically are trying to improve the language?

For the most part, I’m not trying to improve the language itself. Part of the effort in preparing Go 1 was to identify what we wanted to improve and do it. Many of the big changes were based on two or three years of experience writing Go programs, and they were changes we’d been putting off(推迟) because we knew that they’d be disruptive(破坏的；分裂性的；制造混乱的). But now that Go 1 is out, we want to stop changing things and spend another few years using the language as it exists today. At this point we don’t have enough experience with Go 1 to know what really needs improvement.

My Go work is a small amount of fixing bugs in the libraries or in the compiler and a little bit more work trying to improve the performance of what’s already there.

Q. What about talking to databases and web services?

For databases, one of the packages we added in Go 1 is a standard database/sql package. That package defines a standard API for interacting(交互) with SQL databases, and then people can implement drivers that connect the API to specific database implementations like SQLite or MySQL or Postgres.

For web services, you’ve seen the support for JSON and XML encodings. Those are typically good enough for ad hoc REST services. I recently wrote a package for connecting to the SmugMug photo hosting API, and there’s one generic call that unmarshals(数据编出) the response into a struct of the appropriate type, using json.Unmarshal. I expect that XML-based web services like SOAP could be framed this way too, but I’m not aware of anyone who’s done that.

Inside Google, of course, we have plenty of services, but they’re based on protocol buffers, so of course there’s a good protocol buffer library for Go.

Q. What about generics? How far off are they?

People have asked us about generics from day 1. The answer has always been, and still is, that it’s something we’ve put a lot of thought into, but we haven’t yet found an approach that we think is a good fit for Go. We’ve talked to people who have been involved in the design of generics in other languages, and they’ve almost universally cautioned us not to rush into something unless we understand it very well and are comfortable with the implications. We don’t want to do something that we’ll be stuck with forever and regret.

Also, speaking for myself, I don’t miss generics when I write Go programs. What’s there, having built-in support for arrays, slices, and maps, seems to work very well.

Finally, we just made this promise about backwards compatibility with the release of Go 1. If we did add some form of generics, my guess is that some of the existing APIs would need to change, which can’t happen until Go 2, which I think is probably years away.

Q. What types of projects does Google use Go for? (谷歌用go干啥项目)

Most of the things we use Go for I can’t talk about. One notable exception is that Go is an App Engine language, which we announced at I/O last year. Another is vtocc, a MySQL load balancer used to manage database lookups in YouTube’s core infrastructure.

Q. How does the Plan 9 toolchain differ from other compilers?

It’s a completely incompatible toolchain in every way. The main difference is that object files don’t contain machine code in the sense of having the actual instruction bytes that will be used in the final binary. Instead they contain a custom encoding of the assembly listings, and the linker is in charge of turning those into actual machine instructions. This means that the assembler, C compiler, and Go compiler don’t all duplicate this logic. The main change for Go is the support for segmented stacks.

I should add that we love the fact that we have two completely different compilers, because it keeps us honest about really implementing the spec.

Q. What are segmented stacks?

One of the problems in threaded C programs is deciding how big a stack each thread should have. If the stack is too small, then the thread might run out of stack and cause a crash or silent memory corruption, and if the stack is too big, then you’re wasting memory. In Go, each goroutine starts with a small stack, typically 4 kB, and then each function checks if it is about to run out of stack and if so allocates a new stack segment that gets recycled once it’s not needed anymore.

Gccgo supports segmented stacks, but it requires support added recently to the new GNU linker, gold, and that support is only implemented for x86-32 and x86-64.

Segmented stacks are something that lots of people have done before in experimental or research systems, but they have never made it into the C toolchains.

Q. What is the overhead of segmented stacks?

It’s a few instructions per function call. It’s been a long time since I tried to measure the precise overhead, but in most programs I expect it to be not more than 1-2%. There are definitely things we could do to try to reduce that, but it hasn’t been a concern.

Q. Do goroutine stacks adapt in size?

The initial stack allocated for a goroutine does not adapt. It’s always 4k right now. It has been other values in the past but always a constant. One of the things I’d like to do is to look at what the goroutine will be running and adjust the stack accordingly, but I haven’t.

Q. Are there any short-term plans for dynamic loading of modules?

No. I don’t think there are any technical surprises, but assuming that everything is statically linked simplified some of the implementation. Like with calling Go from C++ programs, I believe it’s just work.

Gccgo might be closer to support for this, but I don’t believe that it supports dynamic loading right now either.

Q. How much does the language (投机；说明书；细则) say about reflection?

The spec is intentionally vague(故意含糊不清) about reflection, but package reflect’s API is definitely part of the Go 1 definition. Any conforming implementation would need to implement that API. In fact, gc and gccgo do have different implementations of that package reflect API, but then the packages that use reflect like fmt and json can be shared.

Q. Do you have a release schedule?

We don’t have any fixed release schedule. We’re not keeping things secret, but we’re also not making commitments to specific timelines.

Go 1 was in progress publicly for months, and if you watched you could see the bug count go down and the release candidates announced, and so on.

Right now we’re trying to slow down. We want people to write things using Go, which means we need to make it a stable foundation to build on. Go 1.0.1, the first bug release, was released four weeks after Go 1, and Go 1.0.2 was seven weeks after Go 1.0.1.

Q. Where do you see Go in five years? What languages will it replace?

I hope that it will still be at golang.org, that the Go project will still be thriving and relevant. We built it to write the kinds of programs we’ve been writing in C++, Java, and Python, but we’re not trying to go head-to-head with those languages. Each of those has definite strengths that make them the right choice for certain situations. We think that there are plenty of situations, though, where Go is a better choice.

If Go doesn’t work out, and for some reason in five years we’re programming in something else, I hope the something else would have the features I talked about, specifically the Go way of doing interfaces and the Go way of handling concurrency.

If Go fails but some other language with those two features has taken over the programming landscape, if we can move the computing world to a language with those two features, then I’d be sad about Go but happy to have gotten to that situation.

Q. What are the limits to scalability(可扩展性；可伸缩性；可量测性) with building a system with many goroutines?

The primary limit is the memory for the goroutines. Each goroutine starts with a 4kB stack and a little more per-goroutine data, so the overhead is between 4kB and 5kB. That means on this laptop I can easily run 100,000 goroutines, in 500 MB of memory, but a million goroutines is probably too much.

For a lot of simple goroutines, the 4 kB stack is probably more than necessary. If we worked on getting that down we might be able to handle even more goroutines. But remember that this is in contrast to C threads, where 64 kB is a tiny stack and 1-4MB is more common.

Q. How would you build a traditional barrier using channels?

It’s important to note that channels don’t attempt to be a concurrency(并发) Swiss army knife. Sometimes you do need other concepts, and the standard sync package has some helpers. I’d probably use a sync.WaitGroup.

If I had to use channels, I would do it like in the web crawler example, with a channel that all the goroutines write to, and a coordinator that knows how many responses it expects.

Q. What is an example of the kind of application you’re working on performance for? How will you beat C++?

I haven’t been focusing on specific applications. Go is still young enough that if you run some microbenchmarks(微基准测试) you can usually find something to optimize. For example, I just sped up floating point computation by about 25% a few weeks ago. I’m also working on more sophisticated( 复杂的) analyses for things like escape analysis and bounds check elimination(除去), which address problems that are unique to Go, or at least not problems that C++ faces.

Our goal is definitely not to beat C++ on performance. The goal for Go is to be near C++ in terms of performance but at the same time be a much more productive environment and language, so that you’d rather program in Go.

Q. What are the security features of Go?

Go is a type-safe and memory-safe language. There are no dangling pointers, no pointer arithmetic(指示字运算), no use-after-free(释放后使用) errors, and so on.
(标记--已读)
You can break the rules by importing package unsafe, which gives you a special type unsafe.Pointer. You can convert any pointer or integer to an unsafe.Pointer and back. That’s the escape hatch, which you need sometimes, like for extracting the bits of a float64 as a uint64. But putting it in its own package means that unsafe code is explicitly marked as unsafe. If your program breaks in a strange way, you know where to look.

Isolating this power also means that you can restrict it. On App Engine you can’t import package unsafe in the code you upload for your app.

I should point out that the current Go implementation does have data races, but they are not fundamental to the language. It would be possible to eliminate the races at some cost in efficiency, and for now we’ve decided not to do that. There are also tools such as Thread Sanitizer that help find these kinds of data races in Go programs.

Q. What language do you think Go is trying to displace?

I don’t think of Go that way. We were writing C++ code before we did Go, so we definitely wanted not to write C++ code anymore. But we’re not trying to displace all C++ code, or all Python code, or all Java code, except maybe in our own day-to-day work.

One of the surprises for me has been the variety of languages that new Go programmers used to use. When we launched, we were trying to explain Go to C++ programmers, but many of the programmers Go has attracted have come from more dynamic languages like Python or Ruby.

Q. How does Go make it possible to use multiple cores?

Go lets you tell the runtime how many operating system threads to use for executing goroutines, and then it muxes the goroutines onto those threads. So if you’ve written a program that has four or more goroutines executing simultaneously, you can tell the runtime to use four OS threads and then you’re running on four cores.

We’ve been pleasantly surprised by how easy people find it to write these kinds of programs. People who have not written parallel or concurrent programs before write concurrent Go programs using channels that can take advantage of multiple cores, and they enjoy the experience. That’s more than you can usually say for C threads. Joe Armstrong, one of the creators of Erlang, makes the point that thinking about concurrency in terms of communication might be more natural for people, since communication is something we’ve done for a long time. I agree.

Q. How does the muxing of goroutines work?

It’s not very smart. It’s the simplest thing that isn’t completely stupid: all the scheduling operations are O(1), and so on, but there’s a shared run queue that the various threads pull from. There’s no affinity between goroutines and threads, there’s no attempt to make sophisticated scheduling decisions, and there’s not even preemption.

The goroutine scheduler was the first thing I wrote when I started working on Go, even before I was working full time on it, so it’s just about four years old. It has served us surprisingly well, but we’ll probably want to replace it in the next year or so. We’ve been having some discussions recently about what we’d want to try in a new scheduler.

Q. Is there any plan to bootstrap Go in Go, to write the Go compiler in Go?

There’s no immediate plan. Go does ship with a Go program parser written in Go, so the first piece is already done, and there’s an experimental type checker in the works, but those are mainly for writing program analysis tools. I think that Go would be a great language to write a compiler in, but there’s no immediate plan. The current compiler, written in C, works well.

I’ve worked on bootstrapped languages in the past, and I found that bootstrapping is not necessarily a good fit for languages that are changing frequently. It reminded me of climbing a cliff and screwing hooks into the cliff once in a while to catch you if you fall. Once or twice I got into situations where I had identified a bug in the compiler, but then trying to write the code to fix the bug tickled the bug, so it couldn’t be compiled. And then you have to think hard about how to write the fix in a way that avoids the bug, or else go back through your version control history to find a way to replay history without introducing the bug. It’s not fun.

The fact that Go wasn’t written in itself also made it much easier to make significant language changes. Before the initial release we went through a handful of wholesale syntax upheavals, and I’m glad we didn’t have to worry about how we were going to rebootstrap the compiler or ensure some kind of backwards compatibility during those changes.

Finally, I hope you’ve read Ken Thompson’s Turing Award lecture, Reflections on Trusting Trust. When we were planning the initial open source release, we liked to joke that no one in their right mind would accept a bootstrapped compiler binary written by Ken.

Q. What does Go do to compile efficiently at scale?

This is something that we talked about a lot in early talks about Go. The main thing is that it cuts off transitive dependencies when compiling a single module. In most languages, if package A imports B, and package B imports C, then the compilation of A reads not just the compiled form of B but also the compiled form of C. In large systems, this gets out of hand quickly. For example, in C++ on my Mac, including <iostream> reads 25,326 lines from 131 files. (C and C++ headers aren't “compiled form,” but the problem is the same.) Go promises that each import reads a single compiled package file. If you need to know something about other packages to understand that package’s API, then the compiled file includes the extra information you need, but only that.

Of course, if you are building from scratch and package A imports B which imports C, then of course C has to be compiled first, and then B, and then A. The import point is that when you go to compile A, you don’t reload C’s object file. In a real program, the dependencies are usually not a chain like this. We might have A1, A2, A3, and so on all importing B. It’s a significant win if none of them need to reread C.

Q. How do you identify a good project for Go?

I think a good project for Go is one that you’re excited about writing in Go. Go really is a general purpose programming language, and except for the compiler work, it’s the only language I’ve written significant programs in for the past four years.

Most of the people I know who are using Go are using it for networked servers, where the concurrency features have something contribute, but it’s great for other contexts too. I’ve used it to write a simple mail reader, file system implementations to read old disks, and a variety of other unnetworked programs.

Q. What is the current and future IDE support for Go?

I’m not an IDE user in the modern sense, so really I don’t know. We think that it would be possible to write a really nice IDE specifically for Go, but it’s not something we’ve had time to explore. The Go distribution has a misc directory that contains basic Go support for common editors, and there is a Goclipse project to write an Eclipse-based IDE, but I don’t know much about those.

The development environment I use, acme, is great for writing Go code, but not because of any custom Go support.

If you have more questions, please consult these resources.

Comments? Please join the Google+ discussion.

Elazar Leibovich (4 years ago) Custom Go debugger. If you do that, please make my long time wish come true. Have the ability to put breakpoints on the AST, not (only) on the lines. It makes much more sense, and you want the AST information in the executable anyway.

Quote from real C code style guide: "never write "if (cond) action;" in a single line, how would you set a break point inside the if condition if you did?"

Mathieu Lonjaret (4 years ago) +Russ Cox a couple of typos, I think:
"...but happy to gotten to that situation."
"Go let’s you tell the runtime..."

Peter Fröhlich (4 years ago) +Elazar Leibovich has it right, breakpoints on AST nodes/instructions and not lines, that's how the old AmigaOberon did it and it's the right way to go. Lines are artifacts and only artifacts.

Russ Cox (4 years ago) Lines are an illusion, as is AST. I would like to know what it means to set a breakpoint on a line when instruction reordering in modern compilers often mixes the implementation of nominally sequential lines together. Also, the case that Elazar points out cannot happen in Go when you are using gofmt. :-)

Peter Fröhlich (4 years ago) I'd contend that if you're using a source-level debugger, instructions and therefore the AST are a better approximation of what's happening than lines. But of course your point is well taken when you're also interested in the lower-level stuff.

Elazar Leibovich (4 years ago) +Russ Cox  it is less of an illusion with -O0, and you always have upper and lower bounds (OK, it's somewhat more difficult with inlining, but still possible if you save debug information about optimizations, I think).

Russ Cox (4 years ago) +Mathieu Lonjaret Fixed, thanks.

Santiago Corredoira (4 years ago) Dynamic loading and a custom Go debugger would be amazing!

Sankar P (4 years ago) I am curios to use/see the custom mail reader that you wrote. Can you share the url for it and the other go projects that you have written ? Thanks.

Sankar P (4 years ago) Also, if possible, can you do a screencast explaining how to get and use acme ? I glanced at the PDF but watching an expert working on it will give a better understanding. Thanks.

JT Olds (4 years ago) Sankar: http://en.wikipedia.org/wiki/Acme_(text_editor) http://man.cat-v.org/plan_9/1/acme http://swtch.com/plan9port/

andrey mirtchovski (4 years ago) +Sankar P, apart from the links above you can find several "tutorial" videos about acme use on youtube. For example this is the first hit for "acme editor video":

Acme Editor Demo

Sankar P (4 years ago) Thanks guys

Russ Cox (4 years ago) Most of my Go code is at http://code.google.com/p/rsc; docs at go.pkgdoc.org/code.google.com/p/rsc. Since it's just my personal stuff, the documentation is not always complete, some of it might not build, and other of it might not run. But feel free to play.

Sankar P (4 years ago) +Russ Cox Thanks a lot

Brad Beveridge (4 years ago) I had a quick look & couldn't find an answer: You mention that segmented stacks cost a few instructions per function call to see if the stack needs to grow.  Can't you just hit a page fault handler to grow the stack & avoid any instruction overhead?  I assume that there is some other feature here that prevents this?

roger peppe (4 years ago) +Brad Beveridge If you did this, then goroutines would require more overhead as you'd need to set up some VM context every time a goroutine was created.

Ken Ferry (4 years ago) Cool. Will you also be speaking at Google I/O next week?

Russ Cox (4 years ago) +Brad Beveridge The OS uses a page fault handler to grow the thread stacks, but that requires pre-allocating enough address space for the largest possible stack. Relocating during the fault handler would be too hard, since it could happen at any time, not just on function entry. The nice thing about the Go segmented stacks is that it doesn't have a fixed limit. We've run 100,000 goroutines with tiny stacks and also single goroutines with multi-GB stacks.

Russ Cox (4 years ago) +Ken Ferry I won't be at I/O but lots of the Go team will be, and there are at least three talks.

Brad Beveridge (4 years ago) +Russ Cox Maybe I need to read up on how Go implements stacks - does a grow require potentially require reallocation & movement of the stack?  In that case, yeah page fault growing is out.  I assumed that segmented stacks implied that the stack could be non-contiguous.  
Also, I need to get over my ingrained assumption that more instructions is slower - that hasn't really been the case in years now! :)

Russ Cox (4 years ago) A few people have asked for the code from the talk. Here's the final crawler program in its entirety: http://play.golang.org/p/lZtjYkD2dI

Mikhail Strebkov (4 years ago) As an IDE fun, I really appreciate GoLangIDE ( http://code.google.com/p/golangide/ ) used in conjunction with GoCode tool (auto-completion - https://github.com/nsf/gocode ). Just wanted to mention that there is an alternative to GoClipse.

Ralph Corderoy (4 years ago) +Russ Cox wrote: Segmented stacks are something that lots of people have done before in experimental or research systems, but they have never made it into the C toolchains.

The ARM Procedure Call Standard uses a segmented stack and has done since the mid-80s when the ARM 1 and 2 came along from Acorn. The standard was used by the C, Modula 2, ..., compilers and assembly that wanted to interact. Or was the statement implicitly constrained, e.g. Plan 9 toolchain?

Russ Cox (4 years ago) The statement was constrained to mainstream C toolchains (and I don't consider Plan 9's one of those). Is this ARM procedure call standard used by, say, Linux on ARM? I've never come across it before.

Ralph Corderoy (4 years ago) It may have been a long time ago when Russell King first started ported it to ARM but not now, no. The APCS, as it's known, has been obsoleted by the AAPCS, Procedure Call Standard for ARM Architecture, and that doesn't support segmented stacks IIRC. infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ihi0042d/index.html

Jose Luis Vazquez (4 years ago) Hi Russ, 
What is your opinion on this idea for Go binary distribution on tight net and disk environments?

https://groups.google.com/forum/?fromgroups#!topic/golang-nuts/44646DrbqjE

Russ Cox (4 years ago) Since it's lowercase, fetch must be a local function, not something imported. It's just off the slide. I don't have it handy but it was something like:

func fetch(url string) {
    r, err := http.Get(url)
    if err != nil {
        log.Fatal(err)
    }
    io.Copy(ioutil.Discard, r.Body)
    r.Body.Close()
}

Guilherme Lino (4 years ago) the future

Rocky Bernstein (3 years ago) I was watching the demo video and noticed the cool "play" "run/kill/close" button. I'd like to use that code or code like that in https://code.google.com/p/go-play/. Is it available?

Rocky Bernstein (3 years ago) +Russ Cox
In other debuggers I've written, I've struggled with this too. What I do when the runtime supports it — in one case I've just beefed up the runtime to support this — is to allow breakpoints on a (virtual or real) PC address. This of course is in addition to the usual line number or function name. See https://github.com/rocky/rb-threadframe/wiki/Rubykaigi-2010-present-part3 The exposition there has a bit to be desired, but you'll get the idea.

And when showing a location in the debugger, I also include the PC along with the usual file/line info. Even though I know most programmers aren't interested all that much in the PC, just glancing at the value at a coarse level it is useful: If you find your self at the same file and line position but the PC is different, you are at different places. And chances are that the larger the PC is the further progressed on the line you are at.

But for those that do want to more more about the PC, I also offer a disassemble command.

Mike Williamson (3 years ago) Great talk, thanks!  Aside from the great capabilities of Go, I was also curious about the presentation itself:  what sort of app is being used for the talk?  It's clear the code is "live" on the screen, yet the screen has "sections", parts of which look like a standard "Powerpoint" style presentation, while other parts look like an interactive IDE.

Thanks

Russ Cox (3 years ago) +Rocky Bernstein +Mike Williamson The presentation was done with an early version of the 'present' tool. http://godoc.org/code.google.com/p/go.talks/present

Mike Williamson (3 years ago) Thank you very much!

Jeffrey Lui (3 years ago) Thanks for your great talk.

In your video, you mentions that  "世界" in "Hello, 世界!" is Japanese. But in fact, those are Chinese characters, as the Japanese language was heavily influenced by the Chinese language hundreds of years ago, borrowing tens of thousands of Chinese characters back then. Only Hiragana and Katakana, and some logograms, are truly Japanese inventions.

Caio Ribeiro Pereira (2 years ago) Hi! May I add this video into DevFreeCasts (http://caio-ribeiro-pereira.github.io/devfreecasts/)?
The site wont embed video, it only links to the original video url.
Thanks!

---