Crash after launching ~40k goroutines

I'm trying to experiment with goroutines and launched ~40k goroutines when a POST request reaches my net/http server. This crashes the program without a stacktrace and lists the stack trace of the 40k goroutines.

This, as you can imagine, is gigantic console output - is there a way to quieten it? How do I start debugging what actually went wrong - the 40k goroutine outputs don't seem to be related to the error.

Reducing the number to 10k seems to make it work - but that leads me to think if I'm overrunning memory. Could this be the case? All 40k goroutines are writing to the same channel. How do I make it a buffered channel?

The log looks like this:
Max threads: 32
Listening for requests on port 8002
...
runtime/cgo: pthread_create failed: Resource temporarily unavailable
SIGABRT: abort
PC=0x3ba0432925

goroutine 0 [idle]:

goroutine 1 [IO wait]:
net.(*pollDesc).Wait(0xc208010140, 0x72, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010140, 0x0, 0x0)
        /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc2080100e0, 0x0, 0x7ffff7fa9b70, 0xc208090460)
    /usr/local/go/src/net/fd_unix.go:419 +0x40b
    net.(*TCPListener).AcceptTCP(0xc20807a020, 0x46a3de, 0x0, 0x0)
        /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc20807a020, 0x0, 0x0, 0x0, 0x0)

(followed by 200k lines of log)

---

评论：

klauspost:

You may be running out of file descriptors. You don't say which OS you are running, but that would be a place to start.

hayzeus:

Yeah -- I'd tend to agree. 

davecheney:

Can you put the entire message in a pastebin and post a link to it here, it should be pretty easy to figure out what went wrong. 

bourbondog:

I put it in the original question.

legec:

If I read the first line correctly, it mentions "cgo" and pthread_create.

Does your code call a linked C lirary, which spawns threads (not goroutines) ?

bonekeeper:

   EAGAIN A system-imposed limit on the number of threads was
          encountered.  There are a number of limits that may trigger
          this error: the RLIMIT_NPROC soft resource limit (set via
          setrlimit(2)), which limits the number of processes and
          threads for a real user ID, was reached; the kernel's system-
          wide limit on the number of processes and threads,
          /proc/sys/kernel/threads-max, was reached (see proc(5)); or
          the maximum number of PIDs, /proc/sys/kernel/pid_max, was
          reached (see proc(5)).

http://man7.org/linux/man-pages/man3/pthread_create.3.html

010a:

I'm very curious why there's a reference to runtime/cgo in your stacktrace. Were you using cgo to call C code from your goroutines? What version of Go are you using? What OS?

I'm not totally familiar with how the go compiler creates goroutines, but it doesn't make any sense that it would be calling pthread_create for each goroutine, as that's a standard pthread and it very much would run out of file descriptors after a few thousand, if that. Goroutines are supposed to be much lighter. 

I've tested a simple program (here) up to 10M goroutines and while it takes ~4 minutes to see any output as the main thread is starting up all those goroutines we do eventually start seeing the expected output. Go 1.4.2 on OSX. 

bom_d_van:

Usually, the problems causing the crash are printed at the beginning of the outputs.

hipone:

If it's an experiment, mind sharing code? Folks here wouldn't mind. 

fr4nk3n:

I think you are correct about running out of memory, similar issue here

davecheney:

This error is usually associated with either making a lot of outgoing network connections (requiring a dns lookup, which is handled by cgo by default on linux), or by not closing http request bodies.