<p>I'm struggling to find a way to capture in realtime the output of a sub process of a process launched via exec.Cmd</p>
<p>The scenario is I'm launching youtube-dl from Go. I'm telling youtube-dl to use aria2 to download the content - that launches another process (a sub process of youtube-dl) which outputs the download progress. I want to capture the aria2 progress in realtime. What I find is the youtube-dl output shows up at cmd.Stdout in realtime but the aria2 output is delayed until the youtube-dl process has finished.</p>
<p>Here's a link to my code: <a href="https://github.com/porjo/ytdl-web/blob/stdouterr/command.go#L18" rel="nofollow">https://github.com/porjo/ytdl-web/blob/stdouterr/command.go#L18</a></p>
<p>Any advice appreciated.</p>
<hr/>**评论：**<br/><br/>le_hohoho: <pre><p>Just a wild guess: Scanner.Scan() scans for "newline-delimited lines of text". The aria2 output may be doing something else?</p></pre>jerf: <pre><p>The other thing that leaps to mind is that the program may be emitting on standard error. A lot of programs aren't too clear on the distinction.</p></pre>porjo38: <pre><p>Thanks, I checked that - youtube-dl + aria2 is definitely outputting to stdout.</p></pre>porjo38: <pre><p>Thanks, that's a good thought. When I run youtube-dl directly from command line and capture the output, I see that aria2 is outputting it's progress on the same line (i.e. using carriage returns). However, that doesn't seem to be the case for the output capture by Go. When it finally gets the output, all the progress outputs are on new lines...</p>
<p>I've modified my code to split on either '\n' or '\r' (in that order of preference) but that hasn't helped.</p></pre>fakintheid: <pre><p><a href="https://github.com/qbecker/goprocess" rel="nofollow">https://github.com/qbecker/goprocess</a></p>
<p>This little package does what you’re asking for. </p></pre>tuxlinuxien: <pre><p>Not a package but a snippet from my point of view. Passing that code through gofmt would be great.</p></pre>epiris: <pre><p>If I'm understanding correctly your process tree is <strong>ytdl-web -> youtube-dl -> aria2</strong> - If so start by making your code simply be this:</p>
<pre><code>cmd := exec.Command(command, flags...)
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
</code></pre>
<p>If that works, you can replace the os.Stdout with a os.Pipe() and give the writer to the cmd.Stdout and read from it however you like, it's thread safe. Finally change exec.Command to exec.CommandContext so you don't have to select or scan, this simplifies things to:</p>
<pre><code>func RunCommand(ctx context.Context, w io.WriteCloser, command string, flags ...string) error {
defer w.Close() // this will unblock the reader
cmd := exec.CommandContext(ctx, command, flags...)
cmd.Stdout = w
cmd.Stderr = os.Stderr // or set this to w as well
return cmd.Run()
}
r, w := io.Pipe()
defer r.Close()
go func() {
// you can use a scanner on R here, send to a chan or print, doesn't matter.
} ()
err := RunCommandCh(ctx, w, flags...) // handle err
</code></pre>
<p>If at some point it doesn't work make sure that when you run the youtube-dl command directly it works to rule out youtube-dl doing any buffering.</p></pre>porjo38: <pre><p>Thanks! I'm not sure how I missed CommandContext()! I've incorporated those ideas into my code, but unfortunately still no luck.</p>
<p>I found a post on stackoverflow with someone having a similar issue with PHP, but not sure if that bears any relevance to the Go context:</p>
<p><a href="https://stackoverflow.com/questions/20723319/reading-stdout-from-aria2c-in-php-script" rel="nofollow">https://stackoverflow.com/questions/20723319/reading-stdout-from-aria2c-in-php-script</a></p></pre>epiris: <pre><p>You need to create smaller abstraction here and write a unit test that fetches a local dummy file or something. You also need to see if it works with the pipe just being set to stdout like I previously suggested. When something doesn’t work, remove code until it works, then add code until it breaks and you find your problem. </p></pre>MonkeeSage: <pre><p>Sounds like an issue with stdout buffer not being flushed and because youtube-dl is calling the program <a href="https://github.com/rg3/youtube-dl/blob/master/youtube_dl/downloader/external.py#L98" rel="nofollow">through a buffered pipe</a> you don't see the output until the buffer is flushed after aria terminates.</p>
<p>Turning off buffering using <code>stdbuf -o0 youtube-dl ...</code> might work, but I'm not sure that will actually turn off the buffering on the pipe when youtube-dl spawns the aria child process.</p></pre>porjo38: <pre><p>Thanks, I'm becoming more convinced that's what's going on here.</p>
<p>I've swapped aria2c for axel and can now capture the output in realtime. That would suggest the issue isn't with youtube-dl, but with aria2c itself.</p></pre>
