Python capture stdout from subprocess line by line?

I only tested this on Windows, but it works in 2.6.6, 2.7.2, and 3.2.1.

I only tested this on Windows, but it works in 2.6.6, 2.7.2, and 3.2.1: from __future__ import print_function from subprocess import PIPE, Popen from threading import Thread import sys try: from Queue import Queue, Empty except ImportError: from queue import Queue, Empty # python 3. X ON_POSIX = 'posix' in sys. Builtin_module_names def enqueue_output(out, queue): for line in iter(out.

Readline, b''): line = line. Decode(sys.stdout. Encoding) queue.

Put(line) out.close() def main(): p = Popen('c/main. Exe', stdout=PIPE, bufsize=1, close_fds=ON_POSIX) q = Queue() t = Thread(target=enqueue_output, args=(p. Stdout, q)) t.

Daemon = True # thread dies with the program t.start() #initially the queue is empty and stdout is open #stdout is closed when enqueue_output finishes #then continue printing until the queue is empty while not p.stdout. Closed or not q.empty(): try: line = q. Get_nowait() except Empty: continue else: print(line, end='') return 0 if __name__ == '__main__': sys.

Exit(main()) Output: Outputting: 0 Outputting: 1 Outputting: 2 ... Outputting: 9997 Outputting: 9998 Outputting: 9999 Edit: readline() will block until the program's stdout buffer flushes, which may take a long time if the data stream is intermittent. If you can edit the source, one option is to manually call fflush(stdout), or you can instead disable buffering using setvbuf at the start of the program. For example: #include int main() { setvbuf(stdout, NULL, _IONBF, 0); FILE* fh = fopen("output.

Txt", "w"); int i; for (i = 0; I.

Nice I think we're getting close. If I run everything as-is it works. However the actual C program I need to invoke outputs data very slowly (about 1 line per second).

If I add sleep(1); to the C loop it doesn't output anything, UNLESS I also add a fflush(stdout); I have a feeling that this will give someone smarter than me a clue as to what's wrong. Also let's assume I don't have access to the source for the C program in order to simply add fflush commands. – tanders12 Sep 30 at 23:10 @tanders12: Sorry, close but no cigar.

After adding the sleep(1) call to the C program I also get the problem with readline blocking. I tried setting bufsize=0 to disable buffering. I also tried adding a while loop in enqueue_ouput that calls out.flush(); out.

Seek(0, SEEK_END) until out.tell() is nonzero, but it busy loops until the C program is done. Then I can seek(0) and readline works, but that's a far cry from what you want. – eryksun Oct 1 at 0:25 It worked for me when I added out.flush() right after line = line.

Decode(sys.stdout. Encoding). I think one of the fundamental problems I had was that I already had a separate thread so that my GUI wouldn't lock up, but I needed ANOTHER thread within that one to handle stdout.

Would there be any way to handle reading stdout in one thread? I still don't really understand what's going on, which almost bugs me more than it not working. But it does work, so thanks for your help.

Also, What does close_fds do? The python docs just confused me more. – tanders12 Oct 1 at 0:42 Nevermind I was wrong.It's still not working unless I flush directly in the C code.

Any ideas? It seems like this should be incredibly simple. The output is being put in the stdout buffer.

Why can't I just pull it off the buffer and everyone be happy? Do you think there is a bug in the unbuffered mode? – tanders12 Oct 1 at 1:49 Ok my problem is simple: stdout is buffered, end of story.

Stackoverflow. Com/q/874815/943814.It looks like I can either flush it from the subprocess or use something like pexpect. S.

Lott you are correct this is definitely a duplicate several times over.My apologies. That said, I don't understand why normal terminals are able to read program output in real time without waiting for stdout to flush or for the program to exit. That might be worth asking if I can't find the answer.

– tanders12 Oct 1 at 2:22.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.