EventMachine read and write files in chunks?

You need to inject something in there to break it up.

Up vote 2 down vote favorite share g+ share fb share tw.

I'm using EventMachine and EM-Synchrony in a REST API server. When a receive a POST request with a large binary file in body I receive it in chunks, writing that chunks to a Tempfile, not blocking the reactor. Then, at some point, I need to read this file in chunks and write that chunks to a definitive file.

This is working, but it blocks the reactor as expected, and cant find a way to make it work without blocking. I call this function at some time, passing to it the tempfile and new file name: def self. Save(tmp_file, new_file) tmp = File.

Open(tmp_file, "rb") newf = File. Open(new_file, "wb") md5 = Digest::MD5. New each_chunk(tmp, CHUNKSIZE) do |chunk| newf checksum}) # only with the final md5 value end ruby eventmachine link|improve this question edited Jan 22 at 19:49 asked Jan 21 at 19:38João Pereira756 67% accept rate.

1 new is the name of a class method. So unless this is a module, that could be one bug. Change it to something like new_file and test it.

– Linux_iOS.rb.cpp.c.lisp.m. Sh Jan 21 at 19:52 Whell, its just in this code sample, I'm not using that variable names. – João Pereira Jan 22 at 12:44 OK.

I just thought that that might be the problem, as the smallest problems can show up as the biggest. – Linux_iOS.rb.cpp.c.lisp.m. Sh Jan 22 at 15:19.

You need to inject something in there to break it up: def self. Each_chunk(file, chunk_size=1024) chunk_handler = lambda { unless (file. Eof?

) yield file. Read(chunk_size) EM. Next_tick(&chunk_handler) end } EM.

Next_tick(&chunk_handler) end It's kind of messy to do it this way, but such is asynchronous programming.

Many thanks! I think this solves the read problem, but what about the write part (newf Also, I've found that the file is being successful copied but the #save returns the md5 variable after the first iteration, so the md5 being returned is the md5 of the first chunk, as it does not wait for the copy process completion. I've updated the first post (EDIT 1), so it should be more easy to help.

– João Pereira Jan 22 at 20:00 I don't know that there's a built-in way to channel read and write activity through the EventMachine handler. The best approach is to keep your blocking calls short or at least infrequent. In this case the OS should handle buffering of writes and insulate you from IO issues.

– tadman Jan 23 at 15:26.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.