Ruby on Rails memory leak when looping through large number of records; find_each doesn't help?

I was able to figure this out myself. There are two places to change.

Up vote 5 down vote favorite share g+ share fb share tw.

I have a Rails app that processes a large (millions) number of records in a mysql database. Once it starts working, its memory use quickly grows at a speed of 50MB per second. With tools like oink I was able to narrow down the root cause to one loop that goes through all the records in a big table in the database.

I understand if I use something like Person.all. Each, all the records will be loaded into memory. However if I switch to find_each, I still see the same memory issue.

To further isolate the problem I created the following test controller, which does nothing but looping through the records. I suppose find_each only keeps a small number of objects in memory each time, but memory use grows linearly as it executes. Class TestController Find_each do |person| end end I suspect it has to do with ActiveRecord caching the query results.

But I checked my environment settings and I do have all the caching related options set to false in development (I am using the default settings created by rails). I did some search online but couldn't find a solution. I am using rails 3.1.0 rc1 and ruby 1.9.2 Thanks!

Ruby-on-rails ruby-on-rails-3 activerecord memory-leaks link|improve this question asked Jul 12 '11 at 18:02WYi275128 75% accept rate.

I guess there is a function called find_in_batches in ActiveRecord. May be it will help control the memory outbreak. – rubish Jul 12 '11 at 18:38 I was thinking that too, however, it looks like find_each uses find_in_batches under the covers.

Maybe each individual row is large & can benefit from the :batch_size option (defaults to 1000 rows) – Brian Jul 12 '11 at 18:49 What is the code actually doing that it needs to loop through every record? – Maran Jul 12 '11 at 19:50.

I was able to figure this out myself. There are two places to change. First, disable IdentityMap.

In config/application. Rb config. Active_record.

Identity_map = false Second, use uncached to wrap up the loop class MemoryTestController Hope this helps other people.

As nice as ActiveRecord is, it is not the best tool for all problems. I recommend dropping down to your native database adapter and doing the work at that level.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.