Prevent OutOfMemory when using java.nio.MappedByteBuffer?

Maybe a WeakHashMap to pool those MappedBuffers would work.

Up vote 1 down vote favorite share g+ share fb share tw.

Consider application, which create 5-6 threads, each thread in cycle allocate MappedByteBuffer for 5mb page size. MappedByteBuffer be = ch. Map(FileChannel.MapMode.

READ_ONLY, r, 1024*1024*5); Sooner or later, when application works with big files, oom is thrown java.io. IOException: Map failed at sun.nio.ch.FileChannelImpl. Map(FileChannelImpl.

Java:758) Caused by: java.lang. OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl. Map0(Native Method) at sun.nio.ch.FileChannelImpl.

Map(FileChannelImpl. Java:755) According to specification, MappedBuffer should dispose direct memory as soon as it is GC itself. Looks like the problem is, that MappedBuffer-s are GC-ed too late, later then direct memory finished.

How to avoid this situation? Probably say MappedBuffer to dispose implicitly or use some kind of pool of MappedBuffer java nio outofmemoryerror bytebuffer filechannel link|improve this question asked Dec 18 '11 at 16:41aauser443616 74% accept rate.

– fge Dec 18 '11 at 17:14 The exception is NOT OOM but IOException. You are running out of virtual address space. Show quite a bit more code.

Mapped buffers and reclamation in java is a long time issues (still unresolved elegantly) – bestsss Dec 19 '11 at 6:22.

Maybe a WeakHashMap to pool those MappedBuffers would work. But before you guess about the root cause, I'd recommend hooking your app up to Visual VM 1.3.3, with all the plugins installed, so you can see exactly what's causing the OOM error. You're presuming that these MappedBuffers are doing it, but they're only 5MB each for 5-6 threads - 25-30MB total.

Better to have data than guess. Visual VM will get it for you.

Well actually, each thread can create a lot of buffer in cycle, depends on file size. This mean, that the total amount of buffers are not limited. Probably I should use some kind of semaphore to prevent allocation of huge amount of memory.

Anyway I will try profiler, you suggested – aauser Dec 18 '11 at 16:53 Mapped buffers cannot be pooled! They are created to cover an address returned by the mapping virtual address to a location in file and loading data by page faults + writing back on file handle close or request. – bestsss Dec 19 '11 at 6:20 1 ...and WeakHashMap is totally useless/harmful since the key won't be the Buffer itself, hence it won't be even a GC-available before the WeakHashMap is expunged (internally).

Using a GC dependent structure for the problem is definitely wrong. – bestsss Dec 19 '11 at 6:27 Funny how bestsss has commented on every answer but offered nothing himself. Let's see your answer to the question.

– duffymo Dec 19 '11 at 10:13 I'd answer, I asked for more code. With the one currently provided there is no solution. It runs out of virtual memory and map may not succeed.

It's a hard one and usually requires design changes and thinking ahead. You can read my comment below the question. There is no offense towards your answer, yet any GC related solution is bound to be worse than the bug, itself.

GC just fails to reclaim the virtual memory (by garbaging all views of the address quickly enough) and if the explicit GC is disabled... – bestsss Dec 19 '11 at 12:41.

The error message says "map failed", not "heap space" or "permgen space". This means the JVM doesn't have enough address space available. See this bug in Sun's database, and also this question.

The first link provides a workaround (ewww) which is close the what the second link says: try { buffer = channel. Map(READ_ONLY, ofs, n); } catch (java.io. IOException e) { System.gc(); System.runFinalization(); buffer = channel.

Map(READ_ONLY, ofs, n); }.

Depending on the GC algorithms and its parameters the System.gc() call might be a NOP. Using -XX:+DisableExplicitGC effectively removes the ugly patch of catch the OOM and retrying after the GC request – bestsss Dec 19 '11 at 6:42.

You can avoid having to trigger a GC by cleaning up the mapped byte buffers directly. Public static void clean(ByteBuffer bb) { if(bb == null) return; Cleaner cleaner = ((DirectBuffer) bb).cleaner(); if(cleaner! = null) cleaner.clean(); } Provided you call this before discarding, you won't run out of virtual memory.

Perhaps you can look at creating larger ByteBuffers less often (unless you have a large number of files) Creating a MappedByteBuffer is not free (takes about 50 micro-seconds on some machines).

MappedBuffer should dispose direct memory as soon as it is GC itself It doesn't actually say that anywhere that I can see. There is a long-standing Bug Parade item that says it is never released. It does say this: It is therefore recommended that direct buffers be allocated primarily for large, long-lived buffers.

They are disposed for sure, the main issue is that a tiny java object holds reference to megabytes and the GC doesn't deem it anywhere urgent to releases and finalize/unmap the virtual memory. – bestsss Dec 19 '11 at 6:24 @bestsss Either they are 'disposed' or they aren't. Make up your mind.

You need to look up . It doesn't agree with you. – EJP Dec 19 '11 at 9:09 1 read the comment again.

They are disposed but too often much delayed. I know the bug well and the native impl on both Windows and Linux/Solaris. The bug doesn't tell the mapped buffers are not unmapped, they are AFTER the ByteBuffer (and its views) are garbage collected.

They just cannot be forcibly closed/unmapped via normal means. It has been always possible to hack around and unmap the Buffer risking SIGSEV or using another file, or any undefined result. – bestsss Dec 19 '11 at 9:32 the bug has no simple solution because an address cannot be atomically unmapped (+flush) and then remapped to NUL device.

OS providing such functionality would do the trick. Adding synchronizations would kill performance. If an OS provide m_null_unmap or something alike it'd do the trick.

That's it: marks the pages not owned by the process, flushes them and maps it back to null device. Any concurrent access will lead to page fault, so the OS can handle it. – bestsss Dec 19 '11 at 9:39.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.