Trying to locate a leak! What does anon mean for pmap?

Anon blocks are "large" blocks allocated via malloc or mmap -- see the manpages. As such, they have nothing to do with the Java heap (other than the fact that the entire heap should be stored in just such a block) In my experience, thread stacks also use anon blocks. If you see a lot of anon blocks that all have the same size, and that size is 512k to 4Mb (the example below is repeated over a dozen times for a Tomcat process that I have running), that's the likely cause.

Depending on the program, you may have up to a few dozen of these; if you're seeing thousands, it means you have a problem with threading b089f000 504K rwx-- anon b091d000 12K ----- anon b0920000 504K rwx-- anon b099e000 12K ----- anon b09a1000 504K rwx-- anon b0a1f000 12K ----- anon But that leaves a question: why are you using pmap to diagnose a Java memory issue?

Anon blocks are "large" blocks allocated via malloc or mmap -- see the manpages. As such, they have nothing to do with the Java heap (other than the fact that the entire heap should be stored in just such a block). In my experience, thread stacks also use anon blocks.

If you see a lot of anon blocks that all have the same size, and that size is 512k to 4Mb (the example below is repeated over a dozen times for a Tomcat process that I have running), that's the likely cause. Depending on the program, you may have up to a few dozen of these; if you're seeing thousands, it means you have a problem with threading. B089f000 504K rwx-- anon b091d000 12K ----- anon b0920000 504K rwx-- anon b099e000 12K ----- anon b09a1000 504K rwx-- anon b0a1f000 12K ----- anon But that leaves a question: why are you using pmap to diagnose a Java memory issue?

See this part of System Performance Tuning for anonymous memory.

That tells you how anonymous allocations are managed, not what it is. ALthough the link overall is quite a good one. – kdgregory Sep 27 '09 at 12:05.

Use Eclipse MAT (when you get OutOfMemoryExceptions in the Java Heap not the native heap).

I've seen that pattern before as a thread leak. If you have code that is trying to pool threads, but somehow messes up and leaks a thread, you get a pattern like that in nmap. I think each bit if memory is the minimum stack size for the thread, certainly it was nothing to do with heap in our case.

We still got OutOfMemoryErrors when we hit OS limits, even tho when we analise the heap it is not over allocated. When we had a problem like this nmap pid | grep -c 12K turned out to be the number of threads in use.

Something to understand about JVM memory characteristics: They are very greedy. It is pretty standard to include in the JVM startup the starting memory the process should take and the maximum memory that the process can take. However, what Java then tends to do (and this behavior will vary between version of the JVM) is consume the available memory until it is full, and then garbage URL1 may garbage collect earlier, but it won't be as aggressive, because it has more memory available.

When it does the "stop the world" garbage collection once it has hit max memory it can typically free up a lot of memory (50% is not unusual) however, the jvm doesn't give that memory back to the operating system. It just holds on to it until it needs it. There are some who claim that this isn't an issue because the operating system will simply put that memory in swap space on disk and it will not get in the way if the java process doesn't need it.

That has not been my observation. First, until the big garbage collection, the memory isn't really free, and can be quite fractured, making it impossible to effectively swap it away. And second, even after the garbage collection, I have seen at least on windows real OS memory starvation because of the JVM.

See this question for some advice on how to reclaim memory for the OS in such a scenario. I don't know much about pmap, but it is likely extra memory that the JVM either used and freed, or requested from the OS because it was running low but hasn't used yet.(The comment suggests I really don't know anything about pmap, which is entirely true, but I stand by the rest of the answer in regard to the underlying problem of "where my memory has gone for a java process running in linux.").

The Java heap should be just a few large blobs. These mappings will be outside the Java heap. – Tom Hawtin - tackline Sep 25 '09 at 16:25 @Tom, that might be true, but I was trying to address his underlying problem of "where my memory has gone for a java process running in linux" – Yishai Sep 25 '09 at 16:48 Tom do you have any idea where my memory has gone if the heap did not grow at all?

Is there some weird cases where the JVM will grow in memory without growing the heap? – erotsppa Sep 25 '09 at 18:05 @erotsppa, sure, the JVM can request more memory from the OS when it anticipates needing it, before the actual heap size increases. But the heap should be close to the allocated memory for that to happen if it doesn't actually increase right after.

– Yishai Sep 25 '09 at 19:30.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.