Giving access to shared memory after child processes have already started?

Your problem sounds like a perfect fit for the posix_ipc or sysv_ipc modules which expose either the POSIX or SysV APIs for shared memory, semaphores, and message queues. The feature matrix there includes excellent advice for picking amongst the modules he provides.

Your problem sounds like a perfect fit for the posix_ipc or sysv_ipc modules, which expose either the POSIX or SysV APIs for shared memory, semaphores, and message queues. The feature matrix there includes excellent advice for picking amongst the modules he provides. The problem with anonymous mmap(2) areas is that you cannot easily share them with other processes -- if they were file-backed, it'd be easy, but if you don't actually need the file for anything else, it feels silly.

You could use the CLONE_VM flag to the clone(2) system call if this were in C, but I wouldn't want to try using it with a language interpreter that probably makes assumptions about memory safety. (It'd be a little dangerous even in C, as maintenance programmers five years from now might also be shocked by the CLONE_VM behavior. ) But the SysV and newer POSIX shared memory mappings allow even unrelated processes to attach and detach from shared memory by identifier, so all you need to do is share the identifier from the processes that create the mappings with the processes that consume the mappings, and then when you manipulate data within the mappings, they are available to all processes simultaneously without any additional parsing overhead.

The shm_open(3) function returns an int that is used as a file descriptor in later calls to ftruncate(2) and then mmap(2), so other processes can use the shared memory segment without a file being created in the filesystem -- and this memory will persist even if all processes using it have exited. (A little strange for Unix, perhaps, but it is flexible. ).

Aha! That sounds doable. I'll give it a go and get back to you.

Thanks! (I've been looking for a python module that exposes shm_open and friends, and all I found was shm which looked a little stale). – Shawn Chin Sep 16 at 7:17 You post led me to a solution.

Many thanks. – Shawn Chin Sep 16 at 15:39.

Disclaimer: I am the author of the question. I eventually used the posix_ipc module to create my own version of RawArray. I used mainly posix_ipc.

SharedMemory which calls shm_open() under the hood. My implementation (ShmemRawArray) exposes the same functionality as RawArray but required two additional parameters - a tag to uniquely identify the shared memory region, and a create flag to determine if we should be created a new shared memory segment or attach to an existing one. Here's a copy if anyone's interested: https://gist.github.Com/1222327 ShmemRawArray(typecode_or_type, size_or_initializer, tag, create=True) Usage notes: The first two args (typecode_or_type and size_or_initializer) should work the same as with RawArray.

The shared array is accessible by any process, as long as tag matches. The shared memory segment is unlinked when the origin object (returned by ShmemRawArray(..., create=True)) is deleted Creating an shared array using a tag that currently exists will raise an ExistentialError Accessing a shared array using a tag that doesn't exist (or one that has been unlinked) will also raise an ExistentialError A SSCCE (Short, Self Contained, Compilable Example) showing it in action. #!

/usr/bin/env python2.7 import ctypes import multiprocessing from random import random, randint from shmemctypes import ShmemRawArray class Point(ctypes. Structure): _fields_ = ("x", ctypes. C_double), ("y", ctypes.

C_double) def worker(q): # get access to ctypes array shared by parent count, tag = q.get() shared_data = ShmemRawArray(Point, count, tag, False) proc_name = multiprocessing. Current_process(). Name print proc_name, "%.3f %.3f" % (d.

X, d. Y) for d in shared_data if __name__ == '__main__': procs = np = multiprocessing. Cpu_count() queue = multiprocessing.Queue() # spawn child processes for I in xrange(np): p = multiprocessing.

Process(target=worker, args=(queue,)) procs. Append(p) p.start() # create a unique tag for shmem segment tag = "stack-overflow-%d" % multiprocessing. Current_process().

Pid # random number of points with random data count = randint(3,10) combined_data = Point(x=random(), y=random()) for I in xrange(count) # create ctypes array in shared memory using ShmemRawArray # - we won't be able to use multiprocssing.sharectypes. RawArray here # because children already spawned shared_data = ShmemRawArray(Point, combined_data, tag) # give children info needed to access ctypes array for p in procs: queue. Put((count, tag)) print "Parent", "%.3f %.3f" % (d.

X, d. Y) for d in shared_data for p in procs: p.join() Running this results in the following output: me@home$ . /shmem_test.

Py Parent '0.633 0.296', '0.559 0.008', '0.814 0.752', '0.842 0.110' Process-1 '0.633 0.296', '0.559 0.008', '0.814 0.752', '0.842 0.110' Process-2 '0.633 0.296', '0.559 0.008', '0.814 0.752', '0.842 0.110' Process-3 '0.633 0.296', '0.559 0.008', '0.814 0.752', '0.842 0.110' Process-4 '0.633 0.296', '0.559 0.008', '0.814 0.752', '0.842 0.110'.

I think you are looking for mmap module concerning the serializiation of data this question answer of course if you hope to avoid copy I have not the solution EDIT in fact you can use the non stdlib _mutliprocessing module in CPython 3.2 to have the address of the mmap object and use it with from_address of a ctypes object it is what in fact what does RawArray in fact of course you should not try to resize the mmap object as the address of mmap may change in this case import mmap import _multiprocessing from ctypes import Structure,c_int map = mmap. Mmap(-1,4) class A(Structure): _fields_ = ("x", c_int) x = _multiprocessing. Address_of_buffer(map) b=A.

From_address(x0) b. X = 256 >>> map0:4 '\x00\x01\x00\x00' to expose the memory after the child is created you have to map your memory with a real file that is calling map = mmap. Mmap(open("hello.

Txt", "r+b").fileno(),4).

Unless I'm mistaken, that would require me to serialise my data and then load it in the child process. That's what I'm trying to avoid. – Shawn Chin Sep 14 at 16:08 As I understand the problem you try to avoid broadcasting the messages not really avoid serialization – Xavier Combelle Sep 14 at 18:23 I'm concerned about performance and memory usage.

Having to deserialise the same data on all procs does not sound appealing, hence my comment about serialisation. – Shawn Chin Sep 14 at 18:27 Thanks for the update Xavier. I'm afraid I don't understand how implementing my own mmap module is going to help in my case.

– Shawn Chin Sep 15 at 11:03 re your updates. That's pretty much what's being done under the hood of multiprocessing.sharedctypes. RawArray, and that all works as advertised.

I'm trying to expose shared memory to child processes after they have already been spawned. – Shawn Chin Sep 15 at 14:09.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.