How to store a sparse queryable matrix on disk or database?

The first thing that came to my mind when reading the question heading was a database row per (x,y) as you suggested in your next to last paragraph The other thing to note is that databases often compress the rows, particularly for NULLs, so the straightforward representation may not waste as much space as you think.

The first thing that came to my mind when reading the question heading was a database row per (x,y) as you suggested in your next to last paragraph. The other thing to note is that databases often compress the rows, particularly for NULLs, so the straightforward representation may not waste as much space as you think.

Recently, I've become interested in key-value type storage, and came across this blog entry describing how FriendFeed uses MySQL to implement a similar requirement: bret.appspot.com/entry/how-friendfeed-us....

It depends on your definition of "many or most columns are null", but that sounds like a very reasonable approach, assuming that you really need the random access. If you can do everything by sequential processing (e.g. A scan in row order) then a flat file would be another reasonable option to consider.

The Intersystems Cache database uses structures internally for storing data, which are sparse multidimensional arrays. Maybe check that out. You can query it, and map it to SQL tables.

I'm not sure if you can directly access the multidimensional arrays in Intersystems Cache from java though.

If you only need to save the data as you say "on disk", read it back & make certain queries, you might want to consider simply serializing the class of your own design and choice. Eliminates all need for database infrastructure, but obviously also excludes many of the things that a database would actually help you with.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.