Database choice for 30000 x 5000 Big table, each item may have 100M in size?

I'm impressed Without knowing how you're going to query the database it is hard to say what would work, but a standard filesystem can handle 100-megabyte objects, duplicate objects can be handled with hard or soft links, 'sparse' entries just not populated, and 30,000 directories in a directory should be fine in ext3 with htree turned on. ( tune2fs option dir_index ) But perhaps your SAN vendor will have some strong opinions about what sorts of systems work well when you've scaled to 13 Petabytes -- I suggest talking with your system vendor's sales engineers, the sales engineers I have known have been scary good.

I'm impressed. Without knowing how you're going to query the database it is hard to say what would work, but a standard filesystem can handle 100-megabyte objects, duplicate objects can be handled with hard or soft links, 'sparse' entries just not populated, and 30,000 directories in a directory should be fine in ext3 with htree turned on.(tune2fs option dir_index.) But perhaps your SAN vendor will have some strong opinions about what sorts of systems work well when you've scaled to 13 Petabytes -- I suggest talking with your system vendor's sales engineers, the sales engineers I have known have been scary good.

Some items may have 100M, and some items may have 0K. My expression is not clear. Thanks.

– Wubin Qu Feb 11 at 9:26.

If you're really serious about this, your best bet is Cassandra. Can't help you much more though.

Thanks for your advice. – Wubin Qu Feb 11 at 9:23 "The largest production cluster has over 100 TB of data in over 150 machines. " I hope it can scale 130 times larger.. :) – sarnold Feb 11 at 9:31.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.