Is GAE a viable platform for my application? (if not, what would be a better option?)?

The time of the upload is not considered part of the script execution time, so no worries there.

The time of the upload is not considered part of the script execution time, so no worries there. Google App Engine is very good to perform a vast number of smaller jobs but not so much to do complex long running background jobs (because of the 30 sec limit + even smaller database connection time limit). So probably GAE would be a very good platform to GATHER the data but not for actually ANALYZING it.

You probably would like to separate these two.

Andris thanks for your help. Now it clear to me that uploading part is not an issue. The analysis part is going to be an issue for sure.

If we end up collecting, let's say, a few million records, searching within them might take longer than 30 seconds. How can I separate out collection and analysis parts? You mean transfer the data to EC3 or something like that for analysis?

Appreciate your clarification! – greppz Jul 11 '10 at 1:42 I am doing something similar to this but on a small level (0.5 million records daily). If you know all your analysis logic in advance, you can create some more tables for summary in parallel so that you need not to look detail data.

If it sounds good I can give you an example. – Manjoor Jul 13 '10 at 14:18.

We went ahead an implemented the first version on GAE anyway. The experience has been very much what is described here carlosble.com/?p=719 For a proof-of-concept prototype, what we have built so far is acceptable. However, we have decided not to go with GAE (at least in its current shape) for the production version.

The pains somewhat outweigh the benefits in our case. The problems we faced were numerous. Unlike my experience dealing with J2EE stacks, when you run into an issue, many a times it is a dead end.

Workarounds are very complicated and ugly, if you can find one. By writing good prototypes one could figure out whether GAE is right for the solution being built, however, the hype is a problem. Many newcomers are going get overly excited about GAE due to its hype and end up failing badly.

Because they will choose GAE for all kinds of purpose that it is not suitable for.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.