Speeding up templates in GAE-Py by aggregating RPC calls?

I have been in a similar situation. Instead of ReferenceProperty, I had parent/child relationships but the basics are the same. My current solution is not polished but at least it is efficient enough for reports and things with 200-1,000 entities, each with several subsequent child entities that require fetching You can manually search for data in batches and set it if you want Given the posts, fetches all the data the template will need # with just 2 key-only loads from the datastore.

Posts = get_the_posts() author_keys = Post.author. Get_value_for_datastore(x) for x in posts authors = db. Get(author_keys) city_keys = Author.city.

Get_value_for_datastore(x) for x in authors cities = db. Get(city_keys) for post, author, city in zip(posts, authors, cities): post. Author = author author.

City = city Now when you render the template, no additional queries or fetches will be done. It's rough around the edges but I could not live without this pattern I just described Also you might consider validating that none of your entities are None because db.get() will return None if the key is bad. That is getting into just basic data validation though.

Similarly, you need to retry db.get() if there is a timeout, etc (Finally, I don't think memcache will work as a primary solution. Maybe as a secondary layer to speed up datastore calls, but you need to work well if memcache is empty. Also, Memcache has several quotas itself such as memcache calls and total data transferred.

Overusing memcache is a great way to kill your app dead. ).

I have been in a similar situation. Instead of ReferenceProperty, I had parent/child relationships but the basics are the same. My current solution is not polished but at least it is efficient enough for reports and things with 200-1,000 entities, each with several subsequent child entities that require fetching.

You can manually search for data in batches and set it if you want. # Given the posts, fetches all the data the template will need # with just 2 key-only loads from the datastore. Posts = get_the_posts() author_keys = Post.author.

Get_value_for_datastore(x) for x in posts authors = db. Get(author_keys) city_keys = Author.city. Get_value_for_datastore(x) for x in authors cities = db.

Get(city_keys) for post, author, city in zip(posts, authors, cities): post. Author = author author. City = city Now when you render the template, no additional queries or fetches will be done.It's rough around the edges but I could not live without this pattern I just described.

Also you might consider validating that none of your entities are None because db.get() will return None if the key is bad. That is getting into just basic data validation though. Similarly, you need to retry db.get() if there is a timeout, etc. (Finally, I don't think memcache will work as a primary solution.

Maybe as a secondary layer to speed up datastore calls, but you need to work well if memcache is empty. Also, Memcache has several quotas itself such as memcache calls and total data transferred. Overusing memcache is a great way to kill your app dead.

).

The second code block is very helpful... I didn't think of zip and using it that way.. but the first block is actually a built in feature of the sdk... references are already cached after they're resolved once, so there's absolutely no need for that code.. – Sudhir Jonathan Jan 17 '10 at 6:59 You are correct. My code was copied from a similar situation not using ReferenceProperty. It would be nice to populate the property via some back-door rather than just blowing away the .

City attribute. But I believe that would work in a pinch. – JasonSmith Jan 17 '10 at 9:00 Hmm.. I am reading the code in google/appengine/ext/db/__init__.

Py in the SDK. It looks like a simple assignment works fine because it will call the ReferenceProperty's __set__() method. I will update the answer to be shorter and clearer.

– JasonSmith Jan 17 '10 at 9:05 Done :) Also I forgot to mention, I use itertools. Izip in my real code because I used to hit MemoryErrors from time to time. It's probably not necessary in general though.

– JasonSmith Jan 17 '10 at 9:09 Is the itertools version any faster? Why would there even be a difference? Zip seems like a very simple algo to me.

– Sudhir Jonathan Jan 17 '107 at 6:02.

Here's some great examples of pre-fetching... blog.notdot.net/2010/01/ReferencePropert....

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.