MongoDB: Aggregate Data Using Map/Reduce?

The typical process for writing a map-reduce job is to start with the data format you want as the output of your reduce, build a map function that outputs it, then a reduce function that adds them up. In your example, you'd do something like this.

The typical process for writing a map-reduce job is to start with the data format you want as the output of your reduce, build a map function that outputs it, then a reduce function that adds them up. In your example, you'd do something like this: function map() { var date = new Date( this.timestamp.getFullYear(), this.timestamp.getMonth(), this.timestamp.getDay() ); var out = { regions: {}, browsers: {}, referers: {} }; out. Regions this.visitor.region.

Country_code = 1; out. Browsers this.visitor.browser. Name = 1; out.

Referers this.referer. Host = 1; emit( date, out); } function reduce( key, values ) { var out = { regions: {}, browsers: {}, referers: {} }; values. ForEach(function(value) { for( var region in value.

Regions ) { if( out. Regionsregion ) { out. Regions region += valueregion; } else { out.

Regions region = valueregion; } }; for( var browser in value. Browsers ) { if( out. Browsersbrowser ) { out.

Browsers browser += valuebrowser; } else { out. Browsers browser = valuebrowser; } }; for( var referer in value. Referers ) { if( out.

Referers referer ) { out. Referers referer += valuereferer; } else { out. Referers referer = valuereferer; } } }); return out; } At the end of this, you should have an output collection that looks something like this: { _id: "Sun Mar 13 2011 12:23:58 GMT-0700 (PDT)", value: { regions: { "US" : 235, "CA" : 212, "JP" : 121 }, browsers: { "IE" : 145, "Firefox" : 245, "Chrome" : 95, "Other" : 120 }, referers: { "www.google.com" : 24, "yahoo.Com" 56 } } } Note that there's another way you can do this.. Rather than doing a map reduce job, you can also keep all of this data in real time using atomic increments and upserts.

For example, each time you generate one of your page view docs, you could also do an update like this: db.pageviews.summaries. Update( { _id: new Date( this.timestamp.getFullYear(), this.timestamp.getMonth(), this.timestamp.getDay() ) }, { $inc : { 'visitor.region.US' : 1, 'visitor.browser. IE' : 1, 'referer.

Www.google.Com' : 1 } }, true // upsert ); This would mean that you always have your summary document up to date and don't need any map reduce jobs. Note, you might want to escape the '. ' in your domain names as Mongo will interpret that as a hierarchy of documents rather than an attribute name.

You're almost certainly right about upserts as I go along. Doing a map/reduce operation on 15 million documents would take FOREVER. Thank you for the map/reduce example though.

The concept makes more sense now. Map = build key value pair for each document, reduce = aggregate all those key value pairs. – mellowsoon Mar 13 at 22:00 Extra note, I think I'll need to look into a finalize function.

After all, I wouldn't need every country that visited on a particular day. Only the top 20. Is that where and how a finalize function is used?

– mellowsoon Mar 13 at 22:12 Well, you can do this in a finalize(), but it would be somewhat inefficient because you'd be doing a bunch of aggregations that you were just going to throw away when it's done. If there's a way to skip processing the unwanted countries in the first place, that would be preferable. But in general, yes you can do this kind of filtering in a finalize.

– jared Mar 21 at 23:54 Because I'd be saving the top 20 countries, there's no way to know which ones are unwanted until after the aggregation. – mellowsoon Mar 24 at 0:53.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.