It depends very much on how fast your crawler is crawling. If you have a single-threaded crawler then you're not going to do much better on average than one page per second. So you can use a HashSet to store the URLs you've visited.
Or, if you want to save information about the URLs you've visited, you can use a Dictionary.
I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.