Is it important to set some sort of timer to avoid detection when web scrapping?

I agree with Niklas. However, if you need the data 'faster' I would go with a timeout of 60 (up to 120) seconds. That is good for most of the servers today with the traffic size that you describe.

Also, to be on the good side of things, please make sure you are following the robots. Txt definition and see if there is some limit there (in terms of timeouts and routes).

Robots. Txt? What is that?

– Diskdrive Jun 5 at 3:44 Web site owners use the /robots. Txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. You can read more: robotstxt.Org/robotstxt.

Html – Ido Green Jun 5 at 4:01 Thanks, it turns out that the robots. Txt file doesn't have any instruction about delay. I'll probably just set it like you say to about 60 seconds.

– Diskdrive Jun 7 at 6:03.

If they would be on the lookout for scrapers, it would most definitely stand out. With 10000-20000 hits per day, it would average to about one hit per 4 to 9 seconds. You'd be pushing ~2 hits in-between every real request, and with such short intervals, it wouldn't be difficult to filter out your requests.

A lot safer and polite thing to do would be to set the scraping to be done through the whole 24 hours, so putting the interval up to ~10 minutes. It won't bring them significant load differences (not that 150 requests should do anyway), and it would make it significantly harder to pin point as the requests become far more spread.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.