Php curl data scraping?

The specified options already make curl follow redirects. However, in the case of a long redirect chain, you may want to increase CURLOPT_MAXREDIRS You can use a packet dumper such as wireshark to check which requests are sent by curl. It may be simply a bug in the scraped website which causes it to redirect infinitely.

The specified options already make curl follow redirects. However, in the case of a long redirect chain, you may want to increase CURLOPT_MAXREDIRS. You can use a packet dumper such as wireshark to check which requests are sent by URL1 may be simply a bug in the scraped website which causes it to redirect infinitely.

Gauravstomar you can analyze the redirects a little faster with a web browser compared to wireshark if you are not familiar with wireshark. Just look at the http headers to see how many redirects are occuring. Then set CURLOPT_MAXREDIRS appropriatly – steampowered Oct 24 at 20:27 @JM.

There may be some specific anti-curl or JavaScript-required trickery involved. Setting CURLOPT_MAXREDIRS to, say, 1000, should be more than any browser accepts. – phihag Oct 24 at 20:30 actually they have blocked proxy access to there website, how it is possible?

– gauravstomar Oct 27 at 5:04 @gauravstomar Some proxies set an X-Forwarded-For header, and there are public proxy datbases. Are you certain that the proxy is the problem, and not curl? Oftentimes, changing the User-Agent header is enough to evade such silly blockades or misconfigurations.

– phihag Oct 27 at 9:07 1 @gauravstomar php is not really the best way to set up a proxy, although there are existing implementations. Basically, you use curl to set up a new HTTP request. However, it's way easier not to use php and use a regular proxy such as squid or apache with mod_proxy.

– phihag Oct 27 at 18:28.

I cant really gove you an answer,but what I can give you is a way to a solution, that is you have to find the anglde that you relate to or peaks your interest. A good paper is one that people get drawn into because it reaches them ln some way.As for me WW11 to me, I think of the holocaust and the effect it had on the survivors, their families and those who stood by and did nothing until it was too late.