Proxy implementation

[ska-ibees]

Is proxy support currently present?

Well if not:

To avoid being blocked, how about adding something like this https://github.com/rejoiceinhope/scrapy-proxy-pool

[eliasdabbas]

Great suggestion. This will eventually need to be addressed.

Would you like to dig into the details and suggest how this can be merged into advertools.crawl?

[ska-ibees]

Sure thing!

[ska-ibees]

Well, after spending plenty of time digging into proxy implementation.

Below are the findings:

1. General Free proxies are not recommended becuase of performance and security issues.
   However, if someone wants to implement ip rotation then better to use high availability IP pools. I have tested https://github.com/SpiderClub/haipproxy/ and it serves the purpose so far.

2. Proxy implementation in advertools
   As of now, I have tested proxy within advertools by hard coding it.

def start_requests(self):
       meta = {"proxy": "http://127.0.0.1:3128"}
       for url in self.start_urls:
           try:
               yield Request(url, callback=self.parse, errback=self.errback,meta=meta)
           except Exception as e:
               self.logger.error(repr(e))

Link to file

I have been struggling to pass meta into adv.crawl . However, couldn't get it working without monkey-patching base spider class.
Can you please suggest any way to include proxy within adv.crawl?

[eliasdabbas]

The CrawlerProcess was my first option actually. If I remember correctly, its problem was that it needed to be run in a new session. The idea behind this structure is to allow crawling to run within a Jupyter notebook/lab environment, and have users continue their work normally. Since it had to be restarted (all variables deleted), then I thought it would make sense to just go with the standard Scrapy implementation and run the same way it is run from the command line, with all options provided as is.

I might be wrong, and if there is a good reason to re-write it, and if there are issues with the current implementation we can review them and consider other options.

I'll try the proxy/meta implementation and will let you know.

[ska-ibees]

I might be wrong, and if there is a good reason to re-write it, and if there are issues with the current implementation we can review them and consider other options.

One thing that's coming in mind is the lack of error reporting etc. (configuring the logging and setting shutdown handlers). I am unsure if we can catch any errors reported by the subprocesss. IMHO, it is something important within non-lab enviorment. Well, need further investigation to compare command line vs CrawlerProcess.

I'll try the proxy/meta implementation and will let you know.

Alright. (BTW, I tested with a free account on webshare.io )

[ska-ibees]

Well, i made several attempts to get errors/excemptions from the response but couldn't succeed.

Basically, I need to trigger some actions by having errors as call back.

One possibility would be to inherit/access:

[def errback(self, failure):
        if not failure.check(scrapy.exceptions.IgnoreRequest):
            self.logger.error(repr(failure))
            yield {'url': failure.request.url,
                   'crawl_time': datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S'),
                   'errors': repr(failure)}](https://github.com/eliasdabbas/advertools/blob/3ba5fe9a02965e1dc48cf23582217e3a5e111aaf/advertools/spider.py#L675)

Any idea about how to get it in return of adv.crawl() ?

[eliasdabbas]

One thing that's coming in mind is the lack of error reporting etc. (configuring the logging and setting shutdown handlers)

There are several settings for logging, like the log format, log file and log date format for example. Is that what you meant?
There are also a few CLOSESPIDER_* settings. Unless you meant something else?

Alright. (BTW, I tested with a free account on webshare.io )

I just tested it, and works fine. Great!
I think another thing can be to allow users to introduce several proxies, and each request choses a random proxy. Since meta is flexible.
What do you think?

Any idea about how to get it in return of adv.crawl() ?

You can trigger errback by using a wrong URL:

adv.crawl("doesnt_exit", "output_file.jl")

It will create an errors column in the output file with the error details.

[ska-ibees]

There are several settings for logging, like the log format, log file and log date format for example. Is that what you meant?
There are also a few CLOSESPIDER_* settings. Unless you meant something else?

Alright, logging is clear now. Thanks

I just tested it, and works fine. Great!
I think another thing can be to allow users to introduce several proxies, and each request choses a random proxy. Since meta is flexible.
What do you think?
IP reliability and type of proxy are critical factors here. So, I think let's keep it simple and generic and let proxy provider takes care of ip rotation. Like you see in case of webshare.io.

You can trigger errback by using a wrong URL:

adv.crawl("doesnt_exit", "output_file.jl")

It will create an errors column in the output file with the error details.

I see! So, the output_file.jl is going to be a single source of truth... :) Thanks for pointing it.

[ska-ibees]

Finally some good news!!!

@eliasdabbas Please see this PR

https://github.com/eliasdabbas/advertools/pull/202/files

[ska-ibees]

@eliasdabbas First of all forgive my ignorance that I didn't know much about you earlier. You don't know how much I feel honored to connect with you! Thanks for your contributions and contniued knowledge sharing.

[ska-ibees]

Just curious what you think about the multi proxy idea and randomization?

Do you have any ideas?

AFAIK, IP rotation/multi IP proxy is not supported within Scrapy core. However, it can be implemented using middleware. I have tested this one and it works fine Link

Main problem that I see in this approach is that you have to manually update list of proxy IPs. Also, in order to avoid being banned IP's repute is important. That's why I earliler suggested to keep IP rotation outside of the scrapy. IMHO using a proxy API service is would be better for production use.

[eliasdabbas]

@eliasdabbas First of all forgive my ignorance that I didn't know much about you earlier. You don't know how much I feel honored to connect with you! Thanks for your contributions and contniued knowledge sharing.

Thanks! Very kind of you. The honor is mine.

AFAIK, IP rotation/multi IP proxy is not supported within Scrapy core. However, it can be implemented using middleware. I have tested this one and it works fine Link

Yes, this middleware is more extensive and flexible than what we are planning. Incorporating any Scrapy middleware can be a future project to work on also.

My suggestion is very simple actually. It uses the same approach you suggested, with reputable proxies. The user has the option of inputting a list of one or more proxies to the crawl function.

Those will be sent, and used in making requests here (and other places where relevant). This function call will instead of taking the given proxy will run random.choice([proxy1, proxy2, proxy3, ...]).

If the user specified a single proxy, it will always be "chosen", otherwise a random proxy will be used for each request.

Does that make sense?

[ska-ibees]

Thanks! Very kind of you. The honor is mine.

To be honest, feeling blessed while working with you.

Does that make sense?

Got your point... so basically we are talking about a sort of wraper function within advertools instead of dealing it within Scrapy.

For example: Re-constructing meta with a random proxy from the list. 

yield Request(url, callback=self.parse, errback=self.errback, meta=self.meta)

Yes, it does make lots of sense... Let me try and push PR for it.

[ska-ibees]

PR #207 to support selection of random proxy from the list of proxies.

[ska-ibees]

Syntax for testing multi proxy:

url_list = [
    "https://www.example.com", "https://www.example.com", "https://www.example.com"
]

meta = {
    "proxy": [
        "http://abc1.com:80",
        "http://abc2.com:80",
        "http://abc3.com:80",
        "http://abc4.com:80",
    ]
}
try:
            adv.crawl(
                url_list,
                output_file,
                follow_links=True,
                custom_settings={
                    "ROBOTSTXT_OBEY": True,
                    "USER_AGENT": "Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148",
                }, meta=meta,
            )