首页 文章

不适当的部署Scrapy代理

提问于
浏览
0

当我正在抓取 Profiles 时,我收到了一条错误消息 . 我假设我使用我的代理错了 . 但这里的主要错误是什么?你们能帮忙吗?

2017-06-15 21:35:17 [scrapy.proxies]信息:删除失败的代理,剩下12个代理2017-06-15 21:35:17 [scrapy.core.scraper]错误:下载https://时出错www.linkedin.com/in/jiajie-jacky-fan-80920083/> Traceback(最近一次调用最后一次):文件“/Users/jiajiefan/data_mining/lib/python2.7/site-packages/twisted/internet/defer . py“,第1299行,在_inlineCallbacks中结果= result.throwExceptionIntoGenerator(g)文件”/Users/jiajiefan/data_mining/lib/python2.7/site-packages/twisted/python/failure.py“,第393行,在throwExceptionIntoGenerator中返回g.throw(self.type,self.value,self.tb)File“/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/core /downloader/middleware.py“,第43行,在process_request中defer.returnValue((yield download_func(request = request,spider = spider)))文件”/Users/jiajiefan/data_mining/lib/python2.7/site-packages/ Scrapy-1.4.0-py2.7.egg / scrapy / utils / defer.py“,第45行,在mustbe_deferred结果= f(* args,** kw)文件”/ Users / jiajiefan /data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/core/downloader/handlers/init.py“,第65行,在download_request中返回handler.download_request(请求, spider)文件“/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/core/downloader/handlers/http11.py”,第63行, download_request return agent.download_request(request)File“/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/core/downloader/handlers/http11.py “,第272行,在download_request代理= self._get_agent(请求,超时)文件”/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/ core / downloader / handlers / http11.py“,第252行,在_get_agent ,proxyHost,proxyPort,proxyParams = _parse(proxy)File”/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy -1.4.0-py2.7.egg / scrapy / core / downloader / webclient.py“,第37行,在_parse中返回_parsed_url_args(已解析)文件”/ Users / jiajiefan / data_min在/parsed_url_args port = parsed.port文件中/ / / / / Frameworks / Python.framework / Versions / 2.7 / lib / python2.7 / urlparse.py“,第113行,在端口port = int(port,10)ValueError:无效的文字fo r int(),基数为10:'178.32 . 255.199'

1 回答

  • 0

    代理应该有'http'等地址:

    rq.meta['proxy'] = 'http://127.0.0.1:8123'
    

相关问题