首页 文章

在Python类中使用asyncio和aiohttp

提问于
浏览
1

我一直试图找出如何在类中使用asyncio和aiohttp . 如果我只是尝试运行没有Class的脚本(只是按原样使用函数),一切正常 . 只要我将所有函数带入一个类并尝试使用Main.py中的类,脚本就会锁定而不会出现任何错误 . 不确定从哪里开始,我猜我必须以不同方式设置我的 class 才能工作 . 如果有人知道为什么这不起作用,如果你分享我做错了,我将不胜感激 . 感谢您的时间 .

Fetch.py

import asyncio
from aiohttp import ClientSession

class Fetch:
 def __init__(self, proxy=None):
  self.proxy = proxy
  self.headers =  {'user-agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'}

 def set_headers(self, headers):
  if not headers:
   headers = self.headers
  return headers

 def set_proxy(self, proxy):
  if proxy:
   p = proxy
  else:
   p = self.proxy
  return "http://{}".format(p)

 async def get_fetch(self, session, url, headers=None, proxy=None, params=None, timeout=9):
  array = []
  while True:
   try:
    async with session.get(url, headers=self.set_headers(headers), proxy=self.set_proxy(proxy), params=params, timeout=timeout) as r:
     print (r.status)
     if r.status == 200:
      obj = await r.read()
      array.append(obj)
      break
   except:
    pass
  return array

 async def get_bound(self, sem, session, url):
  async with sem:
   array = await self.get_fetch(session, url)
   return array

 async def get_run(self, urls, semaphores=400):
  tasks = []
  sem = asyncio.Semaphore(semaphores)

  async with ClientSession() as session:
   for url in urls:
    task = asyncio.ensure_future(self.get_bound(sem, session, url))
    tasks.append(task)

  responses = await asyncio.gather(*tasks)
  return responses

 def get(self, urls):
  loop = asyncio.get_event_loop()
  future = asyncio.ensure_future(self.get_run(urls))
  array = loop.run_until_complete(future)
  loop.close()
  return [ent for sublist in array for ent in sublist]

Main.py

from Browser import Fetch
from bs4 import BeautifulSoup

proxy = 'xxx.xxx.xxx.xxx:xxxxx'
fetch = Fetch(proxy)

if __name__ == '__main__':
 urls = ['http://ip4.me','http://ip4.me','http://ip4.me']
 array = fetch.get(urls)
 for obj in array:
  soup = BeautifulSoup(obj, 'html.parser')
  for ip in soup.select('tr +  tr td font'):
   print(ip.get_text())

1 回答

  • 2

    你的缩进是错误的 .

    async with ClientSession() as session:
        for url in urls:
            task = asyncio.ensure_future(self.get_bound(sem, session, url))
            tasks.append(task)
    
    responses = await asyncio.gather(*tasks)
    return responses
    

    将最后两行放回 with 块中 .

    您的代码与https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html类似 . 在此引用中, await responses 和相关语句完全在 with 块内,否则您的代码会在http调用返回之前使 ClientSession 实例超出范围(并且基础会话将被关闭) .

    另外,请考虑代码的标准缩进样式 . 单个空间使得很难发现这些容易出错的错误 .

相关问题