首页 文章

保持aiohttp会话活着

提问于
浏览 次
0

我试图通过并行和单独的会话每X秒访问一个网站，然后分析响应中的内容，看看每个会话是否应该继续 . 但是，一旦代码到达第二个循环，它就会失败 .

import asyncio
from aiohttp import ClientSession
import logging
import time

interval = 30
instances = 2
visit_url = 'http://www.example.org'

tasks = []

logging.basicConfig(
    format='%(asctime)s.%(msecs)03d %(message)s',  # Log in format time.milliseconds {message}
    level=logging.INFO,  # Use with logging.info()
    datefmt='%H:%M:%S')  # Display time as Hours:Minutes:Seconds


class StopException(Exception):
    pass


async def quit_app(session, task_, reason):
    logging.info("[{}] {}.".format(task_, reason))
    session.cookies.clear()  # Reset cookies
    session.headers.clear()  # Reset headers
    session.close()  # End HTTP connection
    raise StopException


async def get_status(response):
    if "abow" in response:
        return "success"
    elif "odoap" or "daoscp" in response:
        return "waiting"
    elif "nullt" in response:
        return "fail"
    elif "issue" in response:
        return "banned"
    elif "pending" in response:
        return "pending"
    else:
        return "pending"


async def initialise(headers):
    session = ClientSession()
    task_ = len(asyncio.Task.all_tasks()) - instances - 1
    passed = False
    while passed is False:
        async with session as session:
            async with session.get(visit_url, headers=headers, allow_redirects=True) as initial:
                status = await get_status(await initial.text())  # Check HTML for status
                if status == "success":
                    logging.info("[{}] {}.".format(task_, "Success"))
                    passed = True
                elif status == "pending":
                    logging.info("[{}] {}.".format(task_, "Pending.."))
                    await asyncio.sleep(interval)
                elif status == "waiting":
                    logging.info("[{}] {}.".format(task_, "Waiting..."))
                    await asyncio.sleep(interval)
                elif status == "banned":
                    await quit_app(initial, task_, "Banned")
                elif status == "fail":
                    await quit_app(initial, task_, "Failed")


if __name__ == "__main__":
    headers = {
        'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
        'accept-encoding': 'gzip, deflate, br',
        'accept-language': 'en-US,en;q=0.9',
        'upgrade-insecure-asks': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36'
    }  # Add appropriate headers
    start = time.clock()
    loop = asyncio.get_event_loop()
    for i in range(instances):
        task = asyncio.ensure_future(initialise(headers))
        tasks.append(task)
    loop.run_until_complete(asyncio.wait(tasks))
    end = time.clock()
    print("Process took {0:.4f} seconds.".format(end - start))

此代码返回以下错误：

13：56：58.604从未检索过任务异常：任务完成coro = initialise（）完成，在C中定义：/Users/x/PycharmProjects/tests/src/aiohttp_main.py:49 exception = RuntimeError（'Session is closed '，）RuntimeError：会话已关闭

我无法解决如何使会话保持活力直到我.close（）他们......

python python-asyncio aiohttp

1 回答

  • 3

    我只是无法弄清楚如何让会话保持活力直到我.close（）他们......

    async with 是关闭会话的请求 . 这意味着你写完之后：

    async with session as session:

    ...一旦 async with 的主体完成执行，你就不能再使用 session 了 . 这不是aiohttp特有的，它是 with 在Python中的工作原理 . 例如，在处理文件时， with 是关闭文件的请求：

    with open('data.csv') as fileobj:
    # ... read stuff from fileobj

# outside the "with" block, fileobj is closed and you
# can no longer read from it

    修复很简单，只需将 with 移到 while 循环之外 . 例如：

    async def initialise(headers):
    async with ClientSession() as session:
        # the rest of the code, including the `while` loop, here

    在一个不相关的说明中，您可能希望将 len(asyncio.Task.all_tasks()) 替换为您自己的全局计数器 . 如果稍后将其他不相关的任务合并到事件循环中（或者第三方库为您执行此操作），则以这种方式使用 Task.all_tasks() 可能会开始产生不正确的结果 .

    回复于

相关问题