异步。队列 - 从 txt 文件放置 url-解网

问：

我需要将 txt 文件中的 URL 传递给获取 URL 的脚本，并在和的帮助下做一些工作。所以我习惯于将 txt 文件中的 URL 放入队列中。细微差别是txt文件可能非常大，可能不适合内存，我尝试使用队列。但是当限制小于 URL 数量时 - 脚本被卡住了。所以我有两个问题：aiohttpasyncioasyncio.Queuemaxsize

我的错误在哪里？
也许有比在这种情况下使用更好的解决方案？asyncio.Queue

script.py：

import asyncio
from aiohttp import ClientSession


async def fetch(url):
    async with ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()


async def main():
    with open("urls.txt", 'r') as f:
        queue = asyncio.Queue(maxsize=5)
        for line in f:
            line = line.strip()
            await queue.put(line)
        # await queue.join() - tried this

    while not queue.empty():
        current_url = await queue.get()
        try:
            content = await fetch(current_url)
            print(f"{current_url}: {content[:15]}")
        except Exception as e:
            print(f"Error fetching {current_url}: {e}")
        finally:
            queue.task_done()

asyncio.run(main())

urls.txt：

https://www.google.com/
https://www.youtube.com/
https://www.facebook.com/
https://www.wikipedia.org/
https://www.amazon.com/
https://www.instagram.com/
https://www.twitter.com/
https://www.tumblr.com/
https://www.pinterest.com/
https://www.reddit.com/

在这个脚本中，我使用 txt 文件中的 URL 数量为 10。我试图添加 after 循环，但没有帮助。脚本仅在没有 >= 个 URL 或未指定时才有效。maxsize=5await queue.join()for line in f:await queue.join()maxsize

异步队列 python-asyncio aiohttp

import asyncio

from aiohttp import ClientSession


async def fetch(url):
    async with ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()


async def consumer(queue):
    while True:
        current_url = await queue.get()
        try:
            if current_url is None:
                break

            content = await fetch(current_url)
            print(f"{current_url}: {content[:15].strip()}")
        except Exception as e:
            print(f"Error fetching {current_url}: {e}")
        finally:
            queue.task_done()


async def main():
    queue = asyncio.Queue(maxsize=5)

    # for example, we create 3 workers consuming the queue
    workers = {asyncio.Task(consumer(queue)) for _ in range(3)}

    with open("urls.txt", "r") as f:
        for line in f:
            line = line.strip()
            await queue.put(line)

    # end the workers
    for _ in range(len(workers)):
        await queue.put(None)

    await queue.join()


asyncio.run(main())

指纹：

https://www.google.com/: <!doctype html>
https://www.facebook.com/: <!DOCTYPE html>
https://www.wikipedia.org/: <!DOCTYPE html>
https://www.amazon.com/: <!doctype html>
https://www.instagram.com/: <!DOCTYPE html>
https://www.tumblr.com/: <!doctype html
https://www.youtube.com/: <!DOCTYPE html>
https://www.twitter.com/: <!DOCTYPE html>
https://www.pinterest.com/: <!DOCTYPE html>
https://www.reddit.com/: <!DOCTYPE

异步。队列 - 从 txt 文件放置 url

asyncio.Queue - put urls from txt file

评论

评论