Python异步爬虫（aiohttp版）

异步协程不太了解的话可以去看我上篇博客：https://www.cnblogs.com/Red-Sun/p/16934843.html
PS：本博客是个人笔记分享，不需要扫码加群或必须关注什么的（如果外站需要加群或关注的可以直接去我主页查看）
欢迎大家光临ヾ(≧▽≦*)o我的博客首页https://www.cnblogs.com/Red-Sun/

1.requests请求

# -*- coding: utf-8 -*- # @Time    : 2022/12/6 16:03 # @Author  : 红后 # @Email   : not_enabled@163.com # @blog    : https://www.cnblogs.com/Red-Sun # @File    : 实例1.py # @Software: PyCharm import aiohttp, asyncio   async def aiohttp_requests(url):  # aiohttp的requests函数     async with aiohttp.request("GET", url=url) as response:         return await response.text(encoding='UTF-8')   async def main():  # 主函数用于异步函数的启动     url = 'https://www.baidu.com'     html = await aiohttp_requests(url)  # await修饰异步函数     print(html)   if __name__ == '__main__':     loop = asyncio.get_event_loop()     loop.run_until_complete(main())  

2.session请求

GET：

# -*- coding: utf-8 -*- # @Time    : 2022/12/6 16:33 # @Author  : 红后 # @Email   : not_enabled@163.com # @blog    : https://www.cnblogs.com/Red-Sun # @File    : 实例2.py # @Software: PyCharm import aiohttp, asyncio   async def aiohttp_requests(url):  # aiohttp的requests函数     async with aiohttp.ClientSession() as session:  # 声明了一个支持异步的上下文管理器         async with session.get(url) as response:             return await response.text(encoding='UTF-8')   async def main():  # 主函数用于异步函数的启动     url = 'https://www.baidu.com'     html = await aiohttp_requests(url)  # await修饰异步函数     print(html)   if __name__ == '__main__':     loop = asyncio.get_event_loop()     loop.run_until_complete(main()) 

其中aiohttp还有post，put, delete...等一系列请求（PS：一般情况下只需要创建一个session，然后使用这个session执行所有的请求。）
PSOT：传参

async def aiohttp_requests(url):  # aiohttp的requests函数     async with aiohttp.ClientSession() as session:         data = {'key': 'value'}         async with session.post(url=url, data=data) as response:             return await response.text(encoding='UTF-8') 

PS：这种传参传递的数据将会被转码，如果不想被转码可以直接提交字符串data=str(data)

附：关于session请求数据修改操作

1.cookies

自定义cookies应该放在ClientSession中，而不是session.get()中

async def aiohttp_requests(url):  # aiohttp的requests函数     cookies = {'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36'}     async with aiohttp.ClientSession(cookies=cookies) as session:         async with session.get(url) as response:             return await response.text(encoding='UTF-8') 

2.headers

放在自定义的headers跟正常的requests一样放在session.get()中

async def aiohttp_requests(url):  # aiohttp的requests函数     async with aiohttp.ClientSession() as session:         headers = {'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36'}         async with session.get(url=url, headers=headers) as response:             return await response.text(encoding='UTF-8') 

3.timeout

默认响应时间为5分钟，通过timeout可以重新设定，其放在session.get()中

async def aiohttp_requests(url):  # aiohttp的requests函数     async with aiohttp.ClientSession() as session:         async with session.get(url=url, timeout=60) as response:             return await response.text(encoding='UTF-8') 

4.proxy

当然代理也是支持的在session.get()中配置

async def aiohttp_requests(url):  # aiohttp的requests函数     async with aiohttp.ClientSession() as session:         async with session.get(url=url, proxy="http://some.proxy.com") as response:             return await response.text(encoding='UTF-8') 

需要授权的代理

async def aiohttp_requests(url):  # aiohttp的requests函数     async with aiohttp.ClientSession() as session:         proxy_auth = aiohttp.BasicAuth('user', 'pass')  # 用户，密码         async with session.get(url=url, proxy="http://some.proxy.com", proxy_auth=proxy_auth) as response:             return await response.text(encoding='UTF-8') 

或者

async def aiohttp_requests(url):  # aiohttp的requests函数     async with aiohttp.ClientSession() as session:         async with session.get(url=url, proxy='http://user:pass@some.proxy.com') as response:             return await response.text(encoding='UTF-8') 

报错处理

错误：RuntimeError: Event loop is closed

报错原因是使用了asyncio.run(main())来运行程序
看到别个大佬的总结是asyncio.run()会自动关闭循环,并且调用_ProactorBasePipeTransport.__del__报错, 而asyncio.run_until_complete()不会。
第一种解决方法换成如下代码运行

loop = asyncio.get_event_loop() loop.run_until_complete(main())  

第二种重写方法以保证run()的运行

from functools import wraps  from asyncio.proactor_events import _ProactorBasePipeTransport  def silence_event_loop_closed(func):     @wraps(func)     def wrapper(self, *args, **kwargs):         try:             return func(self, *args, **kwargs)         except RuntimeError as e:             if str(e) != 'Event loop is closed':                 raise     return wrapper  _ProactorBasePipeTransport.__del__ = silence_event_loop_closed(_ProactorBasePipeTransport.__del__)