crawl4ai

Crawl4AI 是一个开源的 Python 库，旨在简化网络爬取并轻松从网页中提取有价值信息。无论您是将其集成为 REST API 还是直接在 Python 项目中使用，Crawl4AI都提供了一个强大、灵活和完全异步的解决方案，专为大型语言模型（LLMs）和 AI 应用量身定制。

简介

Crawl4AI旨在使网络爬取和从网页中提取数据变得简单高效。无论您是构建复杂的 AI 应用程序还是增强大型语言模型，Crawl4AI都提供了您需要简化工作流程的工具。通过完全的异步支持，Crawl4AI确保您的爬取任务快速、可靠和可扩展。

快速入门 🚀

在短时间内快速上手Crawl4AI！以下是一个展示其强大异步功能的简单示例：

python

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    # 初始化异步网络爬虫
    async with AsyncWebCrawler(verbose=True) as crawler:
        # 爬取指定的 URL
        result = await crawler.arun(url="https://www.nbcnews.com/business")
        # 以 Markdown 格式显示提取的内容
        print(result.markdown)

 __name__ == :
    asyncio.run(main())

import os import asyncio from crawl4ai import AsyncWebCrawler from crawl4ai.extraction_strategy import LLMExtractionStrategy from pydantic import BaseModel, Field class OpenAIModelFee(BaseModel): model_name: str = Field(..., description="OpenAI 模型的名称。") input_fee: str = Field(..., description="OpenAI 模型的输入令牌费用。") output_fee: str = Field(..., description="OpenAI 模型的输出令牌费用。") async def main(): async with AsyncWebCrawler(verbose=True) as crawler: result = await crawler.arun( url='https://openai.com/api/pricing/', word_count_threshold=1, extraction_strategy=LLMExtractionStrategy( provider="openai/gpt-4o", api_token=os.getenv('OPENAI_API_KEY'), schema=OpenAIModelFee.schema(), extraction_type="schema", instruction="""从爬取的内容中提取所有提到的模型名称以及它们的输入和输出令牌费用。不要错过整个内容中的任何模型。提取的模型 JSON 格式应如下所示： {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}。""" ), bypass_cache=True, ) print(result.extracted_content) if __name__ == "__main__": asyncio.run(main()) ``````python import asyncio import re from bs4 import BeautifulSoup from crawl4ai import AsyncWebCrawler async def crawl_typescript_commits(): first_commit = "" async def on_execution_started(page): nonlocal first_commit try: while True: await page.wait_for_selector('li.Box-sc-g0xbh4-0 h4') commit = await page.query_selector('li.Box-sc-g0xbh4-0 h4') commit = await commit.evaluate('(element) => element.textContent') commit = re.sub(r'\s+', '', commit) if commit and commit != first_commit: first_commit = commit break await asyncio.sleep(0.5) except Exception as e: print(f"Warning: New content didn't appear after JavaScript execution: {e}") async with AsyncWebCrawler(verbose=True) as crawler: crawler.crawler_strategy.set_hook('on_execution_started', on_execution_started) url = "https://github.com/microsoft/TypeScript/commits/main" session_id = "typescript_commits_session" all_commits = [] js_next_page = """ const button = document.querySelector('a[data-testid="pagination-next-button"]'); if (button) button.click(); """ for page in range(3): # Crawl 3 pages result = await crawler.arun( url=url, session_id=session_id, css_selector="li.Box-sc-g0xbh4-0", js=js_next_page if page > 0 else None, bypass_cache=True, js_only=page > 0 ) assert result.success, f"Failed to crawl page {page + 1}" soup = BeautifulSoup(result.cleaned_html, 'html.parser') commits = soup.select("li.Box-sc-g0xbh4-0") all_commits.extend(commits) print(f"Page {page + 1}: Found {len(commits)} commits") await crawler.crawler_strategy.kill_session(session_id) print(f"Successfully crawled {len(all_commits)} commits across 3 pages") if __name__ == "__main__": asyncio.run(crawl_typescript_commits())

项目详情

简介

快速入门 🚀

相关项目

Scrapegraph-ai

解释：

特点 ✨

安装 🛠 ️

使用 pip 🐍

基本安装

安装同步版本

开发安装

高级用法 🔬

执行 JavaScript & 使用 CSS 选择器

使用代理

无需LLM提取结构化数据

利用OpenAI提取结构化数据

速度比较 🚀

参考资料