BeautifulSoup 无法识别 <main class= 之后的子类

BeautifulSoup doesn't recognize subclasses after <main class=

提问人：ИНДУС Геймдев 提问时间：7/10/2023 最后编辑：ИНДУС Геймдев 更新时间：7/10/2023 访问量：53

问：

我需要解析一个新闻网站，新闻的标题和链接位于 <main class='single-module__Main-sc-1qdjg1k-0 iMZnZU' 中。我尝试查询子类，但无法查询。soup.prettyfi（） 生成了如此完整的标记

    <div class="single-module__Inner-sc-1qdjg1k-1 mIFFS">
      <header class="sticky-header">
       <div data-fusion-collection="features" data-fusion-message="Could not render component [features:global/main-navigation]" data-fusion-type="global/main-navigation" id="f0fj2XGPHYgA9Rb" style="display:none">
       </div>
      </header>
      <main class="single-module__Main-sc-1qdjg1k-0 iMZnZU">
       <div data-fusion-collection="features" data-fusion-message="Could not render component [features:global/search-page]" data-fusion-type="global/search-page" id="f0f1nZqKTkTE1lq" style="display:none">
       </div>
      </main>
      <footer>
       <div data-fusion-collection="features" data-fusion-message="Could not render component [features:global/footer]" data-fusion-type="global/footer" id="f0fUuwTNo76j9AD" style="display:none">
       </div>
      </footer>

我试图查看子类，但得到相同的结果


q = soup.find('div', class_='layout-container').find('div','single-module__Inner-sc-1qdjg1k-1mIFFS')
print(q.find('main','single-module__Main-sc-1qdjg1k-0 iMZnZU')
python html 解析 网页抓取 beautifulsoup

评论

2赞 Tim Roberts 7/10/2023
请记住，它只返回第一个匹配的元素。可以不止一个吗？find
0赞 ИНДУС Геймдев 7/10/2023
@TimRoberts，当尝试使用 soup.find_all（'div'） 或 soup.find（'div'， class_='layout-container'））.find_all（'div'） 时，大多数类都会给出“无法渲染组件 [features：global /search-page]”
1赞 Barmar 7/10/2023
您在第二次调用时丢失了class_=find()
0赞 Tim Roberts 7/10/2023
但这是正确的答案，不是吗？该元素的内容是包含该文本的内容。你期待什么？页面的实际文本很可能是使用 Javascript 动态填充的。它不会在 HTML 中。<main><div>
0赞 ИНДУС Геймдев 7/10/2023
@TimRoberts 很可能是对的，我指定了class_并多次查看子类，但输出没有改变。我应该使用另一个解析框架而不是bs4吗？

答：

0赞 Barry the Platipus 7/10/2023 #1

以下是获取这些新闻的一种方法 - 它们正在通过单独的 XHR 调用被水化到页面中，因此您可以直接抓取该 API 端点：

import requests
import pandas as pd

headers= {
    'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36'
}

url = 'https://api.queryly.com/json.aspx?queryly_key=d0ab87fd70264c0a&query=btc&batchsize=10&showfaceted=true&maxfacetitems=75&extendeddatafields=basic,creator,creator_slug,subheadlines,primary_section,report_url,section_path,sections_paths,subtype,type,imageresizer,section,sponsored_label,sponsored,promo_image,pubDate&timezoneoffset=-60'
r = requests.get(url, headers=headers)
df = pd.json_normalize(r.json(), record_path=['items'])
display(df[['section', 'subtype', 'title', 'pubdate', 'link', 'section_path', 'image']])

终端结果：

section     subtype     title   pubdate     link    section_path    image
0   Markets     story_news  First Mover Asia: Bitcoin Whales Are Increasin...   Jul 07, 2023    /markets/2023/07/07/first-mover-asia-bitcoin-w...   /markets    https://cloudfront-us-east-1.images.arcpublish...
1   Markets     story_news  Bitcoin Drops Below $30K as Altcoins Tumble; B...   Jun 28, 2023    /markets/2023/06/28/bitcoin-drops-below-30k-as...   /markets    https://cloudfront-us-east-1.images.arcpublish...
2   Policy  story_news  Bankrupt Celsius Can Convert Altcoins to BTC, ...   Jun 30, 2023    /policy/2023/06/30/bankrupt-celsius-can-conver...   /policy     
3   Finance|Markets     story_news  Celsius to Potentially Sell More Than $170M in...   Jun 30, 2023    /business/2023/06/30/celsius-to-potentially-se...   /business   https://cloudfront-us-east-1.images.arcpublish...
4   Markets     story_news  Bitcoin Tumbles on Report of SEC Saying Spot B...   Jun 30, 2023    /markets/2023/06/30/bitcoin-tumbles-on-report-...   /markets    
5   Markets     story_news  First Mover Asia: Bitcoin Crosses $31K After S...   Jul 04, 2023    /markets/2023/07/04/first-mover-asia-bitcoin-c...   /markets    https://cloudfront-us-east-1.images.arcpublish...
6   Mercados|Coindesk-ES    story_news  El gran repunte de bitcoin impulsado por los E...   Jun 22, 2023    /es/markets/2023/06/22/el-gran-repunte-de-bitc...   /es/markets     
7   Markets     story_news  Bitcoin’s Ferocious ETF-Fueled Rally Puts Ethe...   Jun 22, 2023    /markets/2023/06/22/bitcoins-ferocious-etf-fue...   /markets    
8   Finance     story_news  CME Group Announces Plans to Launch ETH to BTC...   Jun 29, 2023    /business/2023/06/29/cme-group-announces-plans...   /business   
9   Markets     story_news  First Mover Asia: Bitcoin Holds Firm Above $30...   Jun 22, 2023    /markets/2023/06/22/first-mover-asia-bitcoin-h...   /markets

上一个：BeautifulSoup 洗牌 html 标签的属性

下一个：简单的 jsoup 解析和 webview 输出