我试图抓取这个网站：https://www.senate.gov/general/contact_information/senators_cfm.cfm
我的代码：
import requests
from bs4 import BeautifulSoup
URL = 'https://www.senate.gov/general/contact_information/senators_cfm.cfm'
page = requests.get(URL)
soup = BeautifulSoup(page.content, 'html.parser')
print(soup)
问题是它实际上并没有到达网站。我在soup var中得到的HTML根本不是正确网页中的HTML。
我不知道从这里去哪里！任何和所有的帮助都将不胜感激。
这对我很管用
headers = {
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
}
r = requests.get(URL,headers=headers)
在此找到信息-https://towards data science . com/5-strategies-to-write-unblock-able-web-scrapers-in-python-5e 40 c 147 bdaf
使用python请求模块时出现重复的HTTP 503错误
试试看：
import requests
from bs4 import BeautifulSoup
URL = 'https://www.senate.gov/general/contact_information/senators_cfm.cfm'
page = requests.post(URL, headers=headers)
soup = BeautifulSoup(page.content, 'html.parser')
print(soup)
