我正在制作一个脚本,用于从网站的页面源中提取所有评论.
for addr in links:
driver.get(addr)
print(addr)
for comments in soup.findAll(text=lambda text: isinstance(text,Comment)):
comments.extract()
print(comments)
为什么它只这样提取CDATA：
//<![CDATA[
Feedback.Bootstrap.InitializeFeedback({page:true},"epf",true,false,false);;
//]]>
//<![CDATA[
function addRemoveListenersOnAll(){var t=_ge("b_content"),n,i,r;t&&(n=_d.createElement("STYLE"),n.id=styleIdString,n.innerText="#b_results h2>a {padding: 16px 40px 0 6px;margin: -16px -40px 0 -6px;}",_d.head&&_d.head.appendChild(n),i=t.getElementsByClassName("b_ad"),i&&AddRemoveListener(i),r=t.getElementsByClassName("b_algo"),r&&AddRemoveListener(r))}function AddRemoveListener(n){for(var t,u=n.length,r=0;r<u;r++)if(t=n[r].getElementsByTagName("CITE"),t&&t.length>0)for(i=0;i<t.length;i++)sj_be(t[i],"click",algo_c)}function mouseMoveAfterTouchHandler(){sj_ue(document,"mousemove",mouseMoveAfterTouchHandler);var n=_d.getElementById(styleIdString);n&&n.parentNode&&n.parentNode.removeChild(n);sj_log("CI.TTC","mouse","started");sj_ue(document,mouseMoveAfterTouchHandler)}function touchStartHandlerAll(n){n.pointerType==="touch"&&(addRemoveListenersOnAll(),sj_log("CI.TTC","touch","started"),sj_ue(document,"pointerdown",touchStartHandlerAll),document.addEventListener("mousemove",mouseMoveAfterTouchHandler))}var styleIdString="ttcDynStyle",algo_c=function(n){function i(n){var t=n.getElementsByTagName("a"),i;t&&t.length>0&&(i=t[0],si_ct(i),"touch"),_w.open(i.href,"_self"))}n=sj_ev(n);var t=sj_et(n);if(t){if(t.tagName=="A")return!0;while(t&&!(t.className.indexOf("b_algo")>=0||t.className.indexOf("sb_add")>=0)){if(t.tagName=="BODY")return;t=t.parentNode}}return t?(i(t),!0):(sj_sp(n),!1)};document.addEventListener("pointerdown",touchStartHandlerAll);Feedback.Bootstrap.InitializeFeedback({page:true},"sb_feedback",1,0);;
//]]>
但它不会提取此类常规注释：
<!--div class="s-bk-lf"><div class="acc-title" >Следите за новостями и акциями нашего проекта!!!</div> </div><br-->
<!--LiveInternet counter-->
<!--img src="/img/ng6.png"width="150" height="" hspace="100" vspace="80" align="left" -->
如何提取常规注释而不是仅提取CDATA？
我发现它正在查看脚本开头的漂亮汤解析的html(来自bing搜索结果.
我通过在循环中添加beautifulsoup来修复它.
for addr in links:
print(addr)
driver.get(addr)
html = driver.page_source
soup = BeautifulSoup(html,"html.parser")
for comments in soup.findAll(text=lambda text: isinstance(text,Comment)):
comments.extract()
print(comments)
这不是很干净,因为我只是从脚本的前面复制了html和汤,但是它确实起作用.
