使用C#刮取html文档中JavaScript动态生成的数据

如果要在浏览器解析后查看页面的外观,则需要使用工具在页面上执行JavaScript.您拥有的一个选项是使用内置的.net Web浏览器控件：http://msdn.microsoft.com/en-au/library/aa752040(v=vs.85).aspx

Web浏览器控件可以导航到并加载页面,然后您可以查询它的DOM,这些DOM将被页面上的JavaScript更改.

编辑(示例)：

Uri uri = new Uri("http://www.somewebsite.com/somepage.htm");

webBrowserControl.AllowNavigation = true;
// optional but I use this because it stops javascript errors breaking your scraper
webBrowserControl.ScriptErrorsSuppressed = true;
// you want to start scraping after the document is finished loading so do it in the function you pass to this handler
webBrowserControl.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(webBrowserControl_DocumentCompleted);
webBrowserControl.Navigate(uri);

private void webBrowserControl_DocumentCompleted(object sender,WebBrowserDocumentCompletedEventArgs e)
{
    HtmlElementCollection divs = webBrowserControl.Document.GetElementsByTagName("div");

    foreach (HtmlElement div in divs)
    {
        //do something
    }
}