Ruby+Erlang抓取贴吧图片

以下代码由PHP站长网 52php.cn收集自互联网

现在PHP站长网小编把它分享给大家，仅供参考

#coding utf-8
#!/usr/bin/env ruby -w

require "open-uri"
require "nokogiri"
require "thread"
require 'erlectricity'

class Spider
  def initialize(dir,url)
    @url = url
    @dir = dir
  end

  def create_dir
    `rm -rf #{@dir}` if File.directory? @dir
    Dir.mkdir @dir
  end

  def get_imgs(doc)
    links = doc.css('img.BDE_Image').collect { |img| img['src'] }
  end

  def run(pn,f)
    create_dir()
    threads = []
    (1..pn).each do |i|
      threads << Thread.new do
        puts "going to #@url in page #{i}"
        doc = Nokogiri::HTML(open("#@url?pn=#{i}"))
        links = get_imgs(doc)
        f.send!([:links,links])
      end
    end
    threads.each &:join
  end
end

receive do |f|
  _pn = 1

  f.when([:pn,Fixnum]) do |pn|
    _pn = pn
    f.send!([:pn,_pn])
    f.receive_loop
  end

  f.when([:url,String]) do |url|
    if url =~ /.*/p/(d+)/?/
      spider = Spider.new($1,url)
      spider.run(_pn,f)
      f.send!(:exit)
    else
      f.send!([:error,"error."])
      f.receive_loop
    end
  end
end

以上内容由PHP站长网【52php.cn】收集整理供大家参考研究

如果以上内容对您有帮助，欢迎收藏、点赞、推荐、分享。

（编辑：李大同）

【声明】本站内容均来自网络，其相关言论仅代表作者个人观点，不代表本站立场。若无意侵犯到您的权利，请及时与联系站长删除相关内容!