Chriskite Anemone

Comments and changes to this ticket

• You flagged this item as spam.
  

  hayato August 3rd, 2009 @ 05:11 PM

  patch attatched

  add on_pre_fetch handler.

  if this block return false,link is not fetched and re-enq to link_queue.

  • crawlslowly.patch 3.3 KB

• You flagged this item as spam.
  

  hayato August 4th, 2009 @ 04:41 PM

  Term "reserved" in patch causes confusion.
  "Delay" is felicity term.

  I attached the new patch.

  if apply this patch,this code behavior improve.

  require 'anemone'
  require 'uri'
  require 'anemone/delayed'

  Anemone.crawl("http://www.yahoo.co.jp/") do |anemone|
  anemone.on_every_page do |page|

  puts "#{Time.now} : #{page.url}"
  

  end

  delayed_crawl = DelayedCrawl.new

  anemone.on_pre_fetch do |link|

  delayed_crawl.can_crawl?(link.host)
  

  end

  end

  following output

  Wed Aug 05 06:36:24 +0900 2009 : http://www.yahoo.co.jp/
  Wed Aug 05 06:36:27 +0900 2009 : http://www.yahoo.co.jp/s/38667
  Wed Aug 05 06:36:30 +0900 2009 : http://www.yahoo.co.jp/s/41546
  Wed Aug 05 06:36:33 +0900 2009 : http://www.yahoo.co.jp/r/c37
  

  Wed Aug 05 06:36:36 +0900 2009 : http://www.yahoo.co.jp/s/gallery/list9.html

  • new.patch 3.3 KB

• You flagged this item as spam.
  

  hayato August 4th, 2009 @ 04:50 PM

  sorry my broken comment......

  Term "reserved" in patch causes confusion. "Delay" is felicity term.
  I attached the new patch.

  if apply this patch,this code behavior improve.

  require 'anemone'
  require 'uri'
  require 'anemone/delayed'
  
  Anemone.crawl("http://www.yahoo.co.jp/") do |anemone|
    anemone.on_every_page do |page|
      puts "#{Time.now} : #{page.url}"
    end
  
    delayed_crawl = DelayedCrawl.new
  
    anemone.on_pre_fetch do |link|
      delayed_crawl.can_crawl?(link.host)
    end
  end
  

  following output

  Wed Aug 05 06:36:24 +0900 2009 : http://www.yahoo.co.jp/
  Wed Aug 05 06:36:27 +0900 2009 : http://www.yahoo.co.jp/s/38667
  Wed Aug 05 06:36:30 +0900 2009 : http://www.yahoo.co.jp/s/41546
  Wed Aug 05 06:36:33 +0900 2009 : http://www.yahoo.co.jp/r/c37
  Wed Aug 05 06:36:36 +0900 2009 : http://www.yahoo.co.jp/s/gallery/list9.html
  

  It implemented 3sec interval

• 

  chris (at chriskite) August 10th, 2009 @ 09:06 PM

  • State changed from “new” to “resolved”

  Hi,

  I've implemented time delay functionality in the latest release of Anemone (0.1.2). You can simply specify a :delay option when starting the crawl, like so:

  require 'anemone'
  Anemone.crawl("http://www.example.com/", :delay => 3) do |anemone|
    anemone.on_every_page do |page|
      puts "#{Time.now} : #{page.url}"
    end
  end
  

• You flagged this item as spam.
  

  hayato August 24th, 2009 @ 04:25 AM

  Your fix looks good.

  But, following test is failed.

  .F.FF.
  
  1)
  'Anemone::Page should store the response headers when fetching a page' FAILED
  expected nil? to return false, got true
  ./spec/page_spec.rb:16:
  
  2)
  'Anemone::Page should have a Nokogori::HTML::Document attribute for the page body' FAILED
  expected nil? to return false, got true
  ./spec/page_spec.rb:29:
  
  3)
  'Anemone::Page should indicate whether it was fetched after an HTTP redirect' FAILED
  expected: true,
       got: false (using ==)
  ./spec/page_spec.rb:38:
  
  Finished in 0.007746 seconds
  
  6 examples, 3 failures
  

  Because useragent is unset.
  Please apply the attached patch.

  • patch2.txt 388 Bytes