#28 new
rb2k

100% cpu + memory filling at certain sites

Reported by rb2k | 2010-06-21 11:33:07 UTC

I don't know what causes it yet, but there are some sites that seem to break the crawling process.
two of them are e.g.
http://www.skiandorre.com/
http://www.hiver2018.com/

happens on 1.8 and 1.9
no network load detectable during that time

here's a short example, just let that run for 15 seconds and it should stop and go 100% cpu

require 'anemone'

Anemone.crawl("http://www.hiver2018.com/") do |anemone|
  anemone.on_every_page do |page|
       puts page.url
  end
end

Comments and changes to this ticket

  • rb2k

    rb2k 2010-06-21 11:36:53 UTC

    also: sites are the same and some sort of spam...
    they all lead to p*.vixns.eu

    my guess would be an xml parser going haywire

  • rb2k

    rb2k 2010-06-21 11:47:27 UTC

    ok, just replaced nokogiri with hpricot, still the same problem

    This page is however weird: http://www.hiver2018.com/partenaires.php
    It's 34 MB and basically looks like this 99% of the time:

    ><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" t

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

Anemone is a Ruby library that makes it quick and painless to write programs that spider a website. It provides a simple DSL for performing actions on every page of a site, skipping certain URLs, and calculating the shortest path to a given page on a site.

Shared Ticket Bins

People watching this ticket

Pages