Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

100% cpu + memory filling at certain sites

#28

I don't know what causes it yet, but there are some sites that seem to break the crawling process.
two of them are e.g.
http://www.skiandorre.com/
http://www.hiver2018.com/

happens on 1.8 and 1.9
no network load detectable during that time

here's a short example, just let that run for 15 seconds and it should stop and go 100% cpu

require 'anemone'

Anemone.crawl("http://www.hiver2018.com/") do |anemone|
  anemone.on_every_page do |page|
       puts page.url
  end
end

Reported by rb2k · June 21st, 2010 @ 11:33 AM

State: new
Milestone: none
Assigned to: chris (at chriskite) chris (at chriskite)

Activity

  1. rb2k
    rb2k

    also: sites are the same and some sort of spam...
    they all lead to p*.vixns.eu

    my guess would be an xml parser going haywire

    June 21st, 2010 @ 11:36 AM

  2. rb2k
    rb2k

    ok, just replaced nokogiri with hpricot, still the same problem

    This page is however weird: http://www.hiver2018.com/partenaires.php
    It's 34 MB and basically looks like this 99% of the time:

    ><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" target="_blank"></a></p><br /><p><a href="http://" t

    June 21st, 2010 @ 11:47 AM

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.