Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

rb2k

rb2k

Not a member

rb2k's latest activity

Monday, June 21 2010
28 100% cpu + memory filling at certain sites was updated in Anemone
  • rb2k
    rb2k commented at 11:47 AM

    ok, just replaced nokogiri with hpricot, still the same problem

    This page is however weird: http://www.hiver2018.com/partenaires.php
    It's 34 MB and basically lo...
  • rb2k
    rb2k commented at 11:37 AM

    also: sites are the same and some sort of spam...
    they all lead to p*.vixns.eu

    my guess would be an xml parser going haywire
  • rb2k
    rb2k created the ticket at 11:33 AM

    I don't know what causes it yet, but there are some sites that seem to break the crawling process.
    two of them are e.g.
    http://www.skiandorre.com/
    http://www.hi...
Tuesday, May 25 2010
24 Memleak with pages? was updated in Anemone
  • rb2k
    rb2k commented at 9:16 PM

    I crawl different domains though.
    Is there a way to "reset" the page cache in between crawls?
Tuesday, May 18 2010
25 Normalize redirects was created in Anemone
  • rb2k
    rb2k created the ticket at 7:46 AM

    in the http.rb:

    redirect_to = response.is_a?(Net::HTTPRedirection) ?  URI(response['location']): nil
    should be
    redirect_to = response.is_a?(Net::HTTPRedirect...
Saturday, May 15 2010
24 Memleak with pages? was created in Anemone
  • rb2k
    rb2k created the ticket at 4:07 PM

    Is it possible that, when running a lot of .crawl() operations, the @pages hash will keep on growing and there is no method that allows the user to empty it?
    (u...
Thursday, May 06 2010
23 Allow root domain redirects was created in Anemone
  • rb2k
    rb2k created the ticket at 2:33 PM

    Anemone isn't currently able to crawl a site if the root domain redirects to another (sub)domain

    an example of this would be http://heise.de which redirects to ...
22 option to stop crawling was created in Anemone
  • rb2k
    rb2k created the ticket at 10:56 AM

    It would be nice if there was a anemone.stop_crawl method that could be called from e.g. within the on_every_page block.
    In my case, I simply want to stop crawl...
Wednesday, May 05 2010
21 pluggable html parser was updated in Anemone
  • rb2k
    rb2k commented at 12:33 PM

    Here are some performance benchmarks.
    Switching from CSS to xpath also results in speed boosts:

    http://gist.github.com/391134

    (tl;dr: hpricot (xpath) took: 3.59...
Tuesday, May 04 2010
21 pluggable html parser was created in Anemone
  • rb2k
    rb2k created the ticket at 3:16 PM

    It would be nice if there was the possibility of using hpricot instead of nokogiri.
    They should be API compatible.

    In my tests, hpricot was always a tiny bit fa...