Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

Memleak with pages?

#24

Is it possible that, when running a lot of .crawl() operations, the @pages hash will keep on growing and there is no method that allows the user to empty it?
(using a simple Hash in my case)

Reported by rb2k · May 15th, 2010 @ 04:06 PM

State: resolved
Milestone: none
Assigned to: chris (at chriskite) chris (at chriskite)

Activity

  1. chris (at chriskite)
    chris (at chriskite)
    • State changed from new to resolved

    Yes, although it's not really a "leak" because it is intentionally storing data about all the pages you crawl. If you crawl a lot of pages, that data has to go somewhere, and if you're using a Hash then it's in memory. Using the TokyoCabinet storage engine is a good solution, as the data is persisted on disk and doesn't use nearly as much memory.

    May 25th, 2010 @ 09:13 PM

  2. rb2k
    rb2k

    I crawl different domains though.
    Is there a way to "reset" the page cache in between crawls?

    May 25th, 2010 @ 09:16 PM

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.