Activity (RSS)

Stack overflow can answer this for you.
Link

learning new markups is harrowing

   rvm pkg install openssl
   rvm remove 1.9.2 
   rvm install 1.9.2 --with-openssl-dir=$HOME/.rvm/usr

correction -- @@@ rvm pkg install openssl

rvm remove 1.9.2 
rvm install 1.9.2 --with-openssl-dir=$HOME/.rvm/usr

The error is due to ruby being installed without openssl support. To resolve this problem, you will have to remove ruby and install it again. In case you are us...

In case you or someone else is still facing this problem, the following solution can be applied.

Anemone.crawl("http://heise.de") do |anemone|
anemone.on_every_...

Can you provide a skip_links_like example.

I am having problems understanding the syntax. Thanks.

Hi,
I am new to Ruby (learning Stage) and am using Anemone for crawling a site.
1) I have use command " gem install anemone" to install anemone on mac and recei...

Agreed completely, if there's a canonical URI/L specified in the source, it should be used (or at least made available).

If page A 301 redirects to page B, Anemone will try to crawl page B again if link occurs later.

Any idea what the memory usage was like on your system towards the end of the crawl? Which storage engine are you using, the default in-memory hash or TokyoCabi...

Do either of you guys have an example of a URL that causes this? I can't reproduce it with the sites I usually crawl.

Great catch, I'll include a fix for this in 0.4.1

The thread body of the tentacle looks like this:

    def run
      loop do
        link, referer, depth = @link_queue.deq

        break if link == :END

     ...

I'm having this same issue.

I changed my crawl to use 2 threads and discard page bodies, which appears to avoid this issue.

Anemone appears to respect nofollow links if it is before the href, but not if it appears later:

Ignores link

Still follows link

Minor issue, when using Tokyo Cabinet the URLs returned in a .on_every_page loop stop being unique quickly if the directory specified for the database file is i...

OS X Snow Leopard
Ruby 1.8.7 (System Ruby)

require 'rubygems'
require 'bundler'
Bundler.setup
require 'anemone'

files = {}

Anemone.crawl 'http://local.acton....

It seems that links with %20's get double encoded while trying to strip the anchor off.

For example, if a page contains the link:

/Company%20Info/103070.aspx

It...

ok, just replaced nokogiri with hpricot, still the same problem

This page is however weird: http://www.hiver2018.com/partenaires.php
It's 34 MB and basically lo...

also: sites are the same and some sort of spam...
they all lead to p*.vixns.eu

my guess would be an xml parser going haywire

I don't know what causes it yet, but there are some sites that seem to break the crawling process.
two of them are e.g.
http://www.skiandorre.com/
http://www.hi...

Hi,
Anemone do not support proxy. I create small patch to fix it.

Hi,

I've just started using Anemone and I'm trying to use it in conjunction with a HTML/CSS validator. I'm getting the following error on my output when running...

I think this makes sense if we utilize the rel=canonical tag on a page. The same script with a different query string should be considered as a different page u...

I crawl different domains though.
Is there a way to "reset" the page cache in between crawls?

Yes, although it's not really a "leak" because it is intentionally storing data about all the pages you crawl. If you crawl a lot of pages, that data has to go ...

Thanks for your patch, I've incorporated your change to parse links with xpath instead of CSS.

I don't plan to include pluggable parser support for a couple of ...

in the http.rb:

redirect_to = response.is_a?(Net::HTTPRedirection) ?  URI(response['location']): nil

redirect_to = response.is_a?(Net::HTTPRedirect...

Is it possible that, when running a lot of .crawl() operations, the @pages hash will keep on growing and there is no method that allows the user to empty it?
(u...

Anemone isn't currently able to crawl a site if the root domain redirects to another (sub)domain

an example of this would be http://heise.de which redirects to ...

It would be nice if there was a anemone.stop_crawl method that could be called from e.g. within the on_every_page block.
In my case, I simply want to stop crawl...

Here are some performance benchmarks.
Switching from CSS to xpath also results in speed boosts:

http://gist.github.com/391134

(tl;dr: hpricot (xpath) took: 3.59...

It would be nice if there was the possibility of using hpricot instead of nokogiri.
They should be API compatible.

In my tests, hpricot was always a tiny bit fa...

Resolved in 0.3.2

While running a crawl, I am sometimes getting page results from other domains. The following code:

require 'rubygems'
require 'anemone'
require 'pp'

Anemone.c...