I plan to add an attr_accessor for Page body as part of ticket #16.

To have Anemone visit img src's, use focus_crawl().

Indeed, this is an issue with installing nokogiri, which anemone depends on.

I'm new to Anemone and would like to ask how to get it to save all files who's URI matches a regular expression, eg. PDF files.

The anemone should support multiple encoding, it involves:
1. URL like: http://www.china.com/tag/中国 should be supported
PS: URI(str_uri) will throw exception fo...

I'm writing a simple spider to save bits of "http://api.rubyonrails.org". The top level page is a frameset, and links to a (nonexistent) HTML-only bit. It'd be ...

Actually, it should probably be attr_reader rather than attr_accessor. Changing the body makes questionable sense in the first place, and doing it without chang...

I'd like to use Anemone to mirror chunks of multiple web sites, and crawl to do it. I don't see any good way to get the actual page contents as text (rather tha...

It would be great if anemone can handle canonical URLs.

For example, if we enable canonicalization,

  http://www.example.com/index.html
  http://www.example.co...

on a fresh ubuntu install 32bit

gem install anemone

Building native extensions. This could take a while...
ERROR: Error installing anemone:

ERROR: Failed to bu...

Hi,
I have forked anemone from github, and add HTTP authentication, accessor for html body (needed for my project).
http://github.com/spk/anemone/commit/420ff3...

Hi,

Anemone ignores silently the username and password from a given URL. I've made a small patch to fix it.

Since Anemone limits the crawl to a single domain, it won't switch over to your www subdomain after the redirect. You'll need to start on the domain you intend ...

0.2.3 includes a bugfix for this issue. Give it a try, feel free to reopen the ticket if you are still experiencing issues.

I have a similar experience of this ticket.

Throwing exception from my code in result.

Please add following flag to your code and retry.
If error in your code, ...

I get this error when I try to crawl URLs from a site. What can cause this? I'm running ruby 1.8.7 (2009-06-12 patchlevel 174) [i686-darwin10] on Mac OS X 10.6....

Doesn't handle redirection properly.

irb(main):171:0 Anemone.crawl("http://treshna.com/") do |a|
irb(main):172:1 a.on_every_page do |x|
irb(main):173:2* pp x
ir...

This ticket looks fixed at anemone 0.2.0.

thanks!

I suggest adding feature to img tag tracking.

• add :follow_img_tag option.
• following img tag if :follow_img_tag => true.
• add body attribute to Page to get raw h...

I suggest to add new function to anemone.

Problem
Anemone 0.1.2 follow link in same domain. In some of the cases of root URL, Anemone crawl too many pages, and ...

Fixed the gemspec to work with GitHub. Decided not to distribute a gem through GitHub (using RubyForge only) to avoid confusion.

Thanks for the patch! I've applied this and pushed it to the GitHub repo.

Your fix looks good.

But, following test is failed.

.F.FF.

1)
'Anemone::Page should store the response headers when fetching a page' FAILED
expected nil? to r...

Unittest of anemone 0.1.2 is written by RSpec.
but test by autotest (included ZenTest) is failed.

[~/git/anemone]> autospec
loading autotest/rspec
/opt/local/b...

I suggest allowing customize queue.

Current implementation page/link queue is instance of Queue object.
But exist following use case.

• use Message Queuing softw...

Hi,

I've implemented time delay functionality in the latest release of Anemone (0.1.2). You can simply specify a :delay option when starting the crawl, like so:

sorry my broken comment......

Term "reserved" in patch causes confusion. "Delay" is felicity term.
I attached the new patch.

if apply this patch,this code behav...

Term "reserved" in patch causes confusion.
"Delay" is felicity term.

I attached the new patch.

if apply this patch,this code behavior improve.

require 'anemone'
...

patch attatched

add on_pre_fetch handler.

if this block return false,link is not fetched and re-enq to link_queue.

anemone is too violent to crawl any web server.

example

require 'anemone'
Anemone.crawl("http://www.yahoo.co.jp/") do |anemone|
  anemone.on_every_page do |pag...

Just realized this last night myself. Thanks for the patch! I've applied it, and this fix is being released in version 0.1.1

Here is the patch correcting this.

With this url :

http://www.mysite.com/a_page.html?param1=foo&param2=bar

Anemone will do a request for this url:

http://www.mysite.com/a_page.html

Which can ...

Current gemspec doesn't work with GitHub's SAFE=3 environment. Disabling GitHub gem support for the time being.

This functionality is covered by the new focus_crawl() method.

If you install via github you only get the bin directory. Without the lib directory, the gem doesn't work.

Steps to reproduce:

• sudo gem install chriskite-anemo...

Something that tripped me up when I first tried to use anemone is to do:

bin/anemone_url_list.rb http://paulbarry.com

And I got back no results. I tracked it ...

Thanks for the patch. Applied and released in 0.0.6.

Here's the patch in a gist, lighthouse's S3 integration seems to be flaky.

http://gist.github.com/140602

It would be nice to be able to skip crawling pages based on more than just the URL matching a regex (skip_links_like). Something like this:

anemone.skip_pages_...

It would be great to be able to do this:

anemone.on_every_page do |page|
  page.h1 = (page.body).at('h1').inner_html
end

Probably this can be accomplished by ...