Allow root domain redirects

#23

Anemone isn't currently able to crawl a site if the root domain redirects to another (sub)domain

an example of this would be http://heise.de which redirects to http://www.heise.de

$ curl -I heise.de HTTP/1.1 301 Moved Permanently
Location: http://www.heise.de/

it would be nice to allow anemone to follow these initial redirects, maybe by setting a special parameter in the options hash?

A modification in the allowed?() method of /lib/anemone/http.rb would probably work.
This is the current one:

#
# Allowed to connect to the requested url?
#
def allowed?(to_url, from_url)
  to_url.host.nil? || (to_url.host == from_url.host)
end

It is however a design decision if this option makes sense.
What would be your take on this?

Reported by rb2k · May 6th, 2010 @ 02:33 PM

Activity

Alex Johnson
- Milestone order changed from 0 to 0
In case you or someone else is still facing this problem, the following solution can be applied.

Anemone.crawl("http://heise.de") do |anemone|
anemone.on_every_page do |page|
```
if page.code==301 
  puts Anemone.crawl(page.redirect_to)
```
This code is intended to give the logic behind the idea. on_every_page yields a page object that has the methods code and redirect_to. One can use them to crawl the redirected page accordingly.
November 18th, 2011 @ 01:51 AM

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.