The thread body of the tentacle looks like this:
def run
loop do
link, referer, depth = @link_queue.deq
break if link == :END
@http.fetch_pages(link, referer, depth).each { |page| @page_queue << page }
delay
end
end
Theoretically, we are protected from exceptions because
fetch_pages has a catch-all rescue...
def fetch_pages(url, referer = nil, depth = nil)
begin
url = URI(url) unless url.is_a?(URI)
pages = []
get(url, referer) do |response, code, location, redirect_to, response_time|
pages << Page.new(location, :body => response.body.dup,
:code => code,
:headers => response.to_hash,
:referer => referer,
:depth => depth,
:redirect_to => redirect_to,
:response_time => response_time)
end
return pages
rescue => e
if verbose?
puts e.inspect
puts e.backtrace
end
return [Page.new(url, :error => e)]
end
end
For the most part, you are safe.
Except in the case of Timeout::Error, which derives from
Interrupt, not StandardError, and therefore bypasses the rescue
block!
The solution is to explicitly capture it with an additional
rescue Timeout::Error => e.