Idle Hacking

I've updated my patch for curb multi interface and tested against valgrind for memory leaks. So far the patch looks stable. I'm still waiting to hear from the curb author. In the meantime here's how to apply the patch if you're interested.

Check out curb trunk

  svn co svn://rubyforge.org/var/svn/curb/TRUNK/curb

Grab my latest patch

  wget http://taf2-patches.s3.amazonaws.com/curb-multi.patch

Apply the patch

  patch -p0 < curb-multi-3.patch

Rebuild and test curb

  rake test

Package and install

  rake package
  sudo gem install pkg/curb-0.2.0.gem

In the patch I updated the curb version number. This may of course change, based on whether the patch is accepted or not. In the meantime, would be great to get a few people to try the patch and provide feedback and or bugs. Thanks!

Update (example usage):

  responses = {}
  requests = ["http://www.google.co.uk/", "http://www.ruby-lang.org/"]
  m = Curl::Multi.new
  # add a few easy handles
  requests.each do |url|
    responses[url] = ""
    c = Curl::Easy.new(url) do|curl|
      curl.follow_location = true
      curl.on_body{|data| responses[url] << data; data.size }
    end
    m.add(c)
  end

  m.perform do
    puts "idling... can do some work here, including add new requests"
  end

  requests.each do|url|
    puts responses[url]
  end

Here's a performance comparison using ruby's native threads vs curb multi interface to make up to 400 concurrent requests. You can see a huge pay off in performance at the 50 concurrent requests.