Idle Hacking

I've made a few more improvements to curb and decided keeping track of all these patches is too much. I'm now tracking my changes on github, and hopefully the curb author will reappear to review the changes soon, so we can get them released in gem form.

I added a new callback hook to curb on_success and on_failure. They work as advertised and are esspecially useful when using the Curl::Multi interface from my last patch.

gc = Curl::Easy.new("http://www.google.com/")
gc.on_success{|curl| puts curl.body_str }
yc = Curl::Easy.new("http://www.yahoo.com/")
yc.on_success{|curl| puts curl.body_str }
   
mc = Curl::Multi.new
mc.add(gc)
mc.add(yc)
   
mc.perform

What's nice about this, is imagine you have some process running and want to add requests to it. When you add the request you also want to register a callback to handle the success or failure cases. Before this patch you'd have to check the content length and listen to the on_body handler. The problem with that, is the content length header is not always correct. These callbacks are guaranteed, because they happen during the cleanup process for each handle.

As an aside they'll work using the Easy handle diretly.

Here's the patch, enjoy!

Vendoring gems definitely makes deployment easier for both rails and merb. For merb, the solution is to create a local gem repository within your project.

  gem install -i gems/ --no-ri --no-rdoc #{gem_to_install}

This works great, once you update your merb/config/init.rb to include the new gem repository.

Gem.clear_paths
Gem.path.unshift(Merb.root / "gems")

I also recommend saving space by deleting the gem cache.

rm -rf gems/cache/*

This creates one problem, if the gems you have installed include binaries (e.g. C extensions), then you'll have an issue if you develop and deploy to different architectures. I think the most common would be developing on Mac OSX and deploying to Linux. There are many different ways to solve this problem. One might use gem pristine, or as I do below maintain a list of the gems that need compiling and rebuild them for the deployment target.

GEM_BUILDS=['rbtagger','hpricot','mongrel','fastthread']
GEM_ROOT=File.join(File.dirname(__FILE__),'gems')

desc 'Refresh gem builds for the current system'
task :build_refresh do
  gem_dir_list = File.join(GEM_ROOT,'gems')
  gems_built = GEM_BUILDS
  Dir["#{gem_dir_list}/*"].each do|gem_path|
    gems_built.each do|gem|
      if gem_path.match(gem)
        if File.exist?("#{gem_path}/Rakefile")
          puts "building #{gem_path}"
          system("cd #{gem_path} && rake compile")
        else
          base = File.basename(gem_path)
          version = base.split('-').last
          cmd = "gem install -i gems --no-ri --no-rdoc #{gem} --version #{version}"
          puts cmd.inspect
          system(cmd)
        end
        gems_built.reject!{|g| g == gem}
      end
    end
  end
end

Now to run the task:

rake build_refresh

To change the list of compiled gems just modify the GEM_BUILDS constant.

I've updated my patch for curb multi interface and tested against valgrind for memory leaks. So far the patch looks stable. I'm still waiting to hear from the curb author. In the meantime here's how to apply the patch if you're interested.

Check out curb trunk

  svn co svn://rubyforge.org/var/svn/curb/TRUNK/curb

Grab my latest patch

  wget http://taf2-patches.s3.amazonaws.com/curb-multi.patch

Apply the patch

  patch -p0 < curb-multi-3.patch

Rebuild and test curb

  rake test

Package and install

  rake package
  sudo gem install pkg/curb-0.2.0.gem

In the patch I updated the curb version number. This may of course change, based on whether the patch is accepted or not. In the meantime, would be great to get a few people to try the patch and provide feedback and or bugs. Thanks!

Update (example usage):

  responses = {}
  requests = ["http://www.google.co.uk/", "http://www.ruby-lang.org/"]
  m = Curl::Multi.new
  # add a few easy handles
  requests.each do |url|
    responses[url] = ""
    c = Curl::Easy.new(url) do|curl|
      curl.follow_location = true
      curl.on_body{|data| responses[url] << data; data.size }
    end
    m.add(c)
  end

  m.perform do
    puts "idling... can do some work here, including add new requests"
  end

  requests.each do|url|
    puts responses[url]
  end

Here's a performance comparison using ruby's native threads vs curb multi interface to make up to 400 concurrent requests. You can see a huge pay off in performance at the 50 concurrent requests.

This morning I discovered Evan's patch for valgrind is missing. I rebuilt the patch. As well here are the bits below:

Index: configure.in
===================================================================
--- configure.in (revision 17858)
+++ configure.in (working copy)
@@ -377,6 +377,10 @@
  : handled by ext/thread/extconf.rb
         ])
 
+AC_ARG_WITH(valgrind,
+        [  --with-valgrind         enable valgrind memcheck support.],
+        [AC_CHECK_HEADERS(valgrind/memcheck.h)])
+
 dnl Checks for libraries.
 case "$target_os" in
 nextstep*) ;;
Index: gc.c
===================================================================
--- gc.c (revision 17858)
+++ gc.c (working copy)
@@ -37,6 +37,19 @@
 void re_free_registers _((struct re_registers*));
 void rb_io_fptr_finalize _((struct rb_io_t*));
 
+#ifdef HAVE_VALGRIND_MEMCHECK_H
+# include 
+# ifndef VALGRIND_MAKE_MEM_DEFINED
+#  define VALGRIND_MAKE_MEM_DEFINED(p, n) VALGRIND_MAKE_READABLE(p, n)
+# endif
+# ifndef VALGRIND_MAKE_MEM_UNDEFINED
+#  define VALGRIND_MAKE_MEM_UNDEFINED(p, n) VALGRIND_MAKE_WRITABLE(p, n)
+# endif
+#else
+# define VALGRIND_MAKE_MEM_DEFINED(p, n) /* empty */
+# define VALGRIND_MAKE_MEM_UNDEFINED(p, n) /* empty */
+#endif
+
 #define rb_setjmp(env) RUBY_SETJMP(env)
 #define rb_jmp_buf rb_jmpbuf_t
 #ifdef __CYGWIN__
@@ -668,6 +681,7 @@
     VALUE v;
     while (n--) {
         v = *x;
+        VALGRIND_MAKE_MEM_DEFINED(&v, sizeof(v));
  if (is_pointer_to_heap((void *)v)) {
      gc_mark(v, 0);
  }

I spent the day working on adding multi interface support to curb. I really like the curb ruby interface and with a little work can now use the same Curb::Easy interface and run the handles through the asynchronous Curb::Multi interface. The patch hasn't been rigorously tested, but hey not bad for my first day.
Here's the patch