Disabling internal services gracefully with resque, resque-scheduler, and redis_feature_control

Moved here

We've got a data warehouse that is separate from our Rails app's database. We aggregate data there and then feed summaries back into our Rails app to power all kinds of statistics for our users (impression data, traffic, etc). We heavily use Resque for our backend jobs, including pulling data from our warehouse. It works great. A user requests some data, resque serves it up. It also sends out periodic update emails which include data sourced by our warehouse. We started running into problems when we wanted to run migrations on our warehouse that took several hours. Being down during this time is not really acceptable and we didn't want to lose jobs that happened to run and depended on the warehouse being up and in a consistent state. We needed to be able to tell Resque to stop processing warehouse jobs (but still come back for them later). We also needed to be able to tell the user that this report was temporarily disabled while we upgraded (rather than timing out). The rest of the website should continue to run as usual. Basically, we needed a central place for processes to look for whether a service (in this case our warehouse) was available. This switch needed to be able to be turned off programatically (ie: during deployment of a magration) or manually (ie: via an admin tool). It also needed to be lightweight so even the tiniest script could use it. We also needed to be able to requeue jobs that needed to wait until the warehouse was back up. But we didn't want to just requeue them because they would immediately get popped again and could potentially starve lower priority jobs. We solved the first problem by coming up with redis_feature_control. Basically, a very simple on/off switch back by redis. Usage looks like this:

  # Check to see if the warehouse is supposed to be up...
  Redis::FeatureControl.enabled?(:warehouse) # => true

  # Disable the warehouse
  Redis::FeatureControl.disable!(:warehouse)

Pretty simple. We then wrapped our capistrano task that migrates our warehouse with disable/enable. We updated our Rails app to display a nice pretty "Please come back in a few minutes" message to our users instantly rather than timing out and detecting errors the hard, ugly way. And we updated our Resque jobs like so:

   def self.perform
     if Redis::FeatureControl.enabled?(:warehouse)
       # do stuff
     else
       # try again in a bit...
     end
   end

Now for the "try again in a bit" part. This was pretty easy with the resque-scheduler (you can read my previous post on it here and here). Basically replace the "try again in a bit" comment with:

  Resque.enqueue_in(1.hour, self)

Done. The job will be pushed back onto the Resque queue in an hour. If the warehouse still isn't available, it will wait another hour and so on. End result: When our warehouse is being migrated, it flags itself as being "off" and the dependent processes take the appropriate action, including delaying jobs to be processed in the future. So far, it's worked like a charm.

Introducing cached_attribute - cached values across object instances

We've been memoizing attributes manually for a long time, and since we're still on Rails 2.1, we haven't been able to use the nifty new features for memoization. I liked the way Rails 2.2 did it, though, and revisited it when we were investigating some perf issues on the site. Memoization is cool and all, but it didn't buy us anything when we had multiple instances of the same object on the same page, so that's where cached_attribute comes in. Cached_attribute is a plugin that stashes the result of an expensive calculation into a cache, indexed by the object's id. That way, multiple object instances will share the same cached value for the same attribute (on the same object). The readme has more information and an example, or you could just check out the code:

ruby script/plugin install git://github.com/avvo/cached_attribute.git

And as usual, the project is available on github.

Multiple delivery method support for ActionMailer

For our test servers (and in development), I wanted to be able to create an action that would display all the emails sent by that mongrel. When using the :test delivery method, sent emails are readily available in ActionMailer::Base.deliveries. However, for our test server I wanted ActionMailer to send the email with smtp in addition to keeping them around in memory. So here's the resulting plugin: http://github.com/bvandenbos/actionmailer_multiple_delivery_methods/tree/master

ruby script/plugin install git@github.com:bvandenbos/actionmailer_multiple_delivery_methods.git

Then in your development.rb (or environment.rb, or wherever) you can do this:

  config.action_mailer.delivery_method = [:test, :smtp]

I've tried it with Rails 2.1.0. No guarantees with any other version, or even 2.1.0 for that matter ;)

Rails hack: Adding html comments to label partials

Ever want to figure out which partial a block of html is coming from without greping through your entire rails project? Render_partial_comments is a rails plugin that renders html comments to mark the begin and end of partial output. Give it a try:

ruby script/plugin install git://github.com/avvo/render_partial_comments.git

Then enable it by adding this global in config/environments/development.rb (or wherever)

$render_partial_comments = true

Then restart your server... Your rendered html will now include html comments wrapping the output from partials. Should look something like this:

<!-- render_begin 'user/view' -->
....
<!-- render_end 'user/view' -->

Update: This may have some adverse effects on autocomplete if you use it

belongs_to :dependent => :destroy with foreign key constraints

Moved here

I discovered what I believe to be a bug in activerecord (2.1.0) today. If you have something like the following:

class Person < ActiveRecord::Base
  belongs_to :person_address, :dependent => :destroy
end

class PersonAddress < ActiveRecord::Base
  has_one :person
end

...and you have for foreign key constraint on person_address_id on the person table to id on the person_address table (which would be a reasonable thing to do) you will get a foreign key constraint error when you try to destroy a person record (if the associated person_address record exists). I submitted a patch to rails for this, but in the meantime, I found this as a work around:

class Person < ActiveRecord::Base
  belongs_to :person_address # note: no :dependent => :destroy

  def destroy
    super() # first destroy ourselves before we destroy the association
    PersonAddress.destroy(self.person_address_id) if self.person_address_id
  end
end

Private Mixins - Including helpers in controllers

Despite being fully aware that controllers should include minimal logic, we still have some code that needs to be shared by multiple controllers (and even some views). The easiest way to share code in ruby is, of course, the mixin module. The trouble in rails is that any public methods on controllers are exposed as actions, even ones that come in via mixin. That's trouble. One solution was to define all methods in our helpers as private. Didn't really work in every scenario and kind of limits testing. What we really wanted was to be able to include modules privately. As far as I can tell, ruby doesn't support this off the shelf, so we decided we give it a shot to see where it took us.

class Module
 
  [:private, :protected].each do |type|
    eval %{
      def include_#{type}(*prms)
        prms.each do |mod|
          include(mod)
          mod.instance_methods.each do |meth|
            #{type}(meth)
          end
        end
      end
    }
  end
 
end

This creates two methods include_private and include_protected. These guys wrap the normal include method but then take all the instance methods for the included module and privatize or protected-ize them, respectively. Then we added this to ApplicationController:

class ApplicationController < ActionController::Base

  class << self
    alias_method :include_helper, :include_private
  end
 
end

Now we can include helpers in our controllers privately:

class MyController < ApplicationController
  include_helper MyHelper
end

Now, it seems to me that there must be a better way of doing this and we would love to see it because our googling came up empty and it seems odd that we had to resort to this, though it seems to fit our needs just fine.