Introducing resque-scheduler

Moved here

At Avvo, we've got a lot of backend ruby/rails jobs running as crons. Jobs to refresh leaderboards, jobs to warm caches, jobs to pull data from third parties. The list is long and distinguished. These jobs run on different machines at different times in different intervals. Up until recently, we've been managing it all with puppet. We guess at where we have spare cycles to run a job and on which machine. We've ignored the problem long enough that now we have cron configuration in a couple dozen files running all different sorts of things. Very annoying and difficult to keep track of. Just a few weeks ago, we integrated Resque and started porting over our jobs. Having distributed, generic workers is awesome. It's another thing we've long put off because I couldn't find just the right fit. Resque is that fit. Only one thing was missing: A way to add things to queues based on a schedule. What I wanted in addition to job processing is job scheduling to replace our gagillion cron jobs. So we hacked up resque-scheduler. Resque-scheduler is a gem that extends resque to support scheduled jobs. Installation (hosted on gemcutter.org):

  gem install resque-scheduler

Resque-scheduler takes the schedule as a hash, which is easily represented in YAML...

queue_documents_for_indexing:
  cron: "0 0 * * *"
  class: QueueDocuments
  args: 
  description: "This job queues all content for indexing in solr

clear_leaderboards_contributors:
  cron: "30 6 * * 1"
  class: ClearLeaderboards
  args: contributors
  description: "This job resets the weekly leaderboard for contributions"

clear_leaderboards_moderator:
  cron: "30 6 * * 1"
  class: ClearLeaderboards
  args: moderators
  description: "This job resets the weekly leaderboard for moderators"

... that can be set in your resque initializer like so:

  require 'resque-scheduler'
  ResqueScheduler.schedule = YAML.load_file(File.join(File.dirname(__FILE__), '../resque_schedule.yml'))

Then to run the scheduler process, a simple rake task:

  $ rake resque:scheduler

This process idles until a schedule item fires, then it stuffs it into the appropriate queue. See http://github.com/bvandenbos/resque-scheduler for the complete details. Obviously, you need to be using resque to take advantage of resque-scheduler For the future, I'd like to find a clean way to extend resque-web to display the schedule and allow users to click a button to manually queue items in the schedule. Many thanks to resque (defunkt) and rufus-scheduler (jmettraux) which are doing all the heavy lifting.

Getting ruby, rubygems, rake, mysql, imagemagick, rmagick, aspell, raspell with fink

Moved here

This is really just for me to remember this. If at some point someone else benefits from this, even better.

1. Download and Install fink
2. Edit /sw/etc/apt/sources.list and add the following:

   deb http://fink.sodan.ecc.u-tokyo.ac.jp/apt/10.5 unstable main crypto
   deb http://sage.ucsc.edu/fink_intel_10.5_only    stable main crypto
   deb http://sage.ucsc.edu/fink_intel_10.5_only    unstable main crypto
   

3. Install ruby:

   sudo apt-get install ruby
   sudo apt-get install ruby18-dev
   sudo apt-get install ruby18-shlibs
   

4. Install rubygems:

   sudo apt-get install rubygems-rb18
   

5. Update rubygems:

   sudo gem update --system
   sudo gem install rubygems-update # this is to get passed v 1.1.1
   sudo update_rubygems
   

6. Install rake

   sudo gem install rake
   

7. Then install mysql:

   sudo apt-get install mysql
   sudo apt-get install mysql-client
   sudo apt-get install mysql15-dev
   sudo gem install mysql
   

8. If you want to use a custom my.cnf, plunk it in /etc
9. Then install imagemagick and rmagick (this one can be a pain)

   sudo apt-get install imagemagick10-dev imagemaigck10-shlibs
   sudo apt-get install freetype freetype-shlibs ghostscript ghostscript-fonts \
    gv libpng-shlibs libjpeg libjpeg-bin libjpeg-shlibs lcms lcms-bin lcms-shlibs \
    libtiff libtiff-bin libtiff-shlibs
   sudo gem install rmagick -v 2.9.2
   

10. Install aspell:

    sudo apt-get install aspell
    sudo apt-get install aspell-en
    sudo apt-get install apsell-dev
    sudo gem install raspell
    

11. Install other gems:

    sudo gem install hoe -v 2.0.0
    sudo gem install nokogiri
    sudo gem install rake-compiler
    sudo gem install taka
    sudo gem install johnson -v 1.1.0 # 1.1.1 wouldn't install
    

Introducing cached_attribute - cached values across object instances

We've been memoizing attributes manually for a long time, and since we're still on Rails 2.1, we haven't been able to use the nifty new features for memoization. I liked the way Rails 2.2 did it, though, and revisited it when we were investigating some perf issues on the site. Memoization is cool and all, but it didn't buy us anything when we had multiple instances of the same object on the same page, so that's where cached_attribute comes in. Cached_attribute is a plugin that stashes the result of an expensive calculation into a cache, indexed by the object's id. That way, multiple object instances will share the same cached value for the same attribute (on the same object). The readme has more information and an example, or you could just check out the code:

ruby script/plugin install git://github.com/avvo/cached_attribute.git

And as usual, the project is available on github.

Twitterss - RSS to Twitter script that can handle multiple accounts

I know there are a bunch of RSS to Twitter scripts out there in a variety of languages. There's even twitterfeed which manages all your feeds for you (which is a pretty cool service). But, I couldn't find one that did all of the following:

1. Took updates as frequently as I wanted
2. Could handle multiple RSS feeds to multiple Twitter accounts (ie: could be configured)
3. Didn't require handing out my Twitter password (not that this is always a bad thing)
4. Didn't have a bunch of dependencies (activerecord, mysqlite, etc...)
5. Preferably written in ruby

So, I hacked twitterss together. Twitterss is just a script that takes a YAML configuration file as a paramter which is meant to be used as a cron job:

  $ twitterss twitterss.yml

twitterss.yml might look something like this:

legalnews:
  rss: http://www.avvo.com/news.rss
  login: avvolegalnews
  password: secret

cnn_money_top_stories:
  rss: http://rss.cnn.com/rss/money_topstories.rss
  login: money_news
  password: secret
  max: 5

cnn_money_markets:
  rss: http://rss.cnn.com/rss/money_markets.rss
  login: money_news
  password: secret
  max: 3

This config would make twitterss pull items from the CNN Money Top Stories and the CNN Money Markets feeds and push them the the money_news Twitter account. It would also pull from the Avvo Legal News feed and push to avvolegalnews. Example result: To make sure it doesn't post the same link twice, it keeps a list of each link it posts for each config entry in a local file (<USER_HOME>/.twitterss_history).

Private Mixins - Including helpers in controllers

Despite being fully aware that controllers should include minimal logic, we still have some code that needs to be shared by multiple controllers (and even some views). The easiest way to share code in ruby is, of course, the mixin module. The trouble in rails is that any public methods on controllers are exposed as actions, even ones that come in via mixin. That's trouble. One solution was to define all methods in our helpers as private. Didn't really work in every scenario and kind of limits testing. What we really wanted was to be able to include modules privately. As far as I can tell, ruby doesn't support this off the shelf, so we decided we give it a shot to see where it took us.

class Module
 
  [:private, :protected].each do |type|
    eval %{
      def include_#{type}(*prms)
        prms.each do |mod|
          include(mod)
          mod.instance_methods.each do |meth|
            #{type}(meth)
          end
        end
      end
    }
  end
 
end

This creates two methods include_private and include_protected. These guys wrap the normal include method but then take all the instance methods for the included module and privatize or protected-ize them, respectively. Then we added this to ApplicationController:

class ApplicationController < ActionController::Base

  class << self
    alias_method :include_helper, :include_private
  end
 
end

Now we can include helpers in our controllers privately:

class MyController < ApplicationController
  include_helper MyHelper
end

Now, it seems to me that there must be a better way of doing this and we would love to see it because our googling came up empty and it seems odd that we had to resort to this, though it seems to fit our needs just fine.