Speeding up your site's assets with CloudFront

Moved here

As part of one of our regular performance passes on Avvo, we decided to give Amazon's CloudFront service a try to speed up the delivery of our images, css, and javascript. CloudFront is the CDN part of Amazon's Web Services suite, and as such, it replicates requested data it across Amazon's CloudFront servers distributed across the world.

By default, Amazon requires you to host anything distributed through CloudFront using S3, and S3 is the only option available through the web interface when configuring CloudFront. At first, this didn't seem like it would be an issue, but after trying to push our assets to S3 and hit them through CloudFront, we began to notice some problems.

Issues with using S3 as a backing store

It started with the slowness. Uploading our new assets to S3 on every deploy only took about a minute, but even a minute represents a significant portion of our total deploy time. Annoying, but still acceptable. What was less acceptable was the inablity for CloudFront and S3 to respond to clients' Accept-Encoding: gzip headers and automatically return the pre-gzip'd files we uploaded. Although there were ways to get around this issue, it was at this point that we decided to try Amazon's less-documented and less-accessible Custom Origins feature to tell CloudFront to use our site itself as its backing store. This, we hoped, would fix both the speed and gzipping issues, as well as be simpler to maintain.

Using Custom Origins

Custom Origins, although less accessible through Amazon's tools than S3 origins, seem to offer a much simpler solution for integrating CloudFront for assets that aren't already on S3. Fortunately, a few of the third-party Amazon Web Services tools already provide support for setting up a CloudFront distribution with Custom Origins. If your curl-fu is better than mine, you can hit their server directly by sending requests documented by Amazon. I prefer Ruby, though, so I tried out the newest version of RightScale's right_aws gem.

Installing RightAWS

(As far as I can tell, the right_aws gem available with gem install doesn't yet support custom origins. You can skip this part once it does).

Installing the gem was pretty easy using git and bundler:

git clone https://github.com/rightscale/right_aws.git
cd right_aws
bundle install

Of course, you could also build and install the gem yourself, if you prefer.

Setting up your Custom Origin

Once right_aws is installed, it's easy to write a script or start an irb session to set up a new CloudFront distribution:

require 'right_aws'
acf = RightAws::AcfInterface.new('aws public key', 'aws secret key')
acf.create_distribution(
  :comment => "My Remote Origin",
  :custom_origin => {
    :dns_name => "www.myserver.com",
    :http_port => 80,
    :https_port => 443,
    :origin_protocol_policy => 'match-viewer',
  },
  :enabled => true
)

The only tricky part of the above is the origin_protocol_policy parameter. This parameter can take one of two values, http-only or match-viewer. This specifies whether https requests to CloudFront assets will make https requests to your custom origin. Amazon suggests that http-only should be used unless there's a particular reason to use match-viewer.

At this point, you should be able to see your new distribution in the CloudFront console and hit your assets through the host they give you. There's still some more work to do to make sure your assets are delivered as quickly as possible, but that's another post that'll be written another time.

Introducing the Avvo API

Moved here

As we announced on the Avvo Blog, we're formally introducing the Avvo API. This means that developers can integrate with the data in our professional directory in a way that makes sense for their sites and applications. The API is RESTful, and returns all responses in JSON, so it should be simple to integrate into your own apps.

For Ruby developers, we went one step further—we released a gem that makes it even simpler to integrate the information on Avvo into your own sites and applications. The Avvo API gem is available on github and can be installed easily into your own apps.

Installing the gem is as simple as

gem install avvo_api

If you use Bundler in your app, you just need to put the following in your Gemfile:

source "http://rubygems.org"
gem 'avvo_api', "~>0.1"

And you should be ready to go. It's easy to get started with the API—all you need is an Avvo account and an API key from Josh King. Here's a short example:

require 'avvo_api'
AvvoApi.setup('user@avvo.com', 'password')
lawyer_id = AvvoApi::Lawyer.search(:q => 'tax', :loc => 'Seattle, WA')["results"].first["id"]

lawyer = AvvoApi::Lawyer.find(lawyer_id)

review = lawyer.reviews.first
puts "#{review.overall_rating} - #{review.title}"    

For more documentation and examples, take a look at the Readme on github and the rdoc built from the gem. The API gem uses another project, Reactive Resource, which you should check out if you're wrapping other REST APIs with ActiveResource.

You can find full documentation and try the API on the API documentation site. We're excited to see what you come up with!

resque-scheduler 2.0.0 pre

Moved here

I just push resque-scheduler 2.0.0.a.

  gem install resque-scheduler --pre

It contains some interesting changes from brianjlandua and davidyang. The gist of it is, you can now schedule items dynamically. Here's what it looks like: Let's say you want your users to configure when some report is generated. You can do something like this:

  Resque.set_schedule('user_16_report', {
    :cron => "0 1 * * *",
    :queue => "reports",
    :class => "GenerateUserReport",
    :args => 16})

That creates a scheduled job and takes effect immediately. This is accomplished by storing the schedule state in a redis hash (so, yes, >=1.3 redis is required) and marking changes in an ordered set. The schedule process looks for changes in the ordered set and then applies them during each loop. The scheduler tab in resque-web will also pull the schedule from redis, so you can see what's currently scheduled. This behavior is not default. You need to set this (for the scheduler process and resque-web):

  Resque::Scheduler.dynamic = true

By default, resque-scheduler behaves as it always has (other than storing the schedule in redis). This functionality just opens up new possibilities for scheduling jobs.

Ummm.. That's What She Said

Moved here

Ok, so we were joking at work the other day about whipping up a "that's what she said" Bayes classifier. Someone pointed out that you could use a Twitter search for #twss to train it. A couple hours later I found my self on the bus coding it up. The result is the twss gem. Check it out on github. Usage looks something like this:

    requre 'twss'
    TWSS("Have you checked out that file?") # => false
    TWSS("Yeah... I think it's too big") # => true

The classifier and twitter gems do all the heavy lifting. Yes, I realize this is childish, lowbrow humor, but I just couldn't resist seeing if it would work. Turns out it works enough to be funny. Expect an IRC bot soon :) UPDATE: IRC bot can be found here

Disabling internal services gracefully with resque, resque-scheduler, and redis_feature_control

Moved here

We've got a data warehouse that is separate from our Rails app's database. We aggregate data there and then feed summaries back into our Rails app to power all kinds of statistics for our users (impression data, traffic, etc). We heavily use Resque for our backend jobs, including pulling data from our warehouse. It works great. A user requests some data, resque serves it up. It also sends out periodic update emails which include data sourced by our warehouse. We started running into problems when we wanted to run migrations on our warehouse that took several hours. Being down during this time is not really acceptable and we didn't want to lose jobs that happened to run and depended on the warehouse being up and in a consistent state. We needed to be able to tell Resque to stop processing warehouse jobs (but still come back for them later). We also needed to be able to tell the user that this report was temporarily disabled while we upgraded (rather than timing out). The rest of the website should continue to run as usual. Basically, we needed a central place for processes to look for whether a service (in this case our warehouse) was available. This switch needed to be able to be turned off programatically (ie: during deployment of a magration) or manually (ie: via an admin tool). It also needed to be lightweight so even the tiniest script could use it. We also needed to be able to requeue jobs that needed to wait until the warehouse was back up. But we didn't want to just requeue them because they would immediately get popped again and could potentially starve lower priority jobs. We solved the first problem by coming up with redis_feature_control. Basically, a very simple on/off switch back by redis. Usage looks like this:

  # Check to see if the warehouse is supposed to be up...
  Redis::FeatureControl.enabled?(:warehouse) # => true

  # Disable the warehouse
  Redis::FeatureControl.disable!(:warehouse)

Pretty simple. We then wrapped our capistrano task that migrates our warehouse with disable/enable. We updated our Rails app to display a nice pretty "Please come back in a few minutes" message to our users instantly rather than timing out and detecting errors the hard, ugly way. And we updated our Resque jobs like so:

   def self.perform
     if Redis::FeatureControl.enabled?(:warehouse)
       # do stuff
     else
       # try again in a bit...
     end
   end

Now for the "try again in a bit" part. This was pretty easy with the resque-scheduler (you can read my previous post on it here and here). Basically replace the "try again in a bit" comment with:

  Resque.enqueue_in(1.hour, self)

Done. The job will be pushed back onto the Resque queue in an hour. If the warehouse still isn't available, it will wait another hour and so on. End result: When our warehouse is being migrated, it flags itself as being "off" and the dependent processes take the appropriate action, including delaying jobs to be processed in the future. So far, it's worked like a charm.

Added Resque.enqueue_in for delayed resque jobs

Moved here

Last week I added support for delayed jobs in resque to the resque-scheduler gem. You can now do stuff like this:

  Resque.enqueue_in(5.days, SendEmailFollowup, :user_id => current_user.id)

And in 5 days, the job will be queued for work as if you called Resque.enqueue directly. Alternatively, you can call enqueue_at and pass it a Time instance (or unix timestamp as an int) instead of a number of seconds. There's also an additional tab in resque-web for viewing delayed jobs. For screen shots and more details, check it out on github. If you're interested in calling jobs on a recurring basis, checkout my previous post on using resque-scheduler to replace cron jobs that just call ruby scripts.

Introducing resque-scheduler

Moved here

At Avvo, we've got a lot of backend ruby/rails jobs running as crons. Jobs to refresh leaderboards, jobs to warm caches, jobs to pull data from third parties. The list is long and distinguished. These jobs run on different machines at different times in different intervals. Up until recently, we've been managing it all with puppet. We guess at where we have spare cycles to run a job and on which machine. We've ignored the problem long enough that now we have cron configuration in a couple dozen files running all different sorts of things. Very annoying and difficult to keep track of. Just a few weeks ago, we integrated Resque and started porting over our jobs. Having distributed, generic workers is awesome. It's another thing we've long put off because I couldn't find just the right fit. Resque is that fit. Only one thing was missing: A way to add things to queues based on a schedule. What I wanted in addition to job processing is job scheduling to replace our gagillion cron jobs. So we hacked up resque-scheduler. Resque-scheduler is a gem that extends resque to support scheduled jobs. Installation (hosted on gemcutter.org):

  gem install resque-scheduler

Resque-scheduler takes the schedule as a hash, which is easily represented in YAML...

queue_documents_for_indexing:
  cron: "0 0 * * *"
  class: QueueDocuments
  args: 
  description: "This job queues all content for indexing in solr

clear_leaderboards_contributors:
  cron: "30 6 * * 1"
  class: ClearLeaderboards
  args: contributors
  description: "This job resets the weekly leaderboard for contributions"

clear_leaderboards_moderator:
  cron: "30 6 * * 1"
  class: ClearLeaderboards
  args: moderators
  description: "This job resets the weekly leaderboard for moderators"

... that can be set in your resque initializer like so:

  require 'resque-scheduler'
  ResqueScheduler.schedule = YAML.load_file(File.join(File.dirname(__FILE__), '../resque_schedule.yml'))

Then to run the scheduler process, a simple rake task:

  $ rake resque:scheduler

This process idles until a schedule item fires, then it stuffs it into the appropriate queue. See http://github.com/bvandenbos/resque-scheduler for the complete details. Obviously, you need to be using resque to take advantage of resque-scheduler For the future, I'd like to find a clean way to extend resque-web to display the schedule and allow users to click a button to manually queue items in the schedule. Many thanks to resque (defunkt) and rufus-scheduler (jmettraux) which are doing all the heavy lifting.