The Avvo Developer Blog

An Event Apart - Seattle (Day 1, part 2)

2011-04-11T15:45:00.000-07:00

Moved here

Web Typography

Jason Santa Maria, self-described font geek, gave a talk on fonts in general and the huge opportunities sites like TypeKit give for improving the look and feel of your own sites. As recently as three years ago, we were still stuck using the 18 Web Fonts or hacks like sIFR to get real fonts on websites. Publications like the New Yorker, who are instantly recognizable from the typefaces they use, have had a hard time carrying that brand identity to their websites. Many have resorted to rendering each headline in the correct font into images, creating a maintenance nightmare. With the ability to use things like font-face, existing publications can carry their brand identity with them to the web, and new publications can enhance their own identity with their font selections. Even simple things like the ability to use condensed fonts on the web has totally changed the way well-designed sites look.

How do I choose the right typeface?

There is a difference between display typefaces, which are most useful for headlines and titles, and text typefaces, which are best for body text. A display typeface like Bodoni can be great for an article's title, but can be difficult to read in a smaller size. Most people will have the best bet finding a few good fonts with many faces, like FF Meta or Proxima that can be used successfully in many different scenarios. Good typography is about building contrast, texture, and feeling on the page that help with comprehension and enjoyment of the thing being read. "Good typography is invisible."

When using typography, there are a few guidelines that can be useful, but should be broken when appropriate:

All things being equal, err on the side of the text size being too big
Contrast is the most important tenet of good design - make parts of the text that differ in some way look different. Play with size, weight, coler, etc. to build the right amount of contrast.
Line spacing should be directly proportional to the line length and color of the font

There are lots of things to consider when picking the right fonts, but there are ways to help choose among them. Noting the qualities you want your site and your content to convey will help you narrow long lists of fonts down. Finding fonts that come in both serif and sans serif faces can make pairing simpler and will help create contrast where it's needed. Fonts that have the same feel as a commonly used font can help you pick similar but more interesting fonts. And finally, trying different fonts out by setting some text in a range of typefaces and playing with them will almost always be really helpful. Jason left us with a few sites that use web typography well: Lost World's Fairs and Voltage: Fashion Amplified.

Why Designers Fail

Scott Berkun was next, with a talk about design failures and how to learn from them. Mindful practice and studying other people are the two primary ways people learn new things, and studying failure in particular is an incredible untapped resource of knowledge. In the design world, it's difficult to find good analysis of design failures. Especially in fields like architecture, where highly in-demand designers tend to move on to new projects frequently, it's sometimes difficult for these designers to see and hear the feedback that only comes from using the product. How do you avoid falling into common design traps? Understand that all designers fail most of the time (whether it's on the drawing board or in the real world), own the mistakes that you do make, analyze the cause (whether it's setting the wrong goals or failing to meet them), and studying common causes of failure and possible mitigations.

Scott showed an amazing video of designer Matt Wiley laying out a magazine article, and how many intermediate failure states he went through in the process of coming up with a great design. These are the things that many people don't see when they look at a finished product. There's a constant process of experimenting, failing, and revising that is absolutely critical to the creative process, but isn't visible at all in the final product.

So why do designers fail? Some of it comes from falling into common traps: problem-solving for its own sake (Puzzle traps), obsessively categorizing (Category traps), confusing measurements with reality (Number traps), and loving the sketch more that what it actually represents (Drawing traps). Some of it is related to skill: there's often a mismatch between the skills that designers have and the skills that an organization needs, especially around persuading decision-makers that good ideas are good ideas. Designers should be "ambassadors for good ideas", and that involves knowing how to sell these ideas to decision makers. Luckily, persuasion, like most skills, can be learned, and it is something that designers should take the time to learn.

Scott did some research to discover which of these failures were the most common and the most important. By far, the biggest overall issue was organizational: "Non-designers making design decisions." Others were psychological: Not seeking enough data before designing, and not being receptive to feedback. Less common and important were skill-related issues, with the top being lack of awareness of business fundamentals, and lack of idea-pitching skills. Scott publicly posted his data, which is really worth a look, and his slides are also available. Hopefully, being able to identify these causes of failure will help us, as a profession, avoid them in our future work.

An Event Apart - Seattle (Day 1, part 1)

2011-04-04T14:50:00.000-07:00

Moved here

I went to An Event Apart Seattle last week and it was the best conference I've ever been to. I've started to gather my notes together and will, over the next few posts, talk about what I was able to take away from each of the presentations. Before I go in detail on the talks, though, I wanted to mention a few themes that seemed to flow through multiple presentations during the conference.

HTML5 and CSS3 are ready now.

This theme was the most related to the work I do on a daily basis. Every talk that touched web development explored features that weren't available in every browser (one talk was built around them), and it seems that the web design community has finally abandoned the idea that sites should act the same way in every browser. Most of the speakers accepted that sites should still work in some way in every browser, but it seemed accepted as given that we should start using HTML5 and CSS3 features tastefully to enhance the experience of people using modern browsers (and especially modern mobile browsers).

Context is everything

Whether it was talking about simplicity as delivering the correct experience to the correct user or pairing the correct font or design or architecture or business goal to the correct content or tailoring a mobile web application to fit the behavior of a user who just wants to get the information they need and get on with their life, context was brought up constantly. Building the right context for the job you're trying to do involves asking pointed, well-thought-out questions, and several of the presentations had slide after slide of thought-provoking questions intended to bring the assumed context out into the open.

The things we have been talking about as important before, are now critical

From the data some of the presentations revealed, we are beginning to hit a critical moment in many areas the web touches. Mobile is the most obvious example: Luke Wroblewski was fond of mentioning that smartphone shipments overtook PC shipments a full two years earlier than expected. There were others, too: the rise of HTML5 and CSS3, the ability of sites like TypeKit to drastically help increase your site's branding, and the growth of content strategy as a discipline to deliver the right information to the right people at the right time. These things that were important before are now becoming necessary, because people are beginning to expect great experiences from the sites they visit, and if they can't get it from your site, they will get it somewhere else.

A whirlwind tour of the history of the web

Jeffrey Zeldman started the conference with a fast and funny history of the web. Beginning with Movable Type (Gutenberg, not Six Apart), passing through AOL, the browser wars, an unfortunately de-pixelated Zeldman on the cover of "Designing with Web Standards", and the fall of XHTML, and finishing up with mobile browsing, HTML5/CSS3, web fonts, and the IE9 beta, it did a fantastic job of setting the context for the rest of the conference. Today, thanks to the widespread adoption of web standards, the web has grown up. We can now make use of skills that were not previously useful online to create a better web experience for everyone. And since the web is now everywhere, those experiences can affect people beyond what was previously possible.

Better user experiences through psychology

Sarah Parmenter was next, giving a talk called "Crafting User Experiences." Sarah discussed how well-known psychological principles could be used to build great user experiences. I noticed a lot of parallels to one of my favorite recent books, Predictably Irrational (which should be next on your list if you haven't yet had the opportunity to read it), and I found it really interesting that a lot of this knowledge that used to seem to be confined to sales and marketing is now making its way into mainstream web design.

Snap judgements and sensation transference were the first topics hit. We usually deny that snap judgements are as important as they are, since we tend to thing that "the quality of a decision is directly related to the time and effort that went into making it." In reality, judgements made in under 5 minutes are drastically more important to our final decisions than we realize. This is related to to sensation transference: our judgement of the packaging of a product can have real effects on our perception of the actual product. For example, adding yellow coloring to 7up made people percieve more lime taste while drinking it. This led to the first appearance of the Dark Patterns wiki during the conference. Like any skill, though, these skills can be used for good as well as evil.

Better user experiences through psychology can be delivered by concentrating on speed, simplicity, surprise, social proof, and stirring emotions. Speed involves getting people the product they want quickly without any fuss. From working closely on your site's structure and building great navigation to creating quick video overviews of a service, the faster you can get information to someone, the happier they'll be. This goes hand-in-hand with simplicity. Simplicity is delivering the right content to the right person at the right time, and involves really studying your audience and what context they're in when they visit your site. Simplicity is heavily colored by perception, and this can be used to your advantage—telling your users it will take 30 seconds to sign up on your site when it actually takes 10 will make your site seem easier to use and will make that user feel smarter and better about themselves.

Surprise is challenging the expectations of your users in some small way. You can evoke this by using complimentary highlight colors on calls to action, or provoking the user to ask questions they need to click through to answer (what is that weird old tip for a tiny belly?), or even picking a product or company name that provokes surprise and questions from a user (like Sarah herself did with "You Know Who"). Social proof is as simple as pre-filling a tip jar (so it seems normal to leave money) or public follow/like counts on your site (30 million people and counting apparently love FarmVille). Finally, Sarah talked about how people are much more likely to make judgments on emotion and only later justify them rationally. Things like using lots of whitespace for luxury items or natural textures for organic items create an emotional attraction that can work beautifully when trying to attract a user to your products. Etsy does a great job of this, as did Apple when creating a "How it's made" for their iPhoto holiday cards. Anthropomorphized icons can also work great for creating an emotional attachment, and can be as easy as adding eyes and a smile to, say, an illustration of a house. There was a ton of great stuff in her talk, and I'm looking forward to trying some of the things she mentioned myself.

Shortcut Keys for the HipChat Fluid App

2011-03-09T09:20:00.000-08:00

Moved here

As I mentioned last week, we've started to use HipChat for our internal chat here at Avvo. I've decided to forego their default AIR app and try running their web interface through Fluid to get a more native mac-like experience. One of the few things that bothered me about the Fluid app was the lack of keyboard controls for accessing and closing the tabs for each chat room. So I added them.

One of the great things about Fluid is its support for userscripts to add functionality to the pages it wraps. I created a userscript for the HipChat web interface that adds the following keyboard shortcuts:

Command-W: Closes the current tab (if the tab can be closed)
Command-1 through Command-9: Selects the tab at the specified position
Command-{ and Command-}: Select the previous tab and next tab, respectively.

To install it, when running your HipChat fluid instance, open 'Preferences', go to the 'Advanced' section and add *.user.js as a pattern. Then, go to 'File', 'Open Location' and paste in the following url:

https://github.com/justinweiss/hipchat_shortcut_keys/raw/master/hipchat_shortcut_keys.user.js

Install the script when it asks, and you should be good to go.

HipChat without AIR

2011-03-02T16:12:00.000-08:00

Moved here

We've been investigating HipChat for internal chat lately, and I was a little disappointed that it used Adobe AIR on all platforms, instead of providing a native app. To be fair, it's the most well-done AIR app this side of Basalmiq, but there were still the little inconsistencies that I couldn't help but get annoyed by. I'm not going to describe the exact things that bothered me here, since Alex Payne does such a great job describing the problems here, but it felt weird enough that I couldn't see myself using the app as often as I'd use, say, Propane.

Heading through their support pages to get an idea of how I could improve my experience, I noticed that there seemed to be a fair amount of support for running the web version of HipChat through Fluid, which, while not as pleasant as a native app, still felt much faster and more native on the Mac than the AIR app did. HipChat did an amazing job of building support for things like Growl notifications and Dock badge notifications into the web interface, and seem to want to make the site work as well in Fluid as it does through AIR. Given the constraints they've set for themselves, HipChat has done a tremendous job of building support for people that would like an alternative to AIR.

Accessing HipChat through Fluid

There were only a few minor gotchas I ran into when creating the HipChat Fluid app. First, after downloading and running Fluid, the URL should be set to https://www.hipchat.com/chat. The icon can be left as the website favicon, which awesomely still takes a full-size icon from HipChat's site, rather than the actual tiny favicon the site uses. Once inside the HipChat Fluid app, you'll have to adjust the preferences. Under the Advanced category, you'll have to add the pattern *hipchat.com/sign_in*, otherwise you won't be able to sign in. Finally, if you want Growl and Dock notifications, you have to enable notifications through the AIR app—HipChat doesn't expose that setting in the web app.

There are some cons to running the web app—although notifications, file drag-and-drop, and most of the other functionality works, the web app is missing the file/link dock on the right hand side, the tabs are a little flakier, you can't video chat, and there's a big "Download App" button that would be much more valuable as room for another chat tab. However, the use of native Mac OS X controls, the smoother scrolling, Growl support (rather than the built-in notifications that show up in the wrong place), better integration with the system, and the ability to script the site with CSS and JavaScript make the Fluid app the way to go.

Finally, while I was digging through the JavaScript, I noticed that HipChat has what looks like decent support for running it as a Chrome app, which might be worth trying if you're not on OS X or just prefer Chrome.

Deploy only tested code with Jenkins and Ruby

2011-02-23T09:36:00.000-08:00

Moved here

Deploying software builds that don't pass the unit tests is a waste of time. It prevents QA from doing their job, it causes delays while a fixed build is prepared, and it could even lead to more ops time to fix if the broken build breaks something on the machines it's being deployed to. If you use Jenkins to run your tests, there's a better way. Using the Jenkins API, we can get the last revision that passed for a project, and adjust our behavior based on that revision number.

Jenkins.rb is a Ruby wrapper for the Jenkins API. gem install jenkins.rb should give you everything you need. Then, you can whip up a quick ruby script:

require 'jenkins'
require 'fileutils'

Jenkins::Api.setup_base_url(:host => 'jenkins-host.example.com')

# Pull the project name from the command line
project = ARGV[0]

# Retrieve the last successful build number (123, for example)
successful_build = Jenkins::Api.job(project)["lastSuccessfulBuild"]["number"]

# Retrieve information about the actual build, and grab the last built revision out of it
build_info = Jenkins::Api.get("/job/#{project}/#{successful_build}/api/json")
sha = build_info['actions'].detect {|h| h["lastBuiltRevision"] }["lastBuiltRevision"]["SHA1"]
puts sha

When this script is run with ruby script.rb project-name, it'll return the most recent revision of project-name that passed all the tests, which the build script can use by calling git reset --hard number or something else like that.

Paginating custom objects with will_paginate

2011-02-16T09:53:00.000-08:00

Moved here

will_paginate makes pagination easy, and we use it whenever we need to show lots of ActiveRecord objects. Once the will_paginate gem is installed, it provides a built-in #paginate method for classes that inherit from ActiveRecord::Base, but we still had our own crazy hacked-up pagination for the responses we would get back from solr. In the interest of making things more consistent and deleting as much code as possible, we thought it'd be a good idea to look into getting our solr objects into the will_paginate model.

This turned out to be much easier than expected. The will_paginate view helper expects an instance of WillPaginate::Collection, which is simple enough to initialize. It was as easy as adding the following method to our search models:

def paginated_docs(options = {})
  WillPaginate::Collection.create(options[:page] || 1, options[:per_page] || PER_PAGE) do |pager|
    
    @response = $solr_client.query(self.search_options.merge({
      :offset => pager.offset,
      :limit => pager.per_page}))
    
    pager.replace(@response.docs)

    # pager.replace sets total_entries if we're on the
    # last page. Otherwise, we'll just populate it.
    unless pager.total_entries
      pager.total_entries = @response.total
    end
  end

As long as a WillPaginate::Collection is returned, will_paginate will do the right thing. Something similar should work for using other services that support limit/offset, but don't have built-in pagination.

Bulletproof Asset Hosting and CDNs with Rails

2011-02-09T08:50:00.000-08:00

Moved here

After setting up a CDN like CloudFront to quickly serve your Rails site's assets worldwide, you'll probably want Rails to actually link to the assets through the CDN, instead of locally. Rails makes this easy with ActionController::Base.asset_host. The easiest way to use asset_host is by putting the following code in a Rails initializer or production.rb:

ActionController::Base.asset_host = "assets.example.com"

After that, calls like image_tag("logo.png") will produce links like:

<img alt="Avvo" src="http://assets.example.com/images/logo.png?1297189653" />

Going past the basics

While this will work for very simple cases, more complicated scenarios need more code. If you put the above snippet in an initializer, you'll have to wrap it in

if Rails.env.production?
  ActionController::Base.asset_host = "assets.example.com"
end

Why would you want to do that when you could stick the code in production.rb? For our site, it's because we rely on configuration set up in other initializers, which won't be ready by the time production.rb runs.

YSlow will tell you about the next thing that needs to be done. It turns out that some browsers will only open two requests to any given hostname during a web request. In order to improve the speed at which a browser can fetch assets, you'll need to do something a little more complex with asset_host.

Giving a Proc to asset_host

Along with taking a string, ActionController::Base.asset_host can also take a proc. This gives us more flexibility when generating these hosts. We can solve the above problem by using a proc instead of a string:

if Rails.env.production?
  # Source is the asset path as a string, request is a request object
  ActionController::Base.asset_host = Proc.new do |source, request|
    # hosts can also be pulled from a config file for more flexibility
    # or skipping the CDN in staging environments
    hosts = ['http://assets1.example.com', 'http://assets2.example.com']

    # Get a semirandom numeric hash that is consistent for the same source,
    # to make sure that the same asset file always maps to the same asset host
    # (This helps the browser cache these assets)
    hash = source.hash

    # Return a semirandomly selected host from the hosts array
    hosts[hash % hosts.length]
  end
end

Dealing with secure pages

The next thing you'll notice, especially if you don't have SSL certificates for your asset hosts, is that you'll start getting warnings when accessing resources on secure pages. This can be easily fixed by adding another condition to the above proc:

if Rails.env.production?
  # Source is the asset path as a string, request is a request object
  ActionController::Base.asset_host = Proc.new do |source, request|

    if request.ssl?
      "#{request.protocol}#{request.host_with_port}"
    else
      # hosts can also be pulled from a config file for more flexibility
      # or skipping the CDN in staging environments
      hosts = ['http://assets1.example.com', 'http://assets2.example.com']
      
      # Get a semirandom numeric hash that is consistent for the same source,
      # to make sure that the same asset file always maps to the same asset host
      # (This helps the browser cache these assets)
      hash = source.hash
      
      # Return a semirandomly selected host from the hosts array
      hosts[hash % hosts.length]
    end
  end
end

It also turns out that when referring to assets in ActionMailer, you sometimes won't get a request. We need to handle that case, too:

if Rails.env.production?
  # Source is the asset path as a string, request is a request object
  ActionController::Base.asset_host = Proc.new do |source, request|

    if !request #request == false for emails, apparently
      "http://#{AppSettings.host}"
    elsif request.ssl?
      "#{request.protocol}#{request.host_with_port}"
    else
      # hosts can also be pulled from a config file for more flexibility
      # or skipping the CDN in staging environments
      hosts = ['http://assets1.example.com', 'http://assets2.example.com']
      
      # Get a semirandom numeric hash that is consistent for the same source,
      # to make sure that the same asset file always maps to the same asset host
      # (This helps the browser cache these assets)
      hash = source.hash
      
      # Return a semirandomly selected host from the hosts array
      hosts[hash % hosts.length]
    end
  end
end

Dynamic assets

There also might be 'dynamic assets' that the app might use, like captchas, images that are generated on the fly by the server, or anything else that's not stored on the filesystem. For these you might just want to use your web server instead of the asset server:

if Rails.env.production?
  # Source is the asset path as a string, request is a request object
  ActionController::Base.asset_host = Proc.new do |source, request|

    # Redirect dynamic assets to the web server
    is_asset = source.match(/images|javascripts|assets|stylesheets/)
    
    if !request #request == false for emails, apparently
      "http://#{AppSettings.host}"
    elsif request.ssl? || !is_asset
      "#{request.protocol}#{request.host_with_port}"
    else
      # hosts can also be pulled from a config file for more flexibility
      # or skipping the CDN in staging environments
      hosts = ['http://assets1.example.com', 'http://assets2.example.com']
      
      # Get a semirandom numeric hash that is consistent for the same source,
      # to make sure that the same asset file always maps to the same asset host
      # (This helps the browser cache these assets)
      hash = source.hash
      
      # Return a semirandomly selected host from the hosts array
      hosts[hash % hosts.length]
    end
  end
end

Hardcoding exceptions

Finally, you might run into some javascript libraries that don't like being loaded from a different domain than the page being loaded. You can hardcode these to be loaded from the web server instead, for the final bit of code:

if Rails.env.production?
  # Source is the asset path as a string, request is a request object
  ActionController::Base.asset_host = Proc.new do |source, request|

    # Some assets aren't happy being loaded from the assets server :-(
    borked_assets = ['tiny_mce']
    
    # Redirect dynamic assets to the web server
    is_asset = source.match(/images|javascripts|assets|stylesheets/)
    
    if !request #request == false for emails, apparently
      "http://#{AppSettings.host}"
    elsif request.ssl? || !is_asset || borked_assets.any? {|asset| source.match(asset)}
      "#{request.protocol}#{request.host_with_port}"
    else
      # hosts can also be pulled from a config file for more flexibility
      # or skipping the CDN in staging environments
      hosts = ['http://assets1.example.com', 'http://assets2.example.com']
      
      # Get a semirandom numeric hash that is consistent for the same source,
      # to make sure that the same asset file always maps to the same asset host
      # (This helps the browser cache these assets)
      hash = source.hash
      
      # Return a semirandomly selected host from the hosts array
      hosts[hash % hosts.length]
    end
  end
end

This snippet of code should handle pretty much any asset you throw at it, and is easily expandable using whatever type of environment-specific configuration you use. Give it a try, and let me know what you think.

Speeding up your site's assets with CloudFront

2011-02-04T07:56:00.000-08:00

Moved here

As part of one of our regular performance passes on Avvo, we decided to give Amazon's CloudFront service a try to speed up the delivery of our images, css, and javascript. CloudFront is the CDN part of Amazon's Web Services suite, and as such, it replicates requested data it across Amazon's CloudFront servers distributed across the world.

By default, Amazon requires you to host anything distributed through CloudFront using S3, and S3 is the only option available through the web interface when configuring CloudFront. At first, this didn't seem like it would be an issue, but after trying to push our assets to S3 and hit them through CloudFront, we began to notice some problems.

Issues with using S3 as a backing store

It started with the slowness. Uploading our new assets to S3 on every deploy only took about a minute, but even a minute represents a significant portion of our total deploy time. Annoying, but still acceptable. What was less acceptable was the inablity for CloudFront and S3 to respond to clients' Accept-Encoding: gzip headers and automatically return the pre-gzip'd files we uploaded. Although there were ways to get around this issue, it was at this point that we decided to try Amazon's less-documented and less-accessible Custom Origins feature to tell CloudFront to use our site itself as its backing store. This, we hoped, would fix both the speed and gzipping issues, as well as be simpler to maintain.

Using Custom Origins

Custom Origins, although less accessible through Amazon's tools than S3 origins, seem to offer a much simpler solution for integrating CloudFront for assets that aren't already on S3. Fortunately, a few of the third-party Amazon Web Services tools already provide support for setting up a CloudFront distribution with Custom Origins. If your curl-fu is better than mine, you can hit their server directly by sending requests documented by Amazon. I prefer Ruby, though, so I tried out the newest version of RightScale's right_aws gem.

Installing RightAWS

(As far as I can tell, the right_aws gem available with gem install doesn't yet support custom origins. You can skip this part once it does).

Installing the gem was pretty easy using git and bundler:

git clone https://github.com/rightscale/right_aws.git
cd right_aws
bundle install

Of course, you could also build and install the gem yourself, if you prefer.

Setting up your Custom Origin

Once right_aws is installed, it's easy to write a script or start an irb session to set up a new CloudFront distribution:

require 'right_aws'
acf = RightAws::AcfInterface.new('aws public key', 'aws secret key')
acf.create_distribution(
  :comment => "My Remote Origin",
  :custom_origin => {
    :dns_name => "www.myserver.com",
    :http_port => 80,
    :https_port => 443,
    :origin_protocol_policy => 'match-viewer',
  },
  :enabled => true
)

The only tricky part of the above is the origin_protocol_policy parameter. This parameter can take one of two values, http-only or match-viewer. This specifies whether https requests to CloudFront assets will make https requests to your custom origin. Amazon suggests that http-only should be used unless there's a particular reason to use match-viewer.

At this point, you should be able to see your new distribution in the CloudFront console and hit your assets through the host they give you. There's still some more work to do to make sure your assets are delivered as quickly as possible, but that's another post that'll be written another time.

Introducing the Avvo API

2011-01-28T11:07:00.000-08:00

Moved here

As we announced on the Avvo Blog, we're formally introducing the Avvo API. This means that developers can integrate with the data in our professional directory in a way that makes sense for their sites and applications. The API is RESTful, and returns all responses in JSON, so it should be simple to integrate into your own apps.

For Ruby developers, we went one step further—we released a gem that makes it even simpler to integrate the information on Avvo into your own sites and applications. The Avvo API gem is available on github and can be installed easily into your own apps.

Installing the gem is as simple as

gem install avvo_api

If you use Bundler in your app, you just need to put the following in your Gemfile:

source "http://rubygems.org"
gem 'avvo_api', "~>0.1"

And you should be ready to go. It's easy to get started with the API—all you need is an Avvo account and an API key from Josh King. Here's a short example:

require 'avvo_api'
AvvoApi.setup('user@avvo.com', 'password')
lawyer_id = AvvoApi::Lawyer.search(:q => 'tax', :loc => 'Seattle, WA')["results"].first["id"]

lawyer = AvvoApi::Lawyer.find(lawyer_id)

review = lawyer.reviews.first
puts "#{review.overall_rating} - #{review.title}"

For more documentation and examples, take a look at the Readme on github and the rdoc built from the gem. The API gem uses another project, Reactive Resource, which you should check out if you're wrapping other REST APIs with ActiveResource.

You can find full documentation and try the API on the API documentation site. We're excited to see what you come up with!

resque-scheduler 2.0.0 pre

2010-12-10T10:42:00.001-08:00

Moved here

I just push resque-scheduler 2.0.0.a.

  gem install resque-scheduler --pre

It contains some interesting changes from brianjlandua and davidyang. The gist of it is, you can now schedule items dynamically. Here's what it looks like: Let's say you want your users to configure when some report is generated. You can do something like this:

  Resque.set_schedule('user_16_report', {
    :cron => "0 1 * * *",
    :queue => "reports",
    :class => "GenerateUserReport",
    :args => 16})

That creates a scheduled job and takes effect immediately. This is accomplished by storing the schedule state in a redis hash (so, yes, >=1.3 redis is required) and marking changes in an ordered set. The schedule process looks for changes in the ordered set and then applies them during each loop. The scheduler tab in resque-web will also pull the schedule from redis, so you can see what's currently scheduled. This behavior is not default. You need to set this (for the scheduler process and resque-web):

  Resque::Scheduler.dynamic = true

By default, resque-scheduler behaves as it always has (other than storing the schedule in redis). This functionality just opens up new possibilities for scheduling jobs.

Ummm.. That's What She Said

2010-08-07T22:46:00.000-07:00

Moved here

Ok, so we were joking at work the other day about whipping up a "that's what she said" Bayes classifier. Someone pointed out that you could use a Twitter search for #twss to train it. A couple hours later I found my self on the bus coding it up. The result is the twss gem. Check it out on github. Usage looks something like this:

    requre 'twss'
    TWSS("Have you checked out that file?") # => false
    TWSS("Yeah... I think it's too big") # => true

The classifier and twitter gems do all the heavy lifting. Yes, I realize this is childish, lowbrow humor, but I just couldn't resist seeing if it would work. Turns out it works enough to be funny. Expect an IRC bot soon :) UPDATE: IRC bot can be found here

Disabling internal services gracefully with resque, resque-scheduler, and redis_feature_control

2010-03-11T09:19:00.000-08:00

Moved here

We've got a data warehouse that is separate from our Rails app's database. We aggregate data there and then feed summaries back into our Rails app to power all kinds of statistics for our users (impression data, traffic, etc). We heavily use Resque for our backend jobs, including pulling data from our warehouse. It works great. A user requests some data, resque serves it up. It also sends out periodic update emails which include data sourced by our warehouse. We started running into problems when we wanted to run migrations on our warehouse that took several hours. Being down during this time is not really acceptable and we didn't want to lose jobs that happened to run and depended on the warehouse being up and in a consistent state. We needed to be able to tell Resque to stop processing warehouse jobs (but still come back for them later). We also needed to be able to tell the user that this report was temporarily disabled while we upgraded (rather than timing out). The rest of the website should continue to run as usual. Basically, we needed a central place for processes to look for whether a service (in this case our warehouse) was available. This switch needed to be able to be turned off programatically (ie: during deployment of a magration) or manually (ie: via an admin tool). It also needed to be lightweight so even the tiniest script could use it. We also needed to be able to requeue jobs that needed to wait until the warehouse was back up. But we didn't want to just requeue them because they would immediately get popped again and could potentially starve lower priority jobs. We solved the first problem by coming up with redis_feature_control. Basically, a very simple on/off switch back by redis. Usage looks like this:

  # Check to see if the warehouse is supposed to be up...
  Redis::FeatureControl.enabled?(:warehouse) # => true

  # Disable the warehouse
  Redis::FeatureControl.disable!(:warehouse)

Pretty simple. We then wrapped our capistrano task that migrates our warehouse with disable/enable. We updated our Rails app to display a nice pretty "Please come back in a few minutes" message to our users instantly rather than timing out and detecting errors the hard, ugly way. And we updated our Resque jobs like so:


   def self.perform
     if Redis::FeatureControl.enabled?(:warehouse)
       # do stuff
     else
       # try again in a bit...
     end
   end

Now for the "try again in a bit" part. This was pretty easy with the resque-scheduler (you can read my previous post on it here and here). Basically replace the "try again in a bit" comment with:

  Resque.enqueue_in(1.hour, self)

Done. The job will be pushed back onto the Resque queue in an hour. If the warehouse still isn't available, it will wait another hour and so on. End result: When our warehouse is being migrated, it flags itself as being "off" and the dependent processes take the appropriate action, including delaying jobs to be processed in the future. So far, it's worked like a charm.

Added Resque.enqueue_in for delayed resque jobs

2010-01-11T11:54:00.001-08:00

Moved here

Last week I added support for delayed jobs in resque to the resque-scheduler gem. You can now do stuff like this:

  Resque.enqueue_in(5.days, SendEmailFollowup, :user_id => current_user.id)

And in 5 days, the job will be queued for work as if you called Resque.enqueue directly. Alternatively, you can call enqueue_at and pass it a Time instance (or unix timestamp as an int) instead of a number of seconds. There's also an additional tab in resque-web for viewing delayed jobs. For screen shots and more details, check it out on github. If you're interested in calling jobs on a recurring basis, checkout my previous post on using resque-scheduler to replace cron jobs that just call ruby scripts.

Introducing resque-scheduler

2009-12-21T16:55:00.000-08:00

Moved here

At Avvo, we've got a lot of backend ruby/rails jobs running as crons. Jobs to refresh leaderboards, jobs to warm caches, jobs to pull data from third parties. The list is long and distinguished. These jobs run on different machines at different times in different intervals. Up until recently, we've been managing it all with puppet. We guess at where we have spare cycles to run a job and on which machine. We've ignored the problem long enough that now we have cron configuration in a couple dozen files running all different sorts of things. Very annoying and difficult to keep track of. Just a few weeks ago, we integrated Resque and started porting over our jobs. Having distributed, generic workers is awesome. It's another thing we've long put off because I couldn't find just the right fit. Resque is that fit. Only one thing was missing: A way to add things to queues based on a schedule. What I wanted in addition to job processing is job scheduling to replace our gagillion cron jobs. So we hacked up resque-scheduler. Resque-scheduler is a gem that extends resque to support scheduled jobs. Installation (hosted on gemcutter.org):

  gem install resque-scheduler

Resque-scheduler takes the schedule as a hash, which is easily represented in YAML...

queue_documents_for_indexing:
  cron: "0 0 * * *"
  class: QueueDocuments
  args: 
  description: "This job queues all content for indexing in solr

clear_leaderboards_contributors:
  cron: "30 6 * * 1"
  class: ClearLeaderboards
  args: contributors
  description: "This job resets the weekly leaderboard for contributions"

clear_leaderboards_moderator:
  cron: "30 6 * * 1"
  class: ClearLeaderboards
  args: moderators
  description: "This job resets the weekly leaderboard for moderators"

... that can be set in your resque initializer like so:

  require 'resque-scheduler'
  ResqueScheduler.schedule = YAML.load_file(File.join(File.dirname(__FILE__), '../resque_schedule.yml'))

Then to run the scheduler process, a simple rake task:

  $ rake resque:scheduler

This process idles until a schedule item fires, then it stuffs it into the appropriate queue. See http://github.com/bvandenbos/resque-scheduler for the complete details. Obviously, you need to be using resque to take advantage of resque-scheduler For the future, I'd like to find a clean way to extend resque-web to display the schedule and allow users to click a button to manually queue items in the schedule. Many thanks to resque (defunkt) and rufus-scheduler (jmettraux) which are doing all the heavy lifting.

Getting ruby, rubygems, rake, mysql, imagemagick, rmagick, aspell, raspell with fink

2009-07-10T09:48:00.001-07:00

Moved here

This is really just for me to remember this. If at some point someone else benefits from this, even better.

Download and Install fink

Edit /sw/etc/apt/sources.list and add the following:

deb http://fink.sodan.ecc.u-tokyo.ac.jp/apt/10.5 unstable main crypto
deb http://sage.ucsc.edu/fink_intel_10.5_only    stable main crypto
deb http://sage.ucsc.edu/fink_intel_10.5_only    unstable main crypto

Install ruby:

sudo apt-get install ruby
sudo apt-get install ruby18-dev
sudo apt-get install ruby18-shlibs

Install rubygems:
```
sudo apt-get install rubygems-rb18
```

Update rubygems:

sudo gem update --system
sudo gem install rubygems-update # this is to get passed v 1.1.1
sudo update_rubygems

Install rake
```
sudo gem install rake
```

Then install mysql:

sudo apt-get install mysql
sudo apt-get install mysql-client
sudo apt-get install mysql15-dev
sudo gem install mysql

If you want to use a custom my.cnf, plunk it in /etc

Then install imagemagick and rmagick (this one can be a pain)

sudo apt-get install imagemagick10-dev imagemaigck10-shlibs
sudo apt-get install freetype freetype-shlibs ghostscript ghostscript-fonts \
 gv libpng-shlibs libjpeg libjpeg-bin libjpeg-shlibs lcms lcms-bin lcms-shlibs \
 libtiff libtiff-bin libtiff-shlibs
sudo gem install rmagick -v 2.9.2

Install aspell:

sudo apt-get install aspell
sudo apt-get install aspell-en
sudo apt-get install apsell-dev
sudo gem install raspell

Install other gems:

sudo gem install hoe -v 2.0.0
sudo gem install nokogiri
sudo gem install rake-compiler
sudo gem install taka
sudo gem install johnson -v 1.1.0 # 1.1.1 wouldn't install

Introducing cached_attribute - cached values across object instances

2009-03-16T14:59:00.001-07:00

We've been memoizing attributes manually for a long time, and since we're still on Rails 2.1, we haven't been able to use the nifty new features for memoization. I liked the way Rails 2.2 did it, though, and revisited it when we were investigating some perf issues on the site. Memoization is cool and all, but it didn't buy us anything when we had multiple instances of the same object on the same page, so that's where cached_attribute comes in. Cached_attribute is a plugin that stashes the result of an expensive calculation into a cache, indexed by the object's id. That way, multiple object instances will share the same cached value for the same attribute (on the same object). The readme has more information and an example, or you could just check out the code:

ruby script/plugin install git://github.com/avvo/cached_attribute.git

And as usual, the project is available on github.

Multiple delivery method support for ActionMailer

2009-03-12T15:04:00.000-07:00

For our test servers (and in development), I wanted to be able to create an action that would display all the emails sent by that mongrel. When using the :test delivery method, sent emails are readily available in ActionMailer::Base.deliveries. However, for our test server I wanted ActionMailer to send the email with smtp in addition to keeping them around in memory. So here's the resulting plugin: http://github.com/bvandenbos/actionmailer_multiple_delivery_methods/tree/master

ruby script/plugin install git@github.com:bvandenbos/actionmailer_multiple_delivery_methods.git

Then in your development.rb (or environment.rb, or wherever) you can do this:

  config.action_mailer.delivery_method = [:test, :smtp]

I've tried it with Rails 2.1.0. No guarantees with any other version, or even 2.1.0 for that matter ;)

Twitterss - RSS to Twitter script that can handle multiple accounts

2009-01-13T10:59:00.000-08:00

I know there are a bunch of RSS to Twitter scripts out there in a variety of languages. There's even twitterfeed which manages all your feeds for you (which is a pretty cool service). But, I couldn't find one that did all of the following:

Took updates as frequently as I wanted
Could handle multiple RSS feeds to multiple Twitter accounts (ie: could be configured)
Didn't require handing out my Twitter password (not that this is always a bad thing)
Didn't have a bunch of dependencies (activerecord, mysqlite, etc...)
Preferably written in ruby

So, I hacked twitterss together. Twitterss is just a script that takes a YAML configuration file as a paramter which is meant to be used as a cron job:

  $ twitterss twitterss.yml

twitterss.yml might look something like this:

legalnews:
  rss: http://www.avvo.com/news.rss
  login: avvolegalnews
  password: secret

cnn_money_top_stories:
  rss: http://rss.cnn.com/rss/money_topstories.rss
  login: money_news
  password: secret
  max: 5

cnn_money_markets:
  rss: http://rss.cnn.com/rss/money_markets.rss
  login: money_news
  password: secret
  max: 3

This config would make twitterss pull items from the CNN Money Top Stories and the CNN Money Markets feeds and push them the the money_news Twitter account. It would also pull from the Avvo Legal News feed and push to avvolegalnews. Example result: To make sure it doesn't post the same link twice, it keeps a list of each link it posts for each config entry in a local file (<USER_HOME>/.twitterss_history).

Method of the Day: index_by

2008-11-20T13:15:00.000-08:00

Just stumbled on to this one. I can't tell you how many times I've wanted this functionality. You have an array, but you want a hash keyed by an attribute of the items in the array. Typically, you'd do something like this:

users_by_name = {}
users.each { |u| users_by_name[u.name] = u }

Which is of course silliness now that we know about index_by.

users_by_name = users.index_by(&:name)

Slick.

Pushing to multiple user accounts on GitHub

2008-11-06T15:32:00.000-08:00

Here at Avvo, we've started to use GitHub for publishing some of our internal libraries that might be useful to the programming community. Although we have a company account, it's nice to have a personal account, too. Unfortunately, github doesn't allow multiple accounts to share the same SSH key. This makes sense, because all communication with github works using the same ssh user account. From my perspective, it can get kind of annoying. The key to being able to check code into multiple github accounts is setting up multiple ssh keys and using ssh aliases to use the correct key for the correct repo. First, I created a company-specific key:

justin-mac:~ jweiss$ ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/jweiss/.ssh/id_rsa): /Users/jweiss/.ssh/id_rsa.avvo
Enter passphrase (empty for no passphrase): 
Enter same passphrase again:

Then I set up an SSH alias in ~/.ssh/config:

host github-avvo
  HostName github.com
  User git
  IdentityFile ~/.ssh/id_rsa.avvo

Next, I copied the id_rsa.avvo.pub key into the avvo github acccount. Finally, when creating the clone of whichever repo I was using, I didn't use the clone path that github gave me, I translated it to use the alias I created. For example, a clone URL like

git@github.com:avvo/fuzzy-find-in-project.git

becomes

github-avvo:avvo/fuzzy-find-in-project.git

Now, when pushing to the central repository, it will pick up the id_rsa.avvo key that I tied to the avvo github account, and send the code to the right place.

Method of the Day: Array()

2008-10-23T14:38:00.001-07:00

Another oldy but a goodie... Array() wraps its parameter in an array. If it's already an array, it just returns it. If it's nil, it returns an empty array. It's super useful when you want to write a method that can take a single instance of an object or an array of objects.

Array(nil) => []
Array([]) => []
Array(1) => [1]
Array([1]) => [1]

def do_stuff(model_or_models)
  models = Array(model_or_models)
  models.each do |model|
    # do stuff...
  end
end

DelSolr 0.0.2 Released - Added Support for Updates/Delete

2008-10-22T11:06:00.000-07:00

A while ago I posted on the first version of delsolr (0.0.1) which provided support for querying solr and not much else :). We just released an update to delsolr (0.0.2). The major change is added support for updating/deleting documents. It's still fairly untested, so, standard disclaimer applies.

c = DelSolr::Client.new(:server => 'solr', :port => 8983)

# updating one document...
doc = DelSolr::Document.new
doc.add_field('id', 'ABC-123')
doc.add_field('name', '8 Gig Shuffle')
doc.add_field('description', 'A crazy-tiny MP3 player')
c.update!(doc) # posts new document to solr

# updating a whole bunch and batching the post to solr
models = SomeModel.find(:all)
models.each do |model|
  doc = DelSolr::Document.new
  doc.add_field('id', model.id)
  doc.add_field('name', model.name)
  doc.add_field('description', model.description)
  c.update(doc)
end
c.post_update! # post all the updates in one batch

c.commit! # post a commit to solr

I used a rails example, but delsolr is really intended for ruby and not necessarily rails. Acts_as_solr and the like might suit some better (but not us). This project came out of a piece of code we use internally to manage solr connections. My goal is to get delsolr to the point where we can replace our old implementation. It's not there yet, but it's getting closer. (It's slow going as this is largely done in our "free time".)

Introducing fuzzy-find-in-project: TextMate-style file browsing in Emacs

2008-10-14T13:28:00.000-07:00

Moved here

I've been trying off and on to learn to use Emacs over the last few months, but I always found myself coming back to TextMate for one reason: the "Command-T" Find File in Project command. I tried a few of the plugins for Emacs that were supposed to replicate the functionality, but none of them had the performance to quickly search through thousands of files. I had just about given up when I saw Jamis Buck's post about porting an enhanced version of TextMate's functionality to a rubygem, which he could wrap with a vim plugin. A few days later, I wrote my own wrapper to the gem for Emacs as a way of learning Emacs-Lisp, and posted it to GitHub: fuzzy-find-in-project. It works pretty much like Jamis' vim plugin, and has been pretty fast as far as I can tell (at least, much faster than the pure Elisp plugins I tried). Here's how it works:

Run the command

Narrowing down the possibilities

Selecting the next match

Opening the matched file

You can modify the project directory with the function:

(fuzzy-find-project-root "~/path/to/project")

Check it out!

Actionview error when naming partials " _next.html.erb"

2008-09-29T15:18:00.000-07:00

So, today I got this wonderfully cryptic error from the erb compiler:

Showing widget/profile_completeness/_next.html.erb where line #52 raised:

compile error
/[path_removed]/_next.html.erb:-3: syntax error, unexpected '=', expecting kEND
next = local_assigns[:next]
     ^

After debugging into action_view, I realized it was because I foolishly named a partial "_next.html.erb". "next" is apparently some pseudo-reserved word in action_view and if you use it, your template crashes in weird and interesting ways. I particularly like the negative line number in the error message which results from rails pre-pending code to the erb block when compiling the method used for rendering. Bug filed here. Update: Apparently, you'll run into this if you name your partial any reserved word in ruby (case, class, etc...)

Rails hack: Adding html comments to label partials

2008-09-23T15:27:00.001-07:00

Ever want to figure out which partial a block of html is coming from without greping through your entire rails project? Render_partial_comments is a rails plugin that renders html comments to mark the begin and end of partial output. Give it a try:

ruby script/plugin install git://github.com/avvo/render_partial_comments.git

Then enable it by adding this global in config/environments/development.rb (or wherever)

$render_partial_comments = true

Then restart your server... Your rendered html will now include html comments wrapping the output from partials. Should look something like this:

<!-- render_begin 'user/view' -->
....
<!-- render_end 'user/view' -->

Update: This may have some adverse effects on autocomplete if you use it

belongs_to :dependent => :destroy with foreign key constraints

2008-09-19T15:42:00.001-07:00

Moved here

I discovered what I believe to be a bug in activerecord (2.1.0) today. If you have something like the following:

class Person < ActiveRecord::Base
  belongs_to :person_address, :dependent => :destroy
end

class PersonAddress < ActiveRecord::Base
  has_one :person
end

...and you have for foreign key constraint on person_address_id on the person table to id on the person_address table (which would be a reasonable thing to do) you will get a foreign key constraint error when you try to destroy a person record (if the associated person_address record exists). I submitted a patch to rails for this, but in the meantime, I found this as a work around:

class Person < ActiveRecord::Base
  belongs_to :person_address # note: no :dependent => :destroy

  def destroy
    super() # first destroy ourselves before we destroy the association
    PersonAddress.destroy(self.person_address_id) if self.person_address_id
  end
end

Private Mixins - Including helpers in controllers

2008-09-12T08:26:00.000-07:00

Despite being fully aware that controllers should include minimal logic, we still have some code that needs to be shared by multiple controllers (and even some views). The easiest way to share code in ruby is, of course, the mixin module. The trouble in rails is that any public methods on controllers are exposed as actions, even ones that come in via mixin. That's trouble. One solution was to define all methods in our helpers as private. Didn't really work in every scenario and kind of limits testing. What we really wanted was to be able to include modules privately. As far as I can tell, ruby doesn't support this off the shelf, so we decided we give it a shot to see where it took us.


class Module
 
  [:private, :protected].each do |type|
    eval %{
      def include_#{type}(*prms)
        prms.each do |mod|
          include(mod)
          mod.instance_methods.each do |meth|
            #{type}(meth)
          end
        end
      end
    }
  end
 
end

This creates two methods include_private and include_protected. These guys wrap the normal include method but then take all the instance methods for the included module and privatize or protected-ize them, respectively. Then we added this to ApplicationController:


class ApplicationController < ActionController::Base

  class << self
    alias_method :include_helper, :include_private
  end
 
end

Now we can include helpers in our controllers privately:


class MyController < ApplicationController
  include_helper MyHelper
end

Now, it seems to me that there must be a better way of doing this and we would love to see it because our googling came up empty and it seems odd that we had to resort to this, though it seems to fit our needs just fine.

DelSolr - Solr Facets Made Easy in Ruby

2008-09-08T13:33:00.000-07:00

delsolr - lightweight solr wrapper for ruby Here's an example of solr faceting made simple via delsolr:

c = DelSolr::Client.new(:server => 'solr1', :port => 8983)

# faceting by field...
rsp = c.query('dismax', :query => 'mp3 player',
                        :facets => [{:field => 'brand', :limit => 5, :mincount => 1}]

# now let's output the facet values and counts
rsp.facet_field_values('brand').each do |brand|
  puts "#{brand} - #{rsp.facet_field_count('brand', brand)}"
end

# faceting by query... with filtering...
rsp = c.query('dismax', :query => 'mp3 player',
                        :filters => {:onsale => true},
                        :facets => [{:query => "instock:true", :name => 'instock'},
                                    {:query => {:price => (0..50), :description => 'cool'}, :name => 'cool_and_cheap'}]) 

# now let's output some query facets
puts "Instock: #{rsp.facet_query_count_by_name('instock')}"
puts "Sweet: #{rsp.facet_query_count_by_name('cool_and_cheap')}"

We've been using this code internally for a bit, so we'd figure we'd open it up as a gem to see if other people find it useful. There are a number of other gems/plugins that do something similar, but we weren't satisfied with the balance of power and simplicity in any of them, particularly when it comes to faceting. The full documentation is available on the delsolr page

Method of the Day: Array#in_groups_of

2008-09-03T10:24:00.001-07:00

This one is great for building rows of tables.

<table>
<% @models.in_groups_of(3).each do |row| %>
  <tr>
    <% row.each do |model| %>
      <td><%= model.to_s %></td>
    <% end %>
  </tr>
<% end %>
</table>

This will render the array in a left-to-right, top-down order. If you want to render in top-down, left-to-right order, try this:

  <% num_cols = 3 %>
  <% @models.in_groups_of((@models.length/num_cols).ceil).transpose.each do |row| %>
  ...

That should give you the (more common) column-first layout.

Method of the Day: Array#to_sentence

2008-09-03T10:19:00.000-07:00

An Oldy but a goody...

  <% a = ['hot dogs', 'hamburgers', 'pizza'] %>
  I like <%= a.to_sentence(:connector => 'and') %>.

renders...

  I like hot dogs, hamburgers, and pizza.

InMemoryCacheStore

2008-08-28T09:19:00.000-07:00

I really like the the acts_as_cached interface for model caching, but there are a few models that we wanted to cache in the process memory, but still maintain expiration functionality. We came up with this:


class InMemoryCacheStore < Hash

  def initialize(options={})
    @expire_entries = options[:expire_entries]
  end

  def set(key, data, ttl = nil)
    if @expire_entries && ttl
      self[key] = Entry.new(data.dup, Time.now.to_i + ttl) rescue data
    else
      self[key] = Entry.new(data.dup, nil) rescue data 
    end
  end

  # If we're expiring entires, first check the expiration and remove it if it's expired
  # key key
  def get(key)
    r = self[key]
    if @expire_entries && r && r.expires_at < Time.now.to_i
      self.delete(key)
      r = nil
    end
    r ? r.obj : nil
  end

private

  class Entry
    attr_accessor :obj        # cached object
    attr_accessor :expires_at # time in seconds

    def initialize(obj, expires_at)
      @obj = obj
      @expires_at = expires_at
    end
  end

end

Example without expiration:


class SomeModel < ActiveRecord::Base
  acts_as_cached :store => InMemoryCacheStore.new
end

Example with expiration:


class SomeModel < ActiveRecord::Base
  acts_as_cached :store => InMemoryCacheStore.new(:expire_entires => true)

  def self.ttl
    24.hours
  end
end

Performance for 1000 lookups: db on localhost: 1.52 secs memcached on localhost: 0.78 secs InMemoryCacheStore: 0.345 secs

Populating fixtures from a database table

2008-08-22T17:28:00.001-07:00

One problem with the add_symbolic_names code posted earlier on the blog is the need to keep the test fixtures in sync with the data in the production database — otherwise, code can reference constants that exist in production, and will break while running tests. Keeping these in sync is annoying enough that I wrote a rake task to dump the contents of a table to the appropriate fixtures file. I ended up with something like this:

      desc "Overwrite a fixture file with the contents of a table (set by TABLE=)"
      task :generate_from_table do 
        ActiveRecord::Base.establish_connection(RAILS_ENV)
        table_name = ENV['TABLE']
        File.open("test/fixtures/#{table_name}.yml", 'w') do |file|
          data = ActiveRecord::Base.connection.select_all("SELECT * FROM #{table_name}")
          rows = {}
          
          data.each do |record|
            rows["#{table_name}_#{record['id']}"] = record
          end
          
          file.write rows.to_yaml
        end
      end

Running this rake task will generate a fixture file containing all the information in the table specified by the environment variable TABLE. In the future, we’ll probably look at automating the fixture generation for these symbolic tables so we don’t have to worry about failing tests when we add new rows. UPDATE: I realized that in most cases, the symbolic name of the database row makes a much better fixture title than the id, which results in the following code:

      desc "Overwrite a fixture file with the contents of a table (set by TABLE=)"
      task :generate_from_table do 
        ActiveRecord::Base.establish_connection(RAILS_ENV)
        table_name = ENV['TABLE']
        File.open("test/fixtures/#{table_name}.yml", 'w') do |file|
          data = ActiveRecord::Base.connection.select_all("SELECT * FROM #{table_name}")
          rows = {}
          
          data.each do |record|
            fixture_name = record['symbolic_name'].blank? ? 
              "#{table_name}_#{record['id']}" : 
              record['symbolic_name'].downcase
            rows[fixture_name] = record
          end
          
          file.write rows.to_yaml
        end
      end

This gives us nice fixtures that look like this:

--- 
dog: 
  name: Dog
  id: "1"
  symbolic_name: DOG
cat: 
  name: Cat
  id: "2"
  symbolic_name: CAT

Fixnum to Models

2008-08-22T15:29:00.000-07:00

I saw andand a while back and lol'ed at how clever and awesome it was. With that in mind, we added a couple methods to Fixnum to clean up record fetching on our more popular models:

Fixnum.class_eval do

  def to_user
    User.find(self)
  end

end

That way when I find myself with something that might be a user id...

puts user_id.andand.to_user.andand.name

Maybe not something you'd want on every model, probably not optimized, but hey... it beats checking for nil, looking up the user id, checking for nil again and printing.

What do you get when you cross...

2008-04-02T14:15:00.001-07:00

+

=

YAML.

AddSymbolicNames - Making constants out of records in static tables with ActiveRecord

2008-02-28T09:07:00.001-08:00

Here's the problem: You've got a fairly complex database schema and you find yourself creating a lot of normalized tables. Then you find yourself writing application code that switches on the values of a normalized table. Then you find your application loading activerecord model after activerecord model to do comparisons. Then you think you need to build an in memory cache to save on lookups. Here's my quick solution that has worked for me in many situations: Say you have the following:

class State < ActiveRecord::Base
  has_many :cities
end

class City < ActiveRecord::Base
  belongs_to :state
end

(Assume State has a code column which stores the two letter state code: 'WA', 'CA', etc) Now somewhere in your application you want to write some code that only runs for cities in WA. You might have something like the following:

if city.state.code == 'WA'
# do something
end

So, activerecord is going to end up hitting the database for the state attribute so it can compare the code of that state to 'WA'. Now to avoid the database lookup to fetch the state model, you could of course use any number of caching mechanisms that exist (acts_as_cached, CachedModel). But it might be nice for something like state that doesn't ever change to be able to treat it more like a symbol and less like a model. So you could do something like this:

if city.state.code == State::WA

or even better

if city.state_id == State::WA

This last one avoids the lookup for state altogether and removes the string constant which could contain typos or change over time. Now, I deal with a bunch of these types of tables (static content except during new releases, <1000 records, lots of application logic around specific values), so I wrote this simple helper to generate namespaced symbols mapping a column to the id.

module AddSymbolicNames

  module ClassMethods
    
    def add_symbolic_names(opts = {})
      symbolic_name_attr = opts[:symbolic_name_attrib] || :symbolic_name
      value_attr = opts[:value_attrib] || :id
      
      find(:all).each do |r|
       class_eval %{
        if !defined? #{r.send(symbolic_name_attr)}
         #{r.send(symbolic_name_attr)} =
         #{r.send(value_attr)}
        end
       }
      end
    end
    
  end
  
  def self.included(base)
    base.extend(ClassMethods)
  end
  
end

ActiveRecord::Base.class_eval do 
  include AddSymbolicNames
end

Now with our previous example we can do this:

class State < ActiveRecord::Base
  has_many :cities
  add_symbolic_names :symbolic_name_attrib => :code
end

# somewhere in app-land
if city.state_id == State::WA
  # do some awesome washington specific logic
end

This creates symbols for every state that you have in the db (ie, State::WA, State::CA) that map to the id for that record. A few obvious restrictions: the :symbolic_name_attrib column must be unique and it also must not contain whitespace since it will become a symbol. It's basically like creating enums in c++ that reflect database values but that are set at run time. You could also do this if you felt like it:

class State < ActiveRecord::Base
  has_many :cities
  add_symbolic_names :symbolic_name_attrib => :code,
                     :value_attrib => :name
end

Now the constant State::WA will return the string "Washington" assuming the 'name' attribute of State returns the name of the state.