One of the other topics I’ve been interested in recently is concurrency control, and how we can do that more effectively within my team’s services. We would like to allow our servers to do as much work as possible without overloading them. Overloading a server can make everything slow down, even cheap fast calls get crazy slow.

In Stop Rate Limiting, Jon Moore describes a method of concurrency control using a dynamically determined limit. The method works like this: given some limit to the number of jobs, when you go to schedule a job beyond the limit, you check to see if the system is overloaded. If it isn’t, you add a constant to the limit and allow the new request. On the other hand, when the system is overloaded, you reduce the limit by dividing by a constant. This is known as Additive Increase/Multiplicative Decrease, and is used in algorithms like TCP congestion control.

My performance problem provided a scenario that I could use to try AIMD. My jobs are all assumed to be similar in size, so the limit is the maximum number of jobs. (If the jobs weren’t consistently sized, I might estimate their weight and use a sum of the weights as the limit.) When the system isn’t overloaded, I increased the limit by 1, at most once per second. When the system was overloaded, I decreased the limit by a factor of 0.9, again at most once per second.

I needed a way to check if the system was overloaded. An obvious choice would be load average. But I’d recently learned about Linux’s Pressure Stall Information (PSI), so I thought I would try using it. PSI has three different counters, one each for CPU, memory, and I/O. PSI tells you how often some or all tasks are blocked waiting for a given resource. I looked at PSI values for some of our other servers, and decided to set up three ranges for each: overloaded, acceptable, underloaded, as a percent of the time that some tasks were stalled waiting for the resource. The max healthy limit that I used for CPU was 10%, for memory 1%, for I/O 1%.

The end result is in this gist.

So, how did it go?

I ended up running a couple different types of jobs inadvertently. One was more compute heavy, and one was more balanced between compute and network transfer. The more balanced version downloaded archives, reassembled them, and then ran a couple of consistency checks across the data. The compute heavy version did mostly the same thing, except that it ended up doing the consistency checks many times per job. Another way to look at it: the balanced job did O(N) downloads and O(N) scans; the compute heavy one did O(N) downloads and O(N²) scans.

For the balanced job, the concurrency control worked pretty well. I ran this job a few times. Each time, utilization was just about right so that work got done quickly but the system remained responsive. The number of concurrent workers was 50 - 70 for the duration. CPU usage was around 80%, load was 73 - 120 (on a 128 core system), I/O utilization was around 40% at the peak. Overall, I was happy with how it turned out.

For the compute heavy job, the concurrency control didn’t work as well. I ran this job once. In this case, the number of concurrent jobs was nearer 140, load was also about 140, and CPU usage was pegged at 100%. The system was overloaded badly enough that I interrupted it and ran it differently so that the network to compute ratio was closer to 1:1 again.

If I were to do this again or for production use, I would add a step to the compute-heavy job that checks in with the controller to see if it’s ok to do the next computational step. This would allow the limiter to claw back some of the work if it accidentally allowed too much through. I would also want to tune the overload detector quite a bit, I need to learn more about how to interpret the PSI values.

My first guess is that it would output this:

Trying &main.Owner{org:(*main.Organization)(nil), user:(*main.User)(nil)}
 => "NIL"
Trying &main.Owner{org:(*main.Organization)(0xc42000e200), user:(*main.User)(nil)}
 => "organization:group"
Trying &main.Owner{org:(*main.Organization)(nil), user:(*main.User)(0xc42000e250)}
 => "user:person"

However, that’s not what actually happens. Instead, you’d see this:

Trying &main.Owner{org:(*main.Organization)(nil), user:(*main.User)(nil)}
 => panic!! "invalid memory address or nil pointer dereference"
Trying &main.Owner{org:(*main.Organization)(0xc42000e200), user:(*main.User)(nil)}
 => "organization:group"
Trying &main.Owner{org:(*main.Organization)(nil), user:(*main.User)(0xc42000e250)}
 => panic!! "invalid memory address or nil pointer dereference"

To see why, it helps to see the output from the fmt.Printf statements that are in Owner.DatabaseString:

Trying &main.Owner{org:(*main.Organization)(nil), user:(*main.User)(nil)}
  [0] <nil>
  [1] (*main.Organization)(nil)
 => panic!! "invalid memory address or nil pointer dereference"
Trying &main.Owner{org:(*main.Organization)(0xc42000e200), user:(*main.User)(nil)}
  [0] <nil>
  [1] &main.Organization{name:"group"}
 => "organization:group"
Trying &main.Owner{org:(*main.Organization)(nil), user:(*main.User)(0xc42000e250)}
  [0] <nil>
  [1] (*main.Organization)(nil)
 => panic!! "invalid memory address or nil pointer dereference"

I think this is happening because Go interface types are really a type plus a value. The call to GetOrganization() returns a nil Organization, but not a nil DatabaseStringer.

In which a programmer puts on his mechanical (?) engineer hat.

Recently we lost a greenhouse cover because the U-channel it was attached to came off of the greenhouse. We’ve already had two other wind related events with this particular greenhouse. I’d like to avoid having any more. So, I sat down with a pen, paper, and the Internet of engineering information.

Lift is what we’re working with here. I’ve seen a couple different renderings of it (NASA’s, Wikipedia’s). Here’s the one we’ll use, so that I don’t have to type Greek letters.

L = 1/2 * d * v^2 * s * CL

• d is the density of air
• v is the speed of the aircraft
• s is the area of the airfoil
• CL is the coefficient of lift

d = 1.225 kg / m^3.

The airfoil is fixed, so I used wind speed instead. Gusts of 50mph have been a problem for us, and that’s about the highest wind gusts that we get. So v = 50mph = 73 ft / s = 22.352 m /s.

The area of the airfoil is the footprint of the greenhouse. If I recall correctly, it’s 96 ft x 30 ft. So s = 96 * 30 ft^2 = 2880 ft^2 = 270 m^2.

The coefficient of lift is a tough one to figure out. NASA has a set of academic problems related to lift, and the accompanying chart shows a peak CL of approximately 1.6. Wikipedia’s Lift coefficient page shows a maximum CL of 1.75. An aerospace engineering school’s high lift slide deck maxes out around 1.75. I’m going to assume that we haven’t found a magic formula for generating even higher lift, so I used CL = 1.75 to have a worst-case estimate.

L = 1/2 * 1.225 * (22.352)^2 * 270 * 1.75
L = 144,468.292 Newtons
L = 32,477 lbf

Where does this force end up? We have 34 metal 2 5/8” ground posts. Each post is bolted to a wooden hip board with a 3/8” carriage bolt. U-channel is screwed onto the hip board with Tek screws or wood screws. The plastic is clamped to the U-channel with wiggle wire.

I don’t know how much force you need to apply to pull the metal pipes out of the ground. We have heavy clay soil. I’m going to assume that this is more than enough (over 1000 lb per post?) to handle the wind.

I’m not sure what grade the bolts are. Nucor has a shear strength data sheet that says that 3/8” bolts have at least 3900 lb shear strength. So our 34 bolts should be able to hold down 132,600 pounds.

The U-channels may have been screwed onto the boards about every 2 ft. This is what pulled off, so I can’t measure it now. We used a combination of Tek screws and wood screws. A random study of wood screw pullout strength shows that screws pull out well before 300 lb. So, if we managed to get 200 lb of pullout resistance per screw (and I doubt that we did), the U-channel should have been able to withstand 28,800 pounds of pull out force. That doesn’t directly translate to the type of force the screws received, but if it’s anything like the actual number, it’s way too low. Our plan is to attach new double-U-channel with carriage bolts. If we use 1/4” carriage bolts at every post, we should be able to handle 60,000 pounds of force. If we use 2 at each post and one in between, we should be good for 160,000 pounds of force.

The resqued daemon can be restarted, paused, or stopped gracefully via signals. On restart, resqued reloads your configuration in a new process. When the new process is ready, the old workers are spun down and new workers are forked. If the configuration does not load, the old workers will continue running. Workers that exit unexpectedly are restarted.

Using resqued

Resqued is configured with a ruby-based configuration file. It includes a command-line program that starts the daemon either in the foreground or background.

$ resqued --help
Usage: resqued [options] resqued-config-files...
    -h, --help                       Show this message
    -v, --version                    Show the version
        --test                       Report which worker would start
    -p, --pidfile PIDFILE            Store the pid of the master process in PIDFILE
    -l, --logfile LOGFILE            Write output to LOGFILE instead of stdout
    -D, --daemonize                  Run daemonized in the background

For a typical rails application, create config/resqued.rb:

# Run 4 workers
worker_pool 4

before_fork do
  # Load the app environment.
  require File.expand_path('config/environment.rb')
  # Free up a DB connection.
  ActiveRecord::Base.connection.disconnect!
end

after_fork do |worker|
  # Reconnect to the database.
  ActiveRecord::Base.establish_connection
end

You can see which workers will be started:

$ resqued --test config/resqued.rb
Workers defined in config/resqued.rb
1: *
2: *
3: *
4: *

To start the daemon in the foreground:

$ resqued config/resqued

In production, you would start the daemon like this:

$ resqued -D -p tmp/pids/resqued.pid -l log/resqued.log config/resqued.rb

Migrating from rake resque:work

If you’ve got an existing pool of workers, you’re probably doing something like this:

$ rake resque:work QUEUE=high
$ rake resque:work QUEUE=high
$ rake resque:work QUEUE=high,normal
$ rake resque:work QUEUE=high,normal

To reproduce this in resqued, you would create a config file that looks like this:

# before_fork and after_fork as above

worker_pool 4
queue 'high'
queue 'normal', :count => 2

without a pool

The worker pool might not work for you if you’re doing something like this:

$ rake resque:work QUEUE=high
$ rake resque:work QUEUE=high
$ rake resque:work QUEUE=normal
$ rake resque:work QUEUE=*

To reproduce this in resqued, you would create a config file like this:

# before_fork and after_fork as above

2.times { worker 'high' }
worker 'normal'
worker '*'

Many variants of worker configuration are documented in the worker specs.

I have used composed_of to do some of these things in the past, so take a look at it if serialize isn’t quite what you’re looking for.

The basics

To get started with serialize, add a text column to your table, and add the serialize directive to your model.

class MyModel < ActiveRecord::Base
  # This serializes whatever you assign.
  serialize :my_field
  # This serializes a hash. If the DB value is nil, the AR value will be {}
  serialize :my_field, Hash
  # This serializes some class that you've written.
  serialize :my_field, MyField
end

If you choose the last option, what aspects of serialization can you control?

Default serialization with YAML

By default, your attribute will be serialized as YAML. Let’s say you define MyField like this:

class MyField
end

It will be serialized like this:

--- !ruby/object:MyField {}

It will deserialize back to a MyField instance.

Custom serialization

You can customize the serialization in a couple of different ways. One way is to implement the instance method to_yaml. I wouldn’t do this, because you’ll need to do it in a way that Yaml.load can use to create an instance of your class.

The other way to do it is to add dump and load class methods to your class. For example:

class MyField
  def self.dump(obj)
    # ...
  end
  def self.load(string)
    # ...
  end
end

Dump converts your object to a string, and load converts a string into an instance of your class. Here’s an example of a custom formatting of attributes:

class MyField
  def self.dump(obj)
    coder.dump(:a => obj.a, :b => obj.b)
  end
  def self.load(string)
    new coder.load(string)
  end
  def self.coder
    ActiveRecord::Coders::YAMLColumn.new(Hash)
  end
  def initialize(options = {})
    @a = options[:a]
    @b = options[:b]
  end
  attr_accessor :a, :b
end

Now, let’s say you want to be able to assign a hash to the serialized object, maybe from a form. You can do that:

class MyField
  def self.dump(obj)
    case obj
    when Hash then coder.dump(obj)
    else           coder.dump(:a => obj.a, :b => obj.b)
    end
  end
  def self.load(string)
    new coder.load(string)
  end
  def self.coder
    ActiveRecord::Coders::YAMLColumn.new(Hash)
  end
  def initialize(options = {})
    @a = options[:a]
    @b = options[:b]
  end
  attr_accessor :a, :b
end

It’s worth noting that you’ll be without the instance of MyField at first.

>>> rec = MyModel.new
>>> rec.my_field.class
 => MyField
>>> rec.my_field = {}
>>> rec.my_field.class
 => Hash
>>> rec.save
 => true
>>> rec.my_field.class
 => MyField

So, another option is to override the setter on MyModel:

class MyModel < ActiveRecord::Base
  serialize :my_field, MyField
  attr_accessible :my_field
  def my_field= value
    case value
    when Hash
      super MyField.new(value)
    else
      super
    end
  end
end

>>> rec = MyModel.new
>>> rec.my_field.class
 => MyField
>>> rec.my_field = {}
>>> rec.my_field.class
 => MyField
>>> rec.save
 => true
>>> rec.my_field.class
 => MyField
>>> rec.attributes = {:my_field => {}}
>>> rec.my_field.class
 => MyField

Here’s the basic class:

class NoCaching
  attr_accessor :a, :b
  def sum
    a + b + a + b + a + b + a + b + a + b + a + b
  end
end

As expected, sum is evaluated every time you call it.

For comparison, I created a cached version using ActiveSupport::Memoizable.

require 'active_support/memoizable'
class CachedWithActiveSupport < NoCaching
  extend ActiveSupport::Memoizable
  memoize :sum
end

As expected, calling sum on CachedWithActiveSupport returns the same thing, regardless of changes to a or b.

Finally, I created a class with a new memoizer.

class CachedWithRedefinedMethod < NoCaching
  extend RedefinedMethodCaching
  memoize :sum
end

This new memoizer will, on the first call to the memoized method, create a singleton method on the instance that just returns the memoized value.

module RedefinedMethodCaching
  def memoize method
    original_method = instance_method(method)
    define_method method do
      value = original_method.bind(self).call
      class_eval do
        define_method method do
          value
        end
      end
      send method
    end
  end
end

CachedWithRedefinedMethod has the same basic behavior as CachedWithActiveSupport.

Performance

At this point, I got curious about performance. One of the reasons to cache a method’s result is to avoid some performance hit, so we don’t want to introduce another performance hit. Here’s the basic benchmark:

require 'benchmark'
Benchmark.bm(35) do |x|
  [1, 10, 100, 1000, 10000].each do |usage|
    iterations = 500000 / usage
    [NoCaching, CachedWithActiveSupport, CachedWithRedefinedMethod].each do |k|
      x.report("#{usage}x#{k.name}") { iterations.times { o = k.new ; o.a = 10 ; o.b = 20 ; usage.times { o.sum }
    end
  end
end

Here’s my ruby:

ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-darwin11.2.0]

Here are the results:

                                          user     system      total        real
1xNoCaching                           0.740000   0.010000   0.750000 (  0.739076)
1xCachedWithActiveSupport             1.080000   0.000000   1.080000 (  1.140270)
1xCachedWithRedefinedMethod           5.100000   0.070000   5.170000 (  5.187266)

10xNoCaching                          0.460000   0.000000   0.460000 (  0.461452)
10xCachedWithActiveSupport            0.250000   0.000000   0.250000 (  0.253478)
10xCachedWithRedefinedMethod          0.560000   0.000000   0.560000 (  0.567997)

100xNoCaching                         0.420000   0.000000   0.420000 (  0.421145)
100xCachedWithActiveSupport           0.230000   0.000000   0.230000 (  0.224381)
100xCachedWithRedefinedMethod         0.120000   0.000000   0.120000 (  0.127860)

1000xNoCaching                        0.420000   0.000000   0.420000 (  0.419429)
1000xCachedWithActiveSupport          0.190000   0.000000   0.190000 (  0.187462)
1000xCachedWithRedefinedMethod        0.080000   0.010000   0.090000 (  0.083700)

10000xNoCaching                       0.380000   0.000000   0.380000 (  0.383493)
10000xCachedWithActiveSupport         0.200000   0.000000   0.200000 (  0.196237)
10000xCachedWithRedefinedMethod       0.100000   0.000000   0.100000 (  0.106058)

Notice that, uncached, the times are pretty consistent. The number of calls to sum is the same in each run (500,000), so this is expected.

For ActiveSupport’s memoize, you can see the overhead on the 1x run, and you can see the performance gain that even a small number of reps gives.

For the new, redefined method style of memoization, you can see there’s a pretty big overhead, but the performance gain is significant. I compared it to the simple style of memoization (@sum ||= ...), and it’s about the same.

Conclusions

This was an interesting experiment, but I probably won’t use it.

While the performance gain is pretty nice, it’s also not a general purpose solution. In my applications, most memoized methods are only called a handful of times. For that type of use, memoization comes with a penalty, rather than a payoff.

Also, comparing to ActiveSupport’s memoization is unfair, as ActiveSupport provides a couple other nice features. One is cache-busting, via an optional force argument. Another is support for arguments (i.e. cached_method(1) is different from cached_method(2)).

var Thing = function(a,b) { this.a = a; this.b = b; }
Thing.prototype.add = function() {
  if(!this._sum) this._sum = this.a + this.b;
  return this._sum;
};

Today, I thought, “Maybe I can make a singleton method (or whatever that’s called in Javascript) that caches the result?”

var Thing = function(a,b) { this.a = a; this.b = b; }
Thing.prototype.add = function() {
  var sum = this.a + this.b;
  this.add = function() { return sum; }
  return this.add();
};

The last line could return sum, but it adds to the cleverness to return what looks like a recursive call.

I tried this out in Chrome’s JS console in order to verify a couple of things.

First, does it return the right answer?

> var x = new Thing(1,2);
> x.add()
3

Did I do the singleton thing right? i.e. does each instance have an independent cached value?

> var y = new Thing(2,3);
> y.add()
5

Does the function change, as expected?

> var x = new Thing(1,2);
> x.add
function () {
  var sum = this.a + this.b;
  this.add = function() { return sum; }
  return this.add();
}
> x.add()
3
> x.add
function () { return sum; }

And, finally, can I invalidate the cache?

> x.a = 4
4
> x.add() // returns the old value
3
> delete x.add // invalidate the cache
true
> x.add() // returns the new value
6

I don’t know if I like this enough to keep using it, but it seems simple and pliable enough to be worth trying. Thoughts? Drop me a line on twitter.

In this blog post, I’d like to show how to do some neat exception handling tricks with rails 3.

Trick #1 - Use a custom exception

Using a custom exception is simple. Just define the exception class

class MyCustomException < StandardError ; end

and raise an exception.

raise MyCustomException

In development, you’ll see the detailed exception page, including the exception’s type, message, and stack trace. In production, rails renders the default, generic exception page, normally stored in public/500.html.

Trick #2 - Custom exception status code

If you want the exception to produce a different HTTP result status, you need to tell rails about it. The default is 500.

The way to customize the result code has changed slightly in different releases of rails. Here are some ways to tell rails how to handle the exception.

In all versions of rails, you can refer to HTTP status codes numerically or with symbols as defined by Rack. For example, you can refer to ‘404 Not Found’ as :not_found or 404.

In Rails 3.0 and 3.1, you can change the status code of an exception’s response with rescue_responses:

ActionDispatch::ShowExceptions.rescue_responses['MyCustomException'] = :unauthorized

There are a few default exception handlers.

In Rails 3.2, this hash moved, to make it easier to update via a railtie. You can change the status code of an exception’s response in your application configuration (e.g. config/application.rb or config/environments/development.rb):

config.action_dispatch.rescue_responses.merge!('MyCustomException' => :unauthorized)

In public (i.e. production), rails will render the exception from a file in public/, with the status code as its name. For example, if the result status is 404, rails will read and return public/404.html. In Rails 3.2, you can override this behavior by setting config.exceptions_app. See ActionDispatch::PublicExceptions for an example of what this should look like.

class MyPublicExceptions
  def call(env)
    [404, {'Content-Type' => 'text/plain'}, ['not found']]
  end
end

# in config/application.rb
config.exceptions_app = MyPublicExceptions.new

Trick #3 - Custom exception handler on controller

If you want to do something else with your exceptions, you can define a custom exception handler on a controller. For example, in ApplicationController, you can add this:

rescue_from 'MyCustomException', :with => :my_exception_handler

def my_exception_handler exception # This method can take 0 or 1 args
  render :template => 'blah/blah'
end

The nice thing about this is that you can completely control how the exception is rendered. Unfortunately, using this form of rescue short-circuits other ‘normal’ exception handling. So, it will always call the handler, even in development. It will not bubble up to the airbrake notification middleware.

Trick #4 - Production exception handling in development

As you build up more intricate exceptions & handlers, you’ll want to try out your custom handlers in development.

In development mode, you’ll need to do a couple of things to view the production-style exception pages. First, you need to turn off the config flag that forces all requests to behave as if they’re local:

# in config/environments/development.rb
config.consider_all_requests_local       = false # true by default

In Rails 3.2, that should be all you need to do. You should now see production-style exception pages.

In Rails 3.0 and 3.1, you’ll also need to convice rails that the request isn’t local. I’m not sure what the best way to do this is, but one way that worked for me was to add this to the bottom of app/controllers/application_controller.rb:

class ActionDispatch::Request
  def local?
    false
  end
end

In short

I’m working on an API that can take one of its arguments in a couple of different forms. TDD helped me quickly see where I was going to end up with some combinatorial complexity, so I quickly changed my design to avoid the complexity.

My Rails controller

It’s a common thing: add a user to a group. The old UI allows adding 1 user at a time. The new UI allows adding any number of users at a time. I’m planning to support both.

Here’s what the original controller, supporting the old UI, might look like:

class GroupMembersController < ApplicationController
  # POST /groups/:group_id/members
  def create
    group = Group.find(params[:group_id])
    if group.users.where(:id => params[:user_id]).count.zero?
      group.users << User.find(params[:user_id])
    redirect_to root_path
  end
end

I got the new UI working, and needed to update the controller. Being that it was so simple of a controller, I had previously skipped unit tests for it. So, I spun up a new unit test suite with tests to catch any regressions…

class GroupMembersControllerTest < ActionController::TestCase
  setup do
    @group = Group.create!
  end

  test 'adds a group member' do
    @user = User.create!
    post :create, :group_id => @group.id, :user_id => @user.id
    assert_equal [@user], @group.reload.users
  end

  test 'does not re-add a member' do
    @user = User.create!
    @group.users << @user
    post :create, :group_id => @group.id, :user_id => @user.id
    assert_equal [@user], @group.reload.users
  end
end

Now to add the multi-user test. (In this case, the UI is building a single input with a ‘,’-delimited list of user ids.) Here’s the first test:

  test 'adds two members' do
    @user1 = User.create!
    @user2 = User.create!
    post :create, :group_id => @group.id, :user_ids => "#{@user1.id},#{@user2.id}"
    assert_equal [@user1, @user2], @group.reload.users
  end

In the back of my mind, I start thinking about other tests I’ll need to write. One set of tests that comes to mind is the case where a :user_id and :user_ids are both passed in.

Hm. How can I avoid writing all these combination-testing tests? Well, seeing as 2! is bigger than 1!, if I can figure out how to drop the number of handled parameters from 2 to 1, I’ll avoid the extra complexity.

In this case, :user_id => '1' and :user_ids => '1' should behave the same, so there’s really no need for two different parameters. A quick UI & test change …

  test 'adds two members' do
    @user1 = User.create!
    @user2 = User.create!
    post :create, :group_id => @group.id, :user_id => "#{@user1.id},#{@user2.id}"
    assert_equal [@user1, @user2], @group.reload.users
  end

… and I’ve crossed off several tests in my mental checklist.

The API isn’t necessarily ideal (I’m mixing up singular and plural concepts). But, I get a simpler API that still supports the old UI.

Wrap up

In this case, the added complexity was relatively small (only two choices), so the savings in test effort was minimal. Also, I don’t know that I would have actually tested the extra cases, because they’re not that likely or interesting.

However, I think the principal applies: TDD helps you spot complexity as it’s forming, and trim it back to a reasonable level.

Also, I just didn’t have to think any extra “what-ifs”.

Some of the values are accessed through Bundler::Settings. Any values accessed there can be set in a local (./.bundle/config) or global (~/.bundle/config) settings file. The order of precedence is: local, environment, global.

Incidentally, while looking through bundler 1.1.1’s code, I learned that you can trigger whether to load a particular gem with environment variables. So, for example, if you say gem 'debugger', :env => 'USE_DEBUGGER', it only loads the debugger gem if USE_DEBUGGER is set; or, if you say gem 'debugger', :env => { 'USE_DEBUGGER' => 'yes' }, then it only loads the debugger gem if USE_DEBUGGER is set to yes.

This was a thought experiment that has migrated through code and into this post.

Here’s the thought: I write a ruby library that you have to configure, but I don’t want to pre-check all the config. I also want the exception that I through to come from the bad config value, not from some weird-o spot in your and my code where we both have a hard time figuring out what the problem is.

Can we pull off being lazy and accurate?

Here’s one way to do it:

class Config
  def driver=(d)
    if((exc = callcc { |cont| @driver_ptr = cont }) && exc.is_a?(Exception))
      raise exc
    end
    @driver = d
  end

  def try
    @driver_ptr.call ArgumentError.new("Driver cannot be :fail") if @driver == :fail
    puts @driver.inspect
  end
end

config = Config.new

config.driver = :ok
config.try

config.driver = :yes
config.try

config.driver = :fail
puts "Note that the error happens the line before this one (this line = #{__LINE__})"
config.try

config.driver = :not_reached
config.try

And, the output:

:ok
:yes
Note that the error happens the line before this one (this line = 24)
wacky.rb:4:in `driver=': Driver cannot be :fail (ArgumentError)
        from wacky.rb:23

I don’t think I’d use this on a general purpose library, because of the ease of creating an infinite loop…

begin
  config.driver = :fail
rescue => e
  puts e
end
config.try

… but it was fun to try it out.

The problem

The pattern is that I have a root model, with a couple of collections living inside it.

class Site < ActiveRecord::Base
  has_many :members
  has_many :posts
end

One of the collections is made of objects that are just nested…

class Member < ActiveRecord::Base
  belongs_to :site
  has_many :posts
end

… and the other is associated with both.

class Post < ActiveRecord::Base
  belongs_to :author, :class_name => 'Member'
  belongs_to :site
end

(An obvious solution to this problem is to drop the belongs_to :site from Post, but that’s not always desirable. Sometimes it just doesn’t work, if Post and Member had a habtm association, and neither depended on the other’s existence.)

The straightforward factories for the models above is this:

Factory.define :site do |x|
end

Factory.define :member do |x|
  x.name 'example member'
  x.association :site
end

Factory.define :post do |x|
  x.title 'example post'
  x.association :author, :factory => :member
  x.association :site
end

The problem I run into is this: If I do Factory(:post), then the post[id:1] has an author[id:1] linked to site[id:1]. But post[id:1] is linked to site[id:2]!

Solution: Use an after_build callback

The best solution I’ve come up with is to define these associations that depend on each other in an after_build block. The simple way to do it is just to try to hook up common associations.

Factory.define :site do |x|
end

Factory.define :member do |x|
  x.name 'example member'
  x.after_build do |member|
    member.site ||= Factory.build(:site)
  end
end

Factory.define :post do |x|
  x.title 'example post'
  x.after_build do |post|
    post.site ||= post.author.try(site) || Factory.build(:site)
    post.author ||= Factory.build(:member, :site => post.site)
  end
end

Solution improved: after_build and a test-global context.

Another method that I like is to create a context class. Because most of my tests deal with data that is well-formed (e.g. it is unexpected that the application would contain a post that links to a member from a different site), the context class can store a global-to-test site instance.

class TestContext
  class << self
    def reset!
      @instance = nil
    end

    def instance
      @instance ||= new
    end

    def method_missing(*args)
      instance.send(*args)
    end
  end

  attr_accessor :site

  def site!
    self.site = Factory :site
  end
end

Factory.define :site do |x|
end

Factory.define :member do |x|
  x.name 'example member'
  x.after_build do |member|
    member.site ||= TestContext.site! || Factory.build(:site)
  end
end

Factory.define :post do |x|
  x.title 'example post'
  x.after_build do |post|
    post.author ||= Factory.build(:member)
    post.site ||= post.author.site || TestContext.site! || Factory.build(:site)
  end
end

class MyTestCase << Test::Unit::TestCase
  def setup
    TestContext.site!
  end
  def teardown
    TestContext.reset!
  end

  def test_stuff
    post = Factory :post
    # ...
  end
end

This works pretty well for me, but it seems like there should be a better way. I think what I want is the concept of a fixed Site instance for the duration of a given test, so, a replacement for the TestContext.

The exact code in this article has not been tried, so it probably doesn’t run. Hopefully the intent is clear enough.

Here’s an example:

class Room
  has_many :members
end

class Member
  belongs_to :room
  class << self
    def find_or_create_for_invitation(id_or_email)
      where(:id => id).first || where(:email => email).first
      # ... or create
    end
  end
end

What I really want, is for my member find-or-create to be scoped to the room it’s on when I do room.members.find_or_create_for_invitation(params[:invitee]).

I figured it was possible, so I set a debugger breakpoint in find_or_create_for_invitation and checked out the available information. What I found is that the static method is called inside of a with_scope block, so Member.scoped_methods has some useful information. In this case, it’s an array with one element, the scoped association relation (i.e. the thing returned by room.members). It’s possible for the array to contain hashes, too, and several elements, if there are several nested scopes. So, any find-like or create-like calls that you make inside the class method will automatically get the right scope.

For example, one application does not yet allow open signups in production. I want to be able to turn that on & off in development so that (a) I can test it and (b) I can not be bothered by it at other times. Also, we expect to allow open signups in production eventually, just not yet, so I would like to be able to flip the switch without necessarily needing to deploy new code.

In another application, we’ve been trying out a couple different background task runners, and it was uncertain enough as to which we’d use that I abstracted the particulars out so that I could focus on getting the jobs running. So now, I can set the queue type in the config file. And, for easing my development setup, I have dev set to ‘immediate’. And test is set to ‘none’. And, of course, production is set to use the real queue, ‘resque’ (for now). Very flexible.

To do this, I wrote static_config.

Installation

To install, you’ll need the static_config gem. You can gem install static_config or use Bundler:

gem 'static_config'

An ad-hoc config file

The simplest case is just a config file that you can put anything into. This would be the case where you just want to collect some configuration into one place.

First, create a file called config.yml, maybe with something like this:

thing: abc
nested:
  thing: def

In your app, load up the config and use it.

require 'static_config'
MyConfig = StaticConfig.build do
  file File.expand_path('config.yml', File.dirname(__FILE__))
end
puts MyConfig.thing        # => abc
puts MyConfig.nested.thing # => def

An ad-hoc, per-environment config file

So now, you want to change nested.thing to ‘ghi’ in production. It’s not much different.

development:
  thing: abc
  nested:
    thing: def
production:
  thing: abc
  nested:
    thing: ghi

#...
MyConfig = StaticConfig.build do
  file File.expand_path('config.yml', File.dirname(__FILE__)),
    :section => ENV['MY_APP_ENV'] || 'development'
end
#...

An ad-hoc, per-environment config file that you can override with environment variables

Now, let’s override some of the configuration with environment variables.

require 'static_config'
MyConfig = StaticConfig.build do
  file File.expand_path('config.yml', File.dirname(__FILE__)),
    :section => ENV['MY_APP_ENV'] || 'development'
  env 'MY_APP'
end
puts MyConfig.nested.thing

Now, you get this output:

$ ruby app.rb
def
$ MY_APP_ENV=production ruby app.rb
ghi
$ MY_APP_NESTED_THING=blah ruby app.rb
blah

All that, and reloading!

Finally, let’s say you’re doing this in a rails app. You’re going to want to have the configuration loaded automatically, and, in development, reloaded, too.

development:
  thing: abc
  nested:
    thing: def
test:
  thing: test
  nested:
    thing: test
production:
  thing: abc
  nested:
    thing: ghi

In config/initializers/config.rb:

MyConfig = StaticConfig.build do
  file Rails.root.join('config/my_app.yml'), :section => Rails.env
  env 'MY_APP'
end

Rails.application.config.to_prepare do
  MyConfig.reload!
end