Robby on Rails: Tag database

Master/Slave Databases with Ruby on Rails

Robby Russell — Thu, 15 Nov 2007 16:02:00 -0600

Not terribly long ago, I announced Active Delegate, which was a really lightweight plugin that I developed to allow models to talk to multiple databases for specific methods. The plugin worked great for really simple situations, like individual models.. but when it came time to test with associations it fell apart. I haven’t had a chance to work on any updates and knew that it was going to take more work to get it going.

Earlier this week, we helped one of our bigger clients launch their new web site¹. For the deployment, we needed to send all writes to a master database and a reads to slaves (initial deployment is talking to almost 10 slaves spread around the globe!). We needed something to get integrated quickly and decided to ditch Active Delegate for the time being and began looking at the following options.

I spoke with Rick Olson² and he pointed me to a new plugin that he hasn’t really released yet. So, I’m going to do him a favor and announce it for him. Of course… I got his permission first… ;-)

Announcing Masochism!

Masochism³ is a new plugin for Ruby on Rails that allows you to delegate all writes to a master database and reads to a slave database. The configuration process is just a few lines in your environment file and the plugin takes care of the rest.

Installing Masochism

With piston, you can import Masochism with:


  $ cd vendor/plugins
  $ piston import http://ar-code.svn.engineyard.com/plugins/masochism/

To learn more about piston, read Every Second Counts with a Piston in your trunk

You can also install it with the old-fashioned way:


  $ ./script/plugin install -x http://ar-code.svn.engineyard.com/plugins/masochism/

Configuring Masochism

The first thing that you’ll need to do is add another database connection in config/database.yml for master_database. By default, Masochism expects you to have a production database, which will be the read-only/slave database. The master_database will be the connection details for your (you guessed it…) master database.


# config/database.yml  
production:
  database: masochism_slave_database
  adapter: postgresql
  host: slavedb1.hostname.tld
  ...

master_database:
  database: masochism_master_database
  adapter: postgresql
  host: masterdb.hostname.tld
  ...

The idea here is that replication will be handled elsewhere and your application can reap the benefits of talking to the slave database for all of it’s read-only operations and let the master database(s) spend their time writing data.

The next step is to set this up in your environment file. In our scenario, this was config/environments/production.rb.


# Add this to config/environments/production.rb
config.after_initialize do 
  ActiveReload::ConnectionProxy.setup!    
end

Voila, you should be good to go now. As I mentioned, we’ve only been using this for this past week and we’ve had to address a few problems that the initial version of the plugin didn’t address. One of our developers, Andy Delcambre, just posted an article to show how we had a problem with using ActiveRecord observers and masochism, which we’re sending over a patch for now.

As we continue to monitor how this solution works, we’ll report any findings on our blog. In the meantime, I’d be interested in knowing what you’re using to solve this problem. :-)

¹ Contiki, a cool travel company we’re working with

² Rick just moved to Portland… welcome to stump town!

³ The Masochism plugin README

Fixing PostgreSQL corruption with Rails?

Robby Russell — Sat, 05 Aug 2006 19:16:00 -0500

People have been emailing me to let me know that the search on my blog was broken. Today I finally set out to fix the problem, which looked like a complex issue with my PostgreSQL database. I’m not sure how long ago it started so I am not sure what was the cause at the moment. If you did a search on my blog you’d get an application error and behind the scenes, you would see the following error.

PGError: ERROR:  missing chunk number 0 for toast value 58441

Eek!

So, I tried to REINDEX the table and it didn’t solve the problem… so I started poking around with different types of queries to see what I could do to generate the error again. Didn’t take me long to figure out that it had something to do with the body column in the contents table.


db_name=# SELECT count(id) FROM contents WHERE body ~* 'postgresql' LIMIT 20;
ERROR:  missing chunk number 0 for toast value 58441

db_name_=# SELECT count(id) FROM contents WHERE excerpt ~* 'postgresql' LIMIT 20;
 count 
-------
     0
(1 row)

What am I to do? I did some googling (and go figure… the error being returned was caught on my blog by google)... which was amusing.

In the #postgresql channel on freenode they recommended that I try and find the specific row in the table that was causing this error. I decided to just run a for loop in script/console on the Content object in Typo and print out the name of each record until it gave me an error.


for i in 1..30000
  puts Content.find( i ).title
  i =+ 1
end

This began to print out titles of older blog entries and sure enough… the loop died when it hit the following error. :-)


  PostgreSQL sequences in Rails
  When TSearch2 Met AJAX
  ActiveRecord::StatementInvalid: PGError: ERROR:  missing chunk number 0 for toast value 58441
  : SELECT * FROM contents WHERE (contents.id = 1678)  LIMIT 1
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/connection_adapters/abstract_adapter.rb:88:in `log'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/connection_adapters/postgresql_adapter.rb:137:in `execute'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/connection_adapters/postgresql_adapter.rb:351:in `select'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/connection_adapters/postgresql_adapter.rb:118:in `select_all'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/base.rb:431:in `find_by_sql'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/base.rb:395:in `find'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/base.rb:393:in `find'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/base.rb:409:in `find'
          from (irb):23
          from (irb):22
  >> exit

15.seconds.later I logged into psql and ran DELETE FROM contents WHERE id = 1678... and all is well!

Rails Migrations and PostgreSQL Constraints

Robby Russell — Fri, 11 Nov 2005 10:42:00 -0600

A question was posed on the Rails mailing list concerning how one would go about adding CONSTRAINTs to the database tables with ActiveRecord::Migration.

One argument was raised stating that it is easier to handle these in plain SQL schema files. I disagree. :-)

Migrations to the Rescue

Databases evolve and I have recently found the Migration structure to be perfect for handling iterations and schema changes. Using the #execute method has helped move more of my code into the Ruby/Rails framework… and that just makes things easier to manage in the long-run. This is the approach that we are using at PLANET ARGON with some of our current client projects.

# db/migrate/6_add_foreign_key.rb
class AddForeignKey < ActiveRecord::Migration
  def self.up
    execute "ALTER TABLE bees ADD CONSTRAINT beehive_id_fkey FOREIGN KEY
(beehive_id) REFERENCES beehives (id);"
  end

  def self.down
    execute "ALTER TABLE bees DROP CONSTRAINT beehive_id_fkey;"
  end
end

This gives us an easy way to use the standard, #create_table syntax for building our tables with Ruby… and then we can slap these constraints on later.

This would add the constraints…


rake migrate VERSION=6

...run tests…


rake

...roll back…


rake migrate VERSION=5

I have found that this approach is really useful with testing in Rails. When I think that I have everything working great (without CONSTRAINTS in PostgreSQL), I run another migration to add a bunch of foreign key and data constraints to the tables and… run my tests again.

Let’s give Active Record a Hug

This has helped me gain some trust in Active Record while still giving me that comforting feeling that PostgreSQL is acting as the body guard for my data.

Even if you don’t end up using Migrations to handle these types of database schema changes, I would highly suggest that you model your implementation after this. I’ve worked with many database schemas and this just makes it easy to add your new change and run one command to commit it to the database.

...and now I go play with beehives…