Read my latest article: Six days to complete the Rails hosting survey (posted Thu, 24 Apr 2014 14:36:00 GMT)

Planting the seeds

Posted by Sat, 05 Sep 2009 13:00:00 GMT

Yesterday, the Rails team released 2.3.4, which includes standardized way for loading seed data into your application so that you didn’t have to clutter your database migrations.

I noticed a few comments on some blogs where people were asking how to use this new feature, so here is a quick runthrough a few ways that you can use it.

Populating Seed Data Approaches

The db/seeds.rb file is your playground. We’ve been evolving our seed file on a new project and it’s been great at allowing us to populate a really large data. Here are a few approaches that we’ve taken to diversify our data so that when we’re working on UI, we can have some diversified content.

Basic example

Any code that add to db/seeds.rb is going to executed when you run rake db:seed. You can do something as simple as:

# db/seeds.rb

Article.create(:title => 'My article title', :body => 'Lorem ipsum dolor sit amet, consectetur adipisicing elit')

Just create database records like you would in your Rails application or in script/console. Simple enough, right? Let’s play with a few other approaches that we’ve begun to use.

Use the names of real people

We’re using the Octopi gem to connect to github, collect all the names of people that follow me there, and using their names to seed our development database.

@robby_on_github = Octopi::User.find('robbyrussell')

# add a bunch of semi-real users
@robby_on_github.followers.each do |follower|
  github_person = Octopi::User.find(follower)
  next if github_person.name.nil?

  # split their name in half... good enough (like the goonies)
  first_name = github_person.name.split(' ')[0]
  last_name = github_person.name.split(' ')[1]
  new_person = Person.create(:first_name => first_name, :last_name => last_name, :email => Faker::Internet.email,
                             :password => 'secret', :password_confirmation => 'secret',
                             :github_username => follower, :website_url => github_person.blog)
  # ...
end

We do this with a few sources (twitter, github, etc..) to pull in the names of real people. If you want to be part of my seed data, you might consider following me on Github. ;-)

Use Faker for Fake data

You may have noticed in the previous code sample, that I used Faker in that code. We are using this a bunch in our seed data file. With Faker, you can generate a ton of fake data really easy.

person.links.create(:title => Faker::Lorem.words(rand(7)+1).join(' ').capitalize,
                    :url => "http://#{Faker::Internet.domain_name}/",
                    :description => Faker::Lorem.sentences(rand(4)+1).join(' '))

We might toss something like that into a method so that we can do the following:

@people = Person.find(:all)

500.times do
  generate_link_for(@people.sort_by{rand}[0])
end

...and we’ll get 500 links added randomly across all of the people we added to our system. You can get fairly creative here.

For example, we might even wanted random amounts of comments added to our links.

def generate_link_for(person)
  link = person.links.create(:title => Faker::Lorem.words(rand(7)+1).join(' ').capitalize,
                             :url => "http://#{Faker::Internet.domain_name}/",
                             :description => Faker::Lorem.sentences(rand(4)+1).join(' '))

  # let's randomly add some comments...
  if link.valid?
    rand(5).times do
      link.comments.create(:person_id => @people.sort_by{rand}[0].id,
                           :body => Faker::Lorem.paragraph(rand(3)+1))
    end
  end
end

It’s not beautiful, but it gets the job done. It makes navigating around the application really easy so that we aren’t having to constantly input new data all the time. As mentioned, it really helps when we’re working on the UI.

Your ideas?

We’re trying a handful of various approaches to seed our database. If you have some fun ways of populating your development database with data, we’d love to hear about it.

Subdomain accounts with Ruby on Rails explained

Posted by Sun, 11 Jan 2009 23:30:00 GMT

DHH recently posted, How to do Basecamp-style subdomains in Rails on SvN and it just happens that I was implementing some similar stuff this last week for a project we’re developing internally.

In our project, not everything needs to be scoped per-account as we are building a namespace for administrators of the application and also want a promotional site for the product. Three different interfaces, with some overlap between them all.

Let’s walk through a few quick steps that you can follow to setup the two interfaces within the same application.

Suppose that we’re going to build a new web-based product and have the following requirements initially.

  • We need a promotional site for sign-ups, frequently-asked-questions, support requests, etc.
  • When people sign-up for an account, they’ll should have their own unique sub-domain
  • There are two different visual layouts (promotional site and the account)

Note: I use RSpec and am going to skip the TDD process here and let you conquer that for yourself. Am using the default Rails commands in this tutorial.

Account model / Database

We’re going to generate a new model for Account, which will be responsible for scoping sub-domains and individual accounts.

  account-demo : ruby script/generate model Account subdomain:string 
       create  app/models/
       create  test/unit/
       create  test/fixtures/
       create  app/models/account.rb
       create  test/unit/account_test.rb
       create  test/fixtures/accounts.yml
       exists  db/migrate
       create  db/migrate/20090111220627_create_accounts.rb

Great, let’s migrate our application.

   account-demo : rake db:migrate
  ==  CreateAccounts: migrating =================================================
  -- create_table(:accounts)
     -> 0.0045s
  ==  CreateAccounts: migrated (0.0052s) ========================================

Before we get too far, let’s make sure that we’re adding an index on this table for the subdomain, as it’ll improve performance in the database as the subdomain will used in SQL conditions quite often.

account-demo : ruby script/generate migration AddIndexToAccountSubdomain
     exists  db/migrate
     create  db/migrate/20090111221009_add_index_to_account_subdomain.rb

Let’s open up this new migration file and toss in a UNIQUE INDEX on subdomain.

  class AddIndexToAccountSubdomain < ActiveRecord::Migration
    def self.up
      add_index :accounts, :subdomain, :unique => true
    end

    def self.down
      remove_index :accounts, :subdomain
    end
  end  

Okay, let’s migrate this bad boy.

   account-demo : rake db:migrate
  ==  AddIndexToAccountSubdomain: migrating =====================================
  -- add_index(:accounts, :subdomain, {:unique=>true})
     -> 0.0047s
  ==  AddIndexToAccountSubdomain: migrated (0.0050s) ============================  

Great, we’re now ready to move on to the fun stuff.

Let’s open up app/models/account.rb and throw some sugar in it.

Data Validation

Because we’re going to be dealing with subdomains, we need to make sure that we’re only allowing people to sign-up with valid data otherwise, there could be issues. URLs need to fit within certain conventions and we need to make it as graceful as possible for our customers.

Let’s make a quick list of what we need to enforce for the subdomain attributes. This can easily be expanded, but let’s cover the basics.

  • Each account should have a subdomain
  • Each subdomain should be unique within the application
  • A subdomain should be alpha-numeric with no characters or spaces with the exception of a dash (my requirement)
  • A subdomain should be stored as lowercase

So, let’s update the following default Account model….

  class Account < ActiveRecord::Base
  end  

..and add some basic validations.

  class Account < ActiveRecord::Base
    validates_presence_of :subdomain
    validates_format_of :subdomain, :with => /^[A-Za-z0-9-]+$/, :message => 'The subdomain can only contain alphanumeric characters and dashes.', :allow_blank => true
    validates_uniqueness_of :subdomain, :case_sensitive => false

    before_validation :downcase_subdomain

    protected

      def downcase_subdomain
        self.subdomain.downcase! if attribute_present?("subdomain")
      end
  end  

Reserved subdomains

In the project that our team is working on, we wanted to reserve several subdomains so that we could use them later on. We tossed in the following validation as well.

  validates_exclusion_of :subdomain, :in => %w( support blog www billing help api ), :message => "The subdomain <strong>{{value}}</strong> is reserved and unavailable."

This will prevent people from using those when they sign up.

Controller / Handling Requests

Let’s now think about how we’ll handle requests so that we can scope the application to the current account when a subdomain is being referenced in the URL.

For example, let’s say that our application is going to be: http://purplecowapp.com/ [1]

Customers will get to sign-up and reserve http://customer-name.purplecowapp.com/. I want my account subdomain to be green.purplecowapp.com and everything under this subdomain should be related to my instance of the application.

I’ve begun working on my own module, which is inspired mostly by the account_location plugin with some additions to meet some of our product’s requirements.

Here is my attempt to simplify it for you (removed some other project-specific references) and have put this into a Gist for you.

  #
  # Inspired by
  # http://dev.rubyonrails.org/svn/rails/plugins/account_location/lib/account_location.rb
  #
  module SubdomainAccounts
    def self.included( controller )
      controller.helper_method(:account_domain, :account_subdomain, :account_url, :current_account, :default_account_subdomain, :default_account_url)
    end

    protected

      # TODO: need to handle www as well
      def default_account_subdomain
        ''
      end

      def account_url( account_subdomain = default_account_subdomain, use_ssl = request.ssl? )
        http_protocol(use_ssl) + account_host(account_subdomain)
      end

      def account_host( subdomain )
        account_host = ''
        account_host << subdomain + '.'
        account_host << account_domain
      end

      def account_domain
        account_domain = ''
        account_domain << request.domain + request.port_string
      end

      def account_subdomain
        request.subdomains.first || ''
      end

      def default_account_url( use_ssl = request.ssl? )
        http_protocol(use_ssl) + account_domain
      end

      def current_account
        Account.find_by_subdomain(account_subdomain)
      end

      def http_protocol( use_ssl = request.ssl? )
        (use_ssl ? "https://" : "http://")
      end
  end  

View gist here (embed wasn’t working right when I tried)

Just include this into your lib/ directory and require it in config/environment.rb. (if people think it’s worth moving into a plugin, I could do that)

Including AccountSubdomains

In the main application controller (app/controllers/application.rb), just include this submodule.

  class ApplicationController < ActionController::Base
    include SubdomainAccounts

    ...
  end  

Now, we’ll want to add a check to verify that the requested subdomain is a valid account. (our code also checks for status on paid memberships, etc… but I’ll just show a basic version without that)

Let’s add in the following to app/controllers/application.rb. This will only check on the status of the account (via subdomain) if the current subdomain is not the default. For example: purplecowapp.com is just our promotion site, so we won’t look up the account status and/or worry about the subdomain. Otherwise, we’ll check on the status.

  before_filter :check_account_status

  protected
    def check_account_status
      unless account_subdomain == default_account_subdomain
        # TODO: this is where we could check to see if the account is active as well (paid, etc...)
        redirect_to default_account_url if current_account.nil? 
      end
    end  

Current Account meets Project model

When requests are made to an account’s subdomain, we want to be able to scope our controller actions.

WARNING: I’m going to gloss over the following steps because this is just standard Rails development stuff and I want to focus on how to scope your Rails code to account subdomains.

I’ll just say that this product gives each account many projects to do stuff within. I’ll assume that you’ll know how to handle all that and we’ll assume you have a Project model already.

What you will need is to add a foreign key to your table (projects in this example) that references Account. So, make sure that your model has an account_id attribute with and that the database table column has an INDEX.

We’ll add our associations in the models so that they can reference each other.

  # app/models/account.rb
  class Account < ActiveRecord::Base
    has_many :projects
    # ...
  end

  # app/models/project.rb
  class Project < ActiveRecord::Base
    belongs_to :account
    # ...
  end  

Okay great… back to our controllers. The SubdomainAccounts module provides you with the current_account variable, which you can use within your controllers/views. This allows us to do the following in our controllers. For example, if we had a ProjectsController.

  class ProjectsController < ApplicationController
    def index
      @projects = current_account.projects.find(:all)
    end

    def new
      @project = current_account.projects.new
    end 

    def show
      @project = current_account.projects.find(params[:id])
    end

    # ...
  end

See, this wasn’t so hard, was it?

Handling layouts

I wanted to highlight one other thing here because I suspect that most projects that fit this will likely need a promotional/resource site where people will sign-up from. In our application, we have two application layouts. One for the main application that customers will interact with via their subdomain and the promotional site layout.

The default layout is just app/views/layouts/application.html.erb and we have our promotional site layout at app/views/layouts/promo_site.html.erb. A few of our controllers are specifically for the promotional site while the rest are for the application itself and in some cases, there is some overlap down to individual action within a controller.

What we did was add a few more before filters to our application controller to a) define the proper layout to render, and b) skip login_required on the promo site.

To have the proper layout get rendered, we’re just checking whether the current request was made to the promotional site or not.

class ApplicationController < ActionController::Base
  # ...
  layout :current_layout_name # sets the proper layout, for promo_site or application

protected

  def promo_site?
    account_subdomain == default_account_subdomain
  end

  def current_layout_name
    promo_site? ? 'promo_site' : 'application'
  end

  # ...
end

Our application is using Restful Authentication and we just want to check to see if the current request is made to the promotional site or not. If it is, we’ll skip the login_required filter. Let’s assume that you have the following before_filter set.

class ApplicationController < ActionController::Base
  # ...
  before_filter :login_required

We’ll just change this to:

class ApplicationController < ActionController::Base
  # ..
  before_filter :check_if_login_is_required 

  protected
    def promo_site?
      account_subdomain == default_account_subdomain
    end

    def current_layout_name
      promo_site? ? 'promo_site' : 'application'
    end    

    def check_if_login_is_required
      login_required unless promo_site?
    end

    # ...

There we go. We can now render the proper layout given the request and only handle authentication when necessary.

Development with account subdomains

When you begin developing an application like this, you need to move beyond using http://locahost:3000 as we need to be able to develop and test with subdomains. You can open up your /etc/hosts (within a Unix-based O/S) file and add the following.


127.0.0.1 purplecowapp.dev
127.0.0.1 green.purplecowapp.dev
127.0.0.1 sample.purplecowapp.dev
127.0.0.1 planetargon.purplecowapp.dev
127.0.0.1 lollipops.purplecowapp.dev
127.0.0.1 help.purplecowapp.dev
127.0.0.1 support.purplecowapp.dev

After you edit that file (with root permissions), you can flush your dns cache with dscacheutil -flushcache (Mac OS X). This will let you make requests to http://purplecowapp.dev:3000/ and http://green.purplecowapp.dev:3000. This is a convention that our team has begun using for our own projects (TLD ending in .dev). It’s important to remember that the subdomain must be specified here in order to work for local requests. Unfortunately, hosts files don’t support wildcards (’*’).

Update

You can also use Ghost, which is a gem for managing DNS entries locally with Mac OS X. Read Get to know a gem: Ghost

Summary

I know that I glossed over some sections, but was hoping that the code itself would be the most beneficial for you. Feel free to leave any questions and/or provide some feedback on our approach. Perhaps you have some suggestions that I could incorporate into this so that we can improve on this pattern.

1 yeah, I’ve been reading more Seth Godin recently…

Master/Slave Databases with Ruby on Rails

Posted by Thu, 15 Nov 2007 22:02:00 GMT

Not terribly long ago, I announced Active Delegate, which was a really lightweight plugin that I developed to allow models to talk to multiple databases for specific methods. The plugin worked great for really simple situations, like individual models.. but when it came time to test with associations it fell apart. I haven’t had a chance to work on any updates and knew that it was going to take more work to get it going.

Earlier this week, we helped one of our bigger clients launch their new web site1. For the deployment, we needed to send all writes to a master database and a reads to slaves (initial deployment is talking to almost 10 slaves spread around the globe!). We needed something to get integrated quickly and decided to ditch Active Delegate for the time being and began looking at the following options.

I spoke with Rick Olson2 and he pointed me to a new plugin that he hasn’t really released yet. So, I’m going to do him a favor and announce it for him. Of course… I got his permission first… ;-)

Announcing Masochism!

Masochism3 is a new plugin for Ruby on Rails that allows you to delegate all writes to a master database and reads to a slave database. The configuration process is just a few lines in your environment file and the plugin takes care of the rest.

Installing Masochism

With piston, you can import Masochism with:


  $ cd vendor/plugins
  $ piston import http://ar-code.svn.engineyard.com/plugins/masochism/

You can also install it with the old-fashioned way:


  $ ./script/plugin install -x http://ar-code.svn.engineyard.com/plugins/masochism/

Configuring Masochism

The first thing that you’ll need to do is add another database connection in config/database.yml for master_database. By default, Masochism expects you to have a production database, which will be the read-only/slave database. The master_database will be the connection details for your (you guessed it…) master database.


# config/database.yml  
production:
  database: masochism_slave_database
  adapter: postgresql
  host: slavedb1.hostname.tld
  ...

master_database:
  database: masochism_master_database
  adapter: postgresql
  host: masterdb.hostname.tld
  ...

The idea here is that replication will be handled elsewhere and your application can reap the benefits of talking to the slave database for all of it’s read-only operations and let the master database(s) spend their time writing data.

The next step is to set this up in your environment file. In our scenario, this was config/environments/production.rb.



# Add this to config/environments/production.rb
config.after_initialize do 
  ActiveReload::ConnectionProxy.setup!    
end



Voila, you should be good to go now. As I mentioned, we’ve only been using this for this past week and we’ve had to address a few problems that the initial version of the plugin didn’t address. One of our developers, Andy Delcambre, just posted an article to show how we had a problem with using ActiveRecord observers and masochism, which we’re sending over a patch for now.

As we continue to monitor how this solution works, we’ll report any findings on our blog. In the meantime, I’d be interested in knowing what you’re using to solve this problem. :-)

1 Contiki, a cool travel company we’re working with

2 Rick just moved to Portland… welcome to stump town!

3 The Masochism plugin README

Fixing PostgreSQL corruption with Rails?

Posted by Sun, 06 Aug 2006 00:16:00 GMT

3 comments Latest by Robert Treat Tue, 08 Aug 2006 17:26:04 GMT

People have been emailing me to let me know that the search on my blog was broken. Today I finally set out to fix the problem, which looked like a complex issue with my PostgreSQL database. I’m not sure how long ago it started so I am not sure what was the cause at the moment. If you did a search on my blog you’d get an application error and behind the scenes, you would see the following error.

PGError: ERROR:  missing chunk number 0 for toast value 58441

Eek!

So, I tried to REINDEX the table and it didn’t solve the problem… so I started poking around with different types of queries to see what I could do to generate the error again. Didn’t take me long to figure out that it had something to do with the body column in the contents table.


db_name=# SELECT count(id) FROM contents WHERE body ~* 'postgresql' LIMIT 20;
ERROR:  missing chunk number 0 for toast value 58441

db_name_=# SELECT count(id) FROM contents WHERE excerpt ~* 'postgresql' LIMIT 20;
 count 
-------
     0
(1 row)

What am I to do? I did some googling (and go figure… the error being returned was caught on my blog by google)... which was amusing.

In the #postgresql channel on freenode they recommended that I try and find the specific row in the table that was causing this error. I decided to just run a for loop in script/console on the Content object in Typo and print out the name of each record until it gave me an error.


for i in 1..30000
  puts Content.find( i ).title
  i =+ 1
end

This began to print out titles of older blog entries and sure enough… the loop died when it hit the following error. :-)


  PostgreSQL sequences in Rails
  When TSearch2 Met AJAX
  ActiveRecord::StatementInvalid: PGError: ERROR:  missing chunk number 0 for toast value 58441
  : SELECT * FROM contents WHERE (contents.id = 1678)  LIMIT 1
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/connection_adapters/abstract_adapter.rb:88:in `log'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/connection_adapters/postgresql_adapter.rb:137:in `execute'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/connection_adapters/postgresql_adapter.rb:351:in `select'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/connection_adapters/postgresql_adapter.rb:118:in `select_all'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/base.rb:431:in `find_by_sql'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/base.rb:395:in `find'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/base.rb:393:in `find'
          from /usr/local/lib/ruby/gems/1.8/gems/activerecord-1.13.1/lib/active_record/base.rb:409:in `find'
          from (irb):23
          from (irb):22
  >> exit

15.seconds.later I logged into psql and ran DELETE FROM contents WHERE id = 1678... and all is well!

Rails Migrations and PostgreSQL Constraints

Posted by Fri, 11 Nov 2005 16:42:00 GMT

A question was posed on the Rails mailing list concerning how one would go about adding CONSTRAINTs to the database tables with ActiveRecord::Migration.

One argument was raised stating that it is easier to handle these in plain SQL schema files. I disagree. :-)

Migrations to the Rescue

Databases evolve and I have recently found the Migration structure to be perfect for handling iterations and schema changes. Using the #execute method has helped move more of my code into the Ruby/Rails framework… and that just makes things easier to manage in the long-run. This is the approach that we are using at PLANET ARGON with some of our current client projects.

# db/migrate/6_add_foreign_key.rb
class AddForeignKey < ActiveRecord::Migration
  def self.up
    execute "ALTER TABLE bees ADD CONSTRAINT beehive_id_fkey FOREIGN KEY
(beehive_id) REFERENCES beehives (id);"
  end

  def self.down
    execute "ALTER TABLE bees DROP CONSTRAINT beehive_id_fkey;"
  end
end

This gives us an easy way to use the standard, #create_table syntax for building our tables with Ruby… and then we can slap these constraints on later.

This would add the constraints…

rake migrate VERSION=6
...run tests…
rake
...roll back…
rake migrate VERSION=5

I have found that this approach is really useful with testing in Rails. When I think that I have everything working great (without CONSTRAINTS in PostgreSQL), I run another migration to add a bunch of foreign key and data constraints to the tables and… run my tests again.

Let’s give Active Record a Hug

This has helped me gain some trust in Active Record while still giving me that comforting feeling that PostgreSQL is acting as the body guard for my data.

Even if you don’t end up using Migrations to handle these types of database schema changes, I would highly suggest that you model your implementation after this. I’ve worked with many database schemas and this just makes it easy to add your new change and run one command to commit it to the database.

...and now I go play with beehives…