Read my latest article: Ezra Zygmuntowicz -- Farewell, Friend. (posted Mon, 01 Dec 2014 17:53:00 GMT)

Reducing MySQL's memory usage on OS X Mavericks

Posted by Sun, 24 Nov 2013 16:22:00 GMT

Recently, I found myself re-installing everything from Homebrew and began to notice that MySQL was consuming nearly half a gig of memory. Given that I don’t do too much with MySQL on a regular basis, I opted to override a handful of default configuration options to reduce the memory footprint.

As you can see, a fresh MySQL install via homebrew was consuming over 400mb of memory.

Here is how I reduced my memory footprint:

$ mkdir -p /usr/local/etc

Unless you already have a custom MySQL config file, you will want to add one into this directory.

$ vim /usr/local/etc/my.cnf

We’ll then paste in the following options into our file… and save it.

  # Robby's MySQL overrides
  [mysqld]
  max_connections       = 10

  key_buffer_size       = 16K
  max_allowed_packet    = 1M
  table_open_cache      = 4
  sort_buffer_size      = 64K
  read_buffer_size      = 256K
  read_rnd_buffer_size  = 256K
  net_buffer_length     = 2K
  thread_stack          = 128K

Finally, we’ll restart MySQL.

$ mysql.server stop

If you have MySQL setup in launchctl, it should restart automatically. After I did this, my MySQL instance was now closer to 80mb.

So far, this has worked out quite well for my local Ruby on Rails development. Mileage may vary…

Having said that, how much memory are you now saving?

Installing Ruby on Rails, Passenger, PostgreSQL, MySQL, Oh My Zsh on Snow Leopard, Fourth Edition

Posted by Mon, 08 Feb 2010 19:14:00 GMT

Welcome to what seems like my tenth installment (actually, it’s the fourth) of showing you how I setup my development environment on a fresh OSX install. In this case, I’m actually getting a MacBook setup for a new employee with Snow Leopard.

Over the years, I’ve evolved these following steps and they’ve helped our team maintain a consistent and stable envirnment for Ruby on Rails development. I know that there are a few other ways to approaching this and I’m sure you’ll get similar results, but this approach has allowed me to maintain a hassle-free setup for the last five years.

As with all things… your milage may vary.

Phase One

During this initial phase, we’re going to install the primary dependencies and setup our environment.

XCode

The first thing that you’ll need to do is install XCode, which almost everything depends upon as this will install developer-friendly tools for you. Apple has been kind enough to ship this on your Snow Leopard DVD.

Go ahead and install XCode from the Optional Installs folder.

(might require a reboot)

You can also download it online.

MacPorts

Now we’ll install MacPorts, which the web site describes itself as, “an open-source community initiative to design an easy-to-use system for compiling, installing, and upgrading either command-line, X11 or Aqua based open-source software on the Mac OS X operating system.”

As I’ve said in past versions of this guide, this tool is about to become one of the most important tools on your operating system. It’ll be used time and time again to maintain your libraries and many of the Unix tools that you’ll be using. If you’re from the Linux or BSD world, you are likely familiar with similar tools… such as: apt-get, port, and yum.

You’ll want to download the latest stable version from http://www.macports.org/. Once downloaded, you can install it.

Once this is installed, you’ll be able to use the port command from your console.

Wget

Let’s test out your MacPorts install by installing a useful tool called wget, which we’ll use to install oh-my-zsh.

sudo port install wget

Git and Subversion

Every development environment should have some source code management tools available. We’ll install both of these with one command.

sudo port install git-core +svn

This will install git and subversion.

oh-my-zsh

Oh My Zsh is the most amazing thing to happen to shells since… well since I said so. It’s one of my open source projects that I encourage you to give a whirl.

wget http://github.com/robbyrussell/oh-my-zsh/raw/master/tools/install.sh -O - | sh

That’s it. The next time you open up your terminal, you’ll be running zsh with a bunch of stuff available. For more information, visit http://github.com/robbyrussell/oh-my-zsh.

Terminal theme (optional)

I never understood why the icon for Terminal has a black background but when you start it up the default theme is black on white.

versus

Anyhow, I’m a fan of the the dark background. To change this, open up preferences in Terminal. Select Pro, then click on the Default window so that this sticks around.

Let’s now open up a new Terminal window..

You should be looking at something like this:

Much better… let’s continue.

Phase Two

We’re now going to start installing everything we need to get this running.

Ruby 1.8.7.x

First up, Ruby.

Snow Leopard includes Ruby and Rails already installed, but we’re going to back these up for a rainy day. Just issue these commands:


$ sudo su -
Password:
:~ root# mv /usr/bin/ruby /usr/bin/ruby.orig
:~ root# mv /usr/bin/gem /usr/bin/gem.orig
:~ root# mv /usr/bin/rails /usr/bin/rails.orig
:~ root# logout

Now we’ll go ahead and install a fresh copy of Ruby and RubyGems via MacPorts.

sudo port install ruby rb-rubygems

You should now see something like this for a bit…

Let’s watch a video about bumble bees.

When it finishes installing, you should check that Ruby is available to you and installed in /opt/local/bin.

We’ll also take a second to create a symlink for this as some tools seem to rely on /usr/bin/ruby being there.

sudo ln -s /opt/local/bin/ruby /usr/bin/ruby

Great, let’s move on.

Passenger (mod_rails)

Now that we have Ruby installed, we’re going to take a quick detour to setup Passenger with the Apache server already available on your machine. I’ve been a big fan of using Passenger for your development for over a year now.

sudo gem install passenger

Once the gem is finished installing, you’ll need to install the apache2 module with the following command:

It’ll ask you to continue by pressing Enter. At this point, it’ll check that you have all the necessary dependencies and then compile everything needed for Apache2.

Now I’ll force you to watch a highlights reel of Fernando Torres… the best striker in the world!

The passenger install will then show you this output, which you’ll want to stop and read for a moment and highlight the following:

Then using vi or emacs, you’ll want to create a new file with the following content:

vi /etc/apache2/other/passenger.conf

Then paste in the following (what you highlighted and copied above.)


LoadModule passenger_module /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.9/ext/apache2/mod_passenger.so
PassengerRoot /opt/local/lib/ruby/gems/1.8/gems/passenger-2.2.9
PassengerRuby /opt/local/bin/ruby

You’ll also want to include the following below what you just pasted.


  # Set the default environment to development
  RailsEnv development

  # Which directory do you want Apache to be able to look into for projects?
  <Directory "/Users/ryangensel/development">
      Order allow,deny
      Allow from all
  </Directory>

You’ll want to quickly start up your web sharing, which will start Apache2 up via your System Preferences.

Simple enough… moving forward.

Passenger Pref Pane

To make things as simple as possible, I’d encourage you to install the Passenger Preference Pane (view this post for a download).

Development directory

I have a directory named development/ in my home directory, which is where I end up storing all of my projects. This should match whatever you put above in the apache configuration (<Directory "/Users/ryangensel/development">).

mkdir development; cd development;

Installing Ruby on Rails via RubyGems

Now we’ll use RubyGems to install the latest version of Ruby on Rails (and all of it’s dependencies).

sudo gem install rails

While this is installing, you can watch a video from my old band that ended around the time that business started picking up for Planet Argon.

Great, let’s test out the install of Rails…

Test Rails and Passenger

In your development directory, let’s quickly a new Rails app…

rails testapp

This will generate a new Rails application in a testapp/ directory.

Now open up the Passenger Preferences Pane and add this directory as a new application.

Press Apply…

You should now fire up your browser of choice and head to http://testapp.local. If all has worked, you’ll see a, “Welcome aboard” screen from the Ruby on Rails application.

Assuming that this worked for you, let’s take a quick break to make some tea…

Phase Three

In this last phase, we’re going to install a few database servers and corresponding rubygems so that you can get to work.

PostgreSQL

At Planet Argon, we build our web applications on top of PostgreSQL. I’ve been a long-time advocate of it and hope you consider using it yourself.

At this point in time, the current stable version of PostgreSQL via MacPorts is 8.4.x. Let’s install that now…

sudo port install postgresql84 postgresql84-server

Once this finishes compiling, you’ll need to run the following commands to setup a new PostgreSQL database.


sudo mkdir -p /opt/local/var/db/postgresql84/defaultdb
sudo chown postgres:postgres /opt/local/var/db/postgresql84/defaultdb
sudo su postgres -c '/opt/local/lib/postgresql84/bin/initdb -D /opt/local/var/db/postgresql84/defaultdb'

Assuming that you want PostgreSQL to always be running, you can run:

sudo launchctl load -w /Library/LaunchDaemons/org.macports.postgresql84-server.plist

...and to start it right now, run:

sudo launchctl start org.macports.postgresql84-server

Before you can start using it, we’ll need to make sure that the PostgreSQL executables are available in your shell path. Since you’re now using oh-my-zsh, you’ll want to edit ~/.zshrc with your favorite editor.

vi ~/.zshrc

Just append this to export PATH= line in the file.

:/opt/local/lib/postgresql84/bin

Your PATH might look something like the following now:

@# Customize to your needs… export PATH=/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/opt/local/lib/postgresql84/bin@

Setup database user

To setup a new database (with superuser credentials), just run:

createuser --superuser ryangensel -U postgres

We’ll now test creating a database:

createdb test_db

Let’s test that we can access it…


➜  ~  psql test_db
psql (8.4.2)
Type "help" for help.

test_db=# \q

Great, let’s drop it now.


➜  ~  dropdb test_db
➜  ~  psql test_db
psql: FATAL:  database "test_db" does not exist
➜  ~

Okay, we’ll now install the library that will allow Ruby to talk to PostgreSQL.

Just run: sudo gem install pg

Voila… let’s move on to the inferior database…

MySQL

We’re going to run through the installation of MySQL really quickly because you might need it.

sudo port install mysql5 mysql5-server

This took ages on my machine… so let’s watch a video.

We’ll now setup the database and make sure it starts on system boot.


 sudo -u _mysql mysql_install_db5
 sudo launchctl load -w /Library/LaunchDaemons/org.macports.mysql5.plist
 sudo launchctl start org.macports.mysql5

Let’s test that we can create a database now (and that it’s running.)


 ➜  ~  mysql5 -u root
 Welcome to the MySQL monitor.  Commands end with ; or \g.
 Your MySQL connection id is 3
 Server version: 5.1.43 Source distribution

 Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

 mysql> create database test1;
 Query OK, 1 row affected (0.00 sec)

 mysql> \q

Great, we’ll now install the library that will allow Ruby to talk to MySQL.

sudo gem install mysql -- --with-mysql-config=/opt/local/lib/mysql5/bin/mysql_config

That should be it!

Phase Four, next steps

Okay… so we’ve installed XCode, MacPorts, Ruby, Rails, PostgreSQL, MySQL… and I’ve also got you to switch your default terminal shell from bash to zsh. You might take a look over the available themes for Oh My Zsh so that you can personalize your terminal experience even further.

You also now have a handful of gems installed as you can see with gem list.

Closing thoughts…

This is the fourth version of this guide and I’ve appreciated the hundreds of comments, questions, and emails that I have received… let’s not forget all those beers that people buy me when I’m at conferences. :-)

I hope you have found some of this useful. If you have any problems and/or questions, don’t hesitate to post them in the comments section below.

Using BETWEEN for SQL comparisons

Posted by Sat, 14 Nov 2009 20:55:00 GMT

Recently, Carlos, suggested that I should start sharing some basic SQL tips that help with performance and/or general usage. I recently came across some code that I didn’t like to read and/or write. For example, let’s take the following…


SELECT * FROM brochures WHERE published_at <= now() AND archived_at >= now()

Essentially, this is pulling back some data WHERE the the brochures are considered published. (We have a project that allows people to manage their brochure launch dates ahead of time.) In fact, in this project, we have no less than 6-8 dates in the database that we’re comparing data on and it’s easy to get lost in the logic when trying to understand it.

Now, there isn’t anything inheriently wrong with how this condition is constuctued. As a matter of personal taste, I find it annoying to mentally parse. Also, I find having to write now() more than once in a WHERE clause to feel like I’m repeating myself.

Read it outloud…

“WHERE the brochures published at date is less than and/or equal to right now AND the archived date is greater than and/or equal to now.”

Who talks like that?

Luckily, there is a better and in my opinion, a more readable way to express this is with the BETWEEN construct in SQL. (postgresql docs, mysql docs)


SELECT * FROM brochures WHERE now() BETWEEN published_at AND archived_at

Let’s read this outloud…

“WHERE the current date is between the published at and archived at dates.”

This sounds more natural to me.

Additionally, you can also do the inverse with NOT.


SELECT ... WHERE now() NOT BETWEEN brochures.published_at AND brochures.archive_at

Remember kids, “code is for humans first and computers second.”—Martin Fowler

MySQL is just a toy

Posted by Thu, 11 Sep 2008 18:04:00 GMT

Twitterrific

Are you using PostgreSQL? EnterpriseDB want’s to hear your story at Postgres Rocks

Master/Slave Databases with Ruby on Rails

Posted by Thu, 15 Nov 2007 22:02:00 GMT

Not terribly long ago, I announced Active Delegate, which was a really lightweight plugin that I developed to allow models to talk to multiple databases for specific methods. The plugin worked great for really simple situations, like individual models.. but when it came time to test with associations it fell apart. I haven’t had a chance to work on any updates and knew that it was going to take more work to get it going.

Earlier this week, we helped one of our bigger clients launch their new web site1. For the deployment, we needed to send all writes to a master database and a reads to slaves (initial deployment is talking to almost 10 slaves spread around the globe!). We needed something to get integrated quickly and decided to ditch Active Delegate for the time being and began looking at the following options.

I spoke with Rick Olson2 and he pointed me to a new plugin that he hasn’t really released yet. So, I’m going to do him a favor and announce it for him. Of course… I got his permission first… ;-)

Announcing Masochism!

Masochism3 is a new plugin for Ruby on Rails that allows you to delegate all writes to a master database and reads to a slave database. The configuration process is just a few lines in your environment file and the plugin takes care of the rest.

Installing Masochism

With piston, you can import Masochism with:


  $ cd vendor/plugins
  $ piston import http://ar-code.svn.engineyard.com/plugins/masochism/

You can also install it with the old-fashioned way:


  $ ./script/plugin install -x http://ar-code.svn.engineyard.com/plugins/masochism/

Configuring Masochism

The first thing that you’ll need to do is add another database connection in config/database.yml for master_database. By default, Masochism expects you to have a production database, which will be the read-only/slave database. The master_database will be the connection details for your (you guessed it…) master database.


# config/database.yml  
production:
  database: masochism_slave_database
  adapter: postgresql
  host: slavedb1.hostname.tld
  ...

master_database:
  database: masochism_master_database
  adapter: postgresql
  host: masterdb.hostname.tld
  ...

The idea here is that replication will be handled elsewhere and your application can reap the benefits of talking to the slave database for all of it’s read-only operations and let the master database(s) spend their time writing data.

The next step is to set this up in your environment file. In our scenario, this was config/environments/production.rb.



# Add this to config/environments/production.rb
config.after_initialize do 
  ActiveReload::ConnectionProxy.setup!    
end



Voila, you should be good to go now. As I mentioned, we’ve only been using this for this past week and we’ve had to address a few problems that the initial version of the plugin didn’t address. One of our developers, Andy Delcambre, just posted an article to show how we had a problem with using ActiveRecord observers and masochism, which we’re sending over a patch for now.

As we continue to monitor how this solution works, we’ll report any findings on our blog. In the meantime, I’d be interested in knowing what you’re using to solve this problem. :-)

1 Contiki, a cool travel company we’re working with

2 Rick just moved to Portland… welcome to stump town!

3 The Masochism plugin README

Starting MySQL after upgrading to OS X Leopard

Posted by Sat, 27 Oct 2007 11:13:00 GMT

If you upgraded to OS X Leopard and are running MySQL from the MySQL.com installer1, you might be having some problems with starting it from the GUI interface. There isn’t a fix from MySQL yet, so to get around that… you can run it from the command-line.

Start MySQL from the command line

cd /usr/local/mysql; ./bin/mysqld_safe &

This should get MySQL up and running for you. If someone wants to share a tip on how to get this to start automatically on reboot, please post a comment and I’ll help get the word out.

1 I didn’t have this problem as I installed MySQL via MacPorts... but this came up for a few members of PLANET ARGON after they upgraded to Leopard.

Older posts: 1 2