Ralph’s son Henry

At work I have been using ApacheBench to help bench mark performance. It is a fabulous tool. After upgrading my OS to Lion, ApacheBench (ab) fails to run. Here was my error:

Scott ~/src: ab -n 5000 -c 100 http://127.0.0.1:3000/ 
This is ApacheBench, Version 2.3 <$Revision: 655654 $> 
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ 
Licensed to The Apache Software Foundation, http://www.apache.org/ 
Benchmarking 127.0.0.1 (be patient) 
Send request failed! 
apr_socket_recv: Connection reset by peer (54)

Apparently there is a patch to fix it, here, but I could not get it to work. That and the post is in another language, that google did a sos-so job of translating. I had read somewhere that this was fixed in new version of apache, version 2.3.14 (download here), which is in beta. I didn’t need to (or want to) install all of Apache, just ApacheBench so I compiled the source and copied over the binary. Here are the steps:

Scott ~/Desktop: cp ./support/.libs/ab /usr/local/apache2/bin/ab
./configure --prefix=/usr/local/apache2
make
make install
which ab output
--> /usr/sbin/ab
cd /usr/local/apache2/bin/
cp ab  /usr/sbin/ab
sudo cp ab  /usr/sbin/ab

Success! Back to normal, everything is working again.

Rails, Passenger & Nginx Performance Tuning: Quick Start Guide

At work we’ve been testing out rails as a candidate for production use. The company as a whole is generally cautious (read:slow) about adopting new technology, specifically when it gets outside of the jvm. For that reason I’ve been tasked with preformance testing rails against one of our existing frameworks and in doing so, I’ve learned a little bit about the out of the box settings for rails, passenger and ruby. Here is the summary:

1. Passenger – By default fusion passenger is set to 6 instances. From the fusion documentation, this probably needs adjustment based on your hardware:

The optimal value depends on your system’s hardware and the server’s average load. You should experiment with different values. But generally speaking, the value should be at least equal to the number of CPUs (or CPU cores) that you have. If your system has 2 GB of RAM, then we recommend a value of 30. If your system is a Virtual Private Server (VPS) and has about 256 MB RAM, and is also running other services such as MySQL, then we recommend a value of 2.

For our small AWS instance, with 1.7mb of ram, we raised it to 25. In your nginx.config, add:

passenger_max_pool_size: 25

2. REE – We switched from standard Ruby to Enterprise Edition. REE make better use of system memory. RVM makes this a cake walk.

3. Ruby Garbage collection – much has been written about how rails isn’t that compatible with ruby’s default garbage collection settings. You can read the full details here, here or here. A summary of the change is to create a new file with the following in it. Make sure the last line points to the location of ruby.

#!/bin/bash
export RUBY_HEAP_MIN_SLOTS=1250000
export RUBY_HEAP_SLOTS_INCREMENT=100000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=30000000
export RUBY_HEAP_FREE_MIN=12500
exec "/opt/ree/bin/ruby" "$@"

The tell nginx to read the file. In your nginx.config the following and replace “PATH” with the path of your newly created file.

passenger_ruby PATH;

The garbage collection setting will need to be tunned based on what your application actually does. Go here for a collection of setting that 37s, twitter and others have published.

Results

Your mileage may vary, but for us, we say tremendous gains. Before the chagnes our performance/load test was reporting an average page load time of 2.6 seconds. After the tweaks it went to a crazy low 0.8 seconds! Holy crap.

Control.js is a ultra light-weight framework for javascript to standardize name-spacing, organize initialization, and enforce reusability. The framework helps:

* Control the HTML api to javascript object
* Control the name space where the object is created
* Control when each widget initialized

Control.js has a dependency on jQuery. Assuming that is already on the page, to get started simply write:

<a data-control="MyControl" data-config="{ config1: 'something' }">...</a>

When the browser hits onDomReady, Control.js will try to initialize a new instance of “MyControl”, pass it a reference to the “a” tag and the value of the data-config. For step two javascript, define the name space, create your object and your done!

Control.define( 'MyControl' ); 
Control.MyControl = function( el, config ) {
  // guts go here
};

Control.js will detect namespace collisions, keep track of what was initialized, generally act as a hub for javascript objects on the page. All while maintaining a consistent interface in HTML. More documentation here.

This is my first encounter with publishing on github. I worked with the brilliant Jakob Heuser off an idea originally conceived by genius G. Scott Olson.

At a friends recommendation, I read the book Switch: How to Change Things When Change Is Hard. Now I generally don’t read business books, nor do I read much at all, but let me say right off that I loved this book.

The authors outline that any type of change, either personal or organizational can be broken down into three main areas that, if addressed will lead to a successful change. They use an analogy of an elephant and a rider. The elephant is strong, but can be trained to be controlled by the rider. However when the elephant gets tired of listening to the rider, it can do what ever it wants. After all it’s an elephant. The elephant represent emotion and the rider self-control or reason. Those are the first two areas that need attention for change. The third aspect of the analogy is the path, or the course that is plotted for the elephant and rider to travel. The book is devoted to explaining that successful change, requires an appeal to all three. Reason, emotion, and a clear path to get there. It sounds obvious but that is what makes it so effective.

The book is a perfect balance of good story telling backed up by empirical data. The tone of the book is very conversational, which makes it a easy to follow light read. This was key for me, as the subject matter, has the potential to be extremely boring.

So far I talked both my parents and my wife into reading it, all thus far, to glowing reviews.

This week I at work I was playing around with Rail’s ability to internationalize content. I was new to this and ran into two gotchas.

1. Using Multiple Files for translations By default rails munges all translations into a single file. For the sake of organization I wanted to set rails up to make a 1-to-1 pairing of one .yml file per view template. Here’s how I did it.

First in your application.rb file the default config needs to be uncommented:

# config.i18n.load_path += Dir[Rails.root.join('my', 'locales', '*.{rb,yml}').to_s] 
# config.i18n.default_locale = :de

Then it in order have the file be recognized if they are in folders, you need to modify the path:

config.i18n.load_path += Dir[Rails.root.join('config', 'locales', '*/*', '*.{rb,yml}').to_s]

2. Picking up new files

Rails does a great job of “seeing” new properties inside of a .yml file and will do so on save & reload. However if you add a new translation file, then rails needs to be restarted. This took me longer to figure out than I wanted it to, but my excuse is that I am new to rails.

Yesterday Eishay Smith came to talk to familysearch. I knew Eishay from my time at LinkedIn and an interview stint I did at KaChing (now wealthfront). I invited him to come out to talk to our engineering staff about how his company is doing continuous delivery and what some of the challenges they had getting to where they are.

If you aren’t familiar with what wealthfront is doing, the contents of his talk can be found here. In short they have a set up where commit to production happens in about 10 minutes. They run 10s of thousands of tests in about 4 minutes and deploy about 50 times a day. Beyond the basics, here is what I learned:

1. They have never found a bug or issue that couldn’t be solved by a unit test
2. Integration and Acceptance tests all occur after code has been deployed to production
3. Since the deploy so frequently, the need for any agile development process has gone away. That means they don’t do XP, scrum, kanbaan etc. They just get crap done.

I also learned that we at familysearch have aways to go. We’ll get there.

At familysearch I am working with each engineering team to explain how our organization should be “continuously delivering” our software.  Our organization is committed to moving in this direction and I am super excited about it. As we with each team about the new model, engineers often get hung up on the sensational vision of being able to publish such a short cycle. The concerns generally come in one two major camps:

1. Time from commit to deployment isn’t long enough to accurately validate the system
2. What if..< insert something catastrophic >

While there is some shock value associated with what we are planning. I think that both of these concerns are largely tied to fear about not fulling understanding the path to the vision. Which is perfectly okay. Continuous Delivery is a process, not a project. It will take time for all of our systems to be fully compliant. But there is immediate value to be seen from starting with the basic principles. In fact, the deeper I get into the changes required for continuous delivery, it’s clear that the fundamentals are really just good coding & business practices.

1. Culturally understand our ability to release code quickly and often in a competitive advantage
2. Test suites are intended to give confidence in the quality of the code
3. Build & test automation lets engineers focus on engineering
4. Engineering rigor. Hard APIs and modularity are not optional

Is it really more than that?

Six months back I started working for familysearch.org. I wanted to put into print my goals for the time that I work here. Before I get into that lets take stalk of where we are at.
Familyseach.org is broken up into three main components:

1. FamilySearch: records search & marketing support content
2. FamilyTree: organizing user content into a tree/pedigree chart + temple flows
3. Digital Pipeline: Flow for getting digital images from paper records preprocessed for human indexing

The frontend applications are written in or off of JSP, struts+XMLc, JSF, flex, grails, gwt, a custom javascript MVC, mediawiki, phpforum and drupal. Builds are done exclusively in maven. Test coverage isn’t terrible, but it isn’t great either, the big issue being execution time. Deploys are done to either a datacenter or AWS.  Deploys happen four times annually with a short downtime window. The sites header and footer are delivered via javascript and the home page talks about 2.6 seconds to load in chrome.

Taking all of this into account my goals for myself (with and from the organization) are:

1. Unify the front-end code base to a single solutions for:
   - html generation and templating
   - Unified javascript API, name spacing and reusable component library
2. Work to help build out a kick-butt web development team, worthy of respect
3. Help drive the entire organization towards a Continuous Delivery release model

Easy peasy right?

This is a script I wrote a while back and wanted to share. The idea is to make frequently ask question (faq) pages easier to browse by toggling the visibility of the answers.  The solution is unobtrusive and fails quietly if the user doesn’t have javascript enabled.

Example:

Test 1

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

Test 2

Lorem excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

Test 3

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

 
Source: 

var faq = {
   init: function() {
    var dt = YAHOO.util.Dom.getElementsBy( function( el ) {
      return ( YAHOO.util.Dom.hasClass( el.parentNode, 'fancy-faq' ) );
    }, 'dt' );
    for(var i = 0; dt.length > i; i++) {
      new FancyFaq( dt[i] );
    }
  }
}
function FancyFaq( el ) {
  var sibling = YAHOO.util.Dom.getNextSibling( el );
  YAHOO.util.Dom.setStyle( sibling , 'display', 'none' );
  YAHOO.util.Event.on( el, 'click', this.toggle );
}
FancyFaq.prototype = {
  toggle: function() {
    var nextSibiling = YAHOO.util.Dom.getNextSibling( this );
    nextSibiling.style.display = (
      nextSibiling.style.display == 'block'
    ) ? 'none' : 'block';
  }
}
YAHOO.util.Event.on( window, 'load', faq.init );

Download here Includes fancy_faq,js, yahoo-dom-event.js, demo.html and a ReadMe file. Comments / suggestions welcome.