Leonard Garvey, on programming, ruby, mongo and more

Dear Backbone. Love letters from a Rails dev.

2013-05-16

In this blog post I'm going to present a few things. Most of all I'm going to demonstrate how I would implement nice, structured front-end code in an already existing traditional Rails application. I'm going to do this using Backbone but it's probably unlike how you may have seen Backbone used before. Finally I'm also going to present the technique I use to integrate this in with your Rails application. Importantly I think that we should be adopting sensibile application defaults in addition to the defaults that Rails provides us. By adopting our own per-application defaults we can avoid writing lots of code.

On front-end libraries/frameworks

Before I get started I'd like to state that Backbone is a library and not a framework so most comparisons between Backbone and Ember or Angular aren't really relevant. But I would like to demonstrate why I chose Backbone instead of Ember or Angular since it's such a commonly asked question.

I like Backbone because it doesn't make assumptions about how my application is built. Most importantly it doesn't assume that I suddenly have a well-tested JSON API that is suitable for a web interface.

Traditionally with Backbone, Ember or Angular you query the server and get back JSON. Then you load that JSON into a "model" which is used by the framework to render a template in memory which is then inserted into the DOM. There are a few problems here:

Assuming I have a good JSON API.
Forcing me to either abandon my Rails views and to start writing sections of my app in JST (or similar), or to maintain both.
It means my app doesn't work without JavaScript. My issue here is that analytics tools generally don't capture users who don't have JS enabled. It's difficult to get a sense of how large this market but I'd tend to think that it's a small but vocal minority.

Why I use Rails

Rails is unabashedly an opinionated framework. Many of us Rails developers do what we do because (for the most part) we approve of those opinions. What led me down this path was trying to work in Ember into an existing fairly traditional Rails application. I was in the situation where suddenly I needed to either abandon my existing views or duplicate my HTML across precompiled JavaScript views.

Personally, I want to work with Rails. Rails has been developed over all these years as an amazing DSL for concatenating strings together to form HTML. I don't want JS to take over any of that responsibility. I'm happy to build JSON APIs but I think that much of the time we'll throw together a poorly thought out ad-hoc action that calls to_json on some model, particularly if we're retro-fitting some fancy MVC thing into our front-end.

I will be going through a simplistic and largely contrived example of how I've implemented this previously and how this works in both Backbone and Rails. I think that we're often missing a good direction on how to leverage the strengths of both JavaScript and Rails in our applications.

Build your applications to work with plain HTML. I think this is good practice.
Enhance your views with JavaScript even going so far as to use the History API to make a truly rich application.

Backbone provides my JS some structure, lets me write as little jQuery crap as I possibly can.

The code

All the code is actually available on Github. Check it out here.

So lets get into it. Here's a brief description of the feature I'd like to build:

Feature: AJAX commenting
  In order to have a nice commenting experience
  As a troll
  I want to be able to submit comments without a page refresh

And here's my Rails ERB view. There's absolutely nothing special here. It's a standard form for submitting a comment.

<%= form_for [@post, Comment.new] do |f| %>
  <p>
    <%= f.label :body, "New comment" %><br/>
    <%= f.text_area :body %>
  </p>
  <p><%= f.submit "Add comment" %></p>
<% end %>

Now here is the Backbone "view" which I can bind to the form above.

QuickBlog.Views.Comments ||= {}

class QuickBlog.Views.Comments.CreateCommentView extends Backbone.View
  events:
    "submit": "commentSubmitted"

  commentSubmitted: (e) =>
    e.preventDefault()
    @submitForm()

  url: ->
    @$el.attr('action')

  data: ->
    @$el.serialize()

  submitForm: ->
    $.post(@url(), @data(), @commentPosted)

  commentPosted: (data) =>
    @appendComment(data)
    @resetForm()

  resetForm: ->
    @$el.map ->
      @reset()

  appendComment: (comment) ->
    $('#comments').append(comment)

In order to bind that view to my form I only need to write: new QuickBlog.Views.Comments.CreateCommentView({el: $('#new_comment')}). Right now I do this in my ERB views as inline JavaScript but it would be totally fine to do it in CoffeeScript too.

This view listens and intercepts the submit event, then it posts it off using AJAX. Once that's successfully occured the commentPosted callback is executed with the data that is returned from the server.

Traditionally with Backbone we'd return JSON from the server, stick it in a Backbone model then there'd be a change event in the view which would update the DOM by rendering a template. BLAAAHHH so many things.

I'd rather just return a HTML snippet which gets put straight into the DOM. More code on the Backend and simpler JS. I've got two options. I could take the approach of rails-ujs which alters the Accepts HTTP Header to prefer JS. This allows us to write:

respond_with(@comment) do |format|
  format.html { redirect_to post }
  format.js { render @comment, :partial => true }
end

But I'm bad at JavaScript and don't want to constantly set accept headers so that I get th right format in Rails so instead we end up with this code:

if xhr?
  render @comment, :partial => true
else
  redirect_to post
end

But that code sucks. How about this instead:

respond_with(@comment) do |format|
  format.html { redirect_to post }
end

That's better but we need to do a couple of things to set that up.

respond_to :html, :xhr in the controller.
Mime::Type.register_alias "text/html", :xhr to setup a new mime type in config/initializers/mime_types.rb
Then create and use a new responder. It looks like:

class XhrResponder < ActionController::Responder
  def xhr?
    @controller.request.xhr?
  end

  # xhr is "rendered" when you request html but do so via AJAX.
  #
  def respond
    if xhr? && format == :html
      method = "to_xhr"
    else
      method = "to_#{format}"
    end
    respond_to?(method) ? send(method) : to_format
  end

  # render the appropriate partial for the resource
  #
  def to_xhr
    controller.default_render(@resource, options.merge(:partial => true))
  end
end

and to use it in your controller:

  private

  def self.responder
    XhrResponder
  end

Summary

I think this is a really nice way of enhancing existing traditional Rails applications with some front-end magic while at the same time doing as little of that crazy JavaScript stuff as possible.

PS: This blog post was taken from the notes I used to give this talk at the Ruby or Rails Oceania (RORO) meetup in Sydney on May 14th 2013.

RubyConf AU was Awesome

2013-02-25

The first (and hopefully annual) RubyConf AU was held in Melbourne in late February and as the title "suggests" it was pretty awesome. 380 Ruby and Rails geeks descended upon the Jasper Hotel for 2 days packed full of top quality talks, by amazing speakers. On the day preceeding the conference proper there was a full day of workshops.

Workshops

I attended a Code Retreat with Corey Haines. If you haven't been to a Code Retreat then you absolutely have to go. I felt like I levelled up as a developer several times in a single day and I'm very keen to go along to more. Basically the idea is that in pairs you do several 50ish minute programming spikes attempting to implement Conway's Game of Life. After each iteration you throw away your code and start again. Finally, to make things interesting Corey places restrictions on code you can write which forces you to think about your language and how it can be used to solve the problem given the restrictions.

Other workshops (which were all apparently fantastic) were:

Advanced Ruby with Dave Thomas
JRuby with Charles Nutter and Hiro Asari
Rails for iOS developers with Lori Olsen and Tim Breutkreutz
A Rails Girls workshop

The Conference itself

Every talk I attended was fantastic but there were some particular highlights:

Obviously all the keynotes were brilliant.
Keith and Mario's guide to fast websites was really top notch.
Carina Zona's talk about schemas in the real world was very eye-opening.
Ben Orenstein's live-coding refactoring session was fantastic too.

You can watch all the sessions on Vimeo

I do have some criticisms:

While the talks were fantastic it would have been nicer to see some more diversity in the speaker lineup. Hopefully next year we can put more effort into balancing things out a bit.
Lack of wireless and no 3G connectivity at the venue. There are arguments for and against here, but I feel that it negatively harms future conferences for there to be less "buzz" about our conference on Twitter. That said I'd prefer no internet to really bad internet.

I'm very much looking forward to RubyConf AU 2014!

Founders Five Tech Concerns

2013-01-11

Working in Tank Stream Labs (a great new co-working space here in Sydney) I come into contact with plenty of entrepreneurs looking to start their own Next Big Thing. Sometimes they have plenty of technology experience in the founding team, but often it's "business" people looking to start a great new technology-based startup. They're eager to learn anything about creating an online business. So rather than explaining everything to them in person, my hope is that I can direct them to here and give everyone a nice easy list of tech things you should have covered in your new startup.

These tips fall into a few key areas but here goes!

Owning your stuff

1) Your code - A business is worth as much as the assets it possesses. One of the biggest assets will be the code used to power the product. If you're going to get a contractor/consultancy to build your product for you make sure you still own the code. The best way of doing this is to make sure your developers put it into a Github account that you own. It's really cheap and extremely easy to setup.

Ask your contractor what version control system they use. Full marks if they use git. If they don't use version control the chances of your code being lost increase dramatically. All good developers use some sort of version control. Also if they don't use version control it's much harder for a different contractor to fix any issues with your system.

2) Your online properties - One of the next largest assets will be your domain and web hosting. While you'll probably be guided by your developer on what the best option here is, it's vitally important you understand exactly what you're paying for. It's also important you understand how to move to a different hosting solution should the need arise. If they say they host with AWS, Heroku, or a reputable hosting company like Anchor you're in safe hands.

If you're able, create all accounts yourself and create secure passwords with a tool like 1Password. You can then easily create accounts for your developers or share passwords with them if required.

3) Backup your data (and code). It's imperative you backup your data and your code. Imagine if your website gets hacked and all the data is wiped. What is your disaster recovery plan? At minimum get your developer to setup an automated backup script. It's pretty easy to backup to Amazon S3 or Glacier. If you store your code on Github it's pretty safe.

Choosing your Developer

4) Testing - How does the developer know the site works if it isn't tested? Ask any prospective contractor what sort of testing they recommend for your system. Make sure they have automated tests that get run every time before code is deployed to your website. If they don't do this it's an indicator of bad quality.

Ideally your developer will mention the words "Continuous Integration" and possibly talk about "Continuous Deployment". If they do it's a great thing, it means your dev can push to production multiple times a day.

5) Community - What sort of community does the developer belong to? Believe it or not, tech people network to. There are plenty of tech meetups in any large city covering all types of technology from Ruby on Rails, to PHP and MongoDB. If your developer is part of that community it's an indicator that they are passionate about development because they spend their own time pursuing it.

Both these are positive indicators, although many would consider no testing a very big negative. It's completely possible to find a fantastic solitary developer who doesn't attend community events and don't test.

Do you have any other concerns that you think I could answer? Get it contact with me, either by commenting below or tweeting me @lgarvey.

Bundler and Gemfile

2013-01-10

With the very recent and extremely serious Rails security vulnerability there have been more than a few people around asking how they update their Rails apps in order to be safe. Luckily Rails makes the update process relatively easy for any simple apps. If you have a large app with a large team, you probably shouldn't be getting basic advice from a blog on the internet anyway!

How to Update

I'll be assuming you're running Rails 3.2.x.

Open your Gemfile
Make sure there's a line like: gem 'rails', '~> 3.2.0' in there.
Run: bundle update rails
Run your test suite (you do have a test suite... right?!)
Deploy your code.

It's really that easy!

What is happening here

Understanding those steps is probably just as important as actually performing them. So I'm going to break it down for anyone reading this.

The Gemfile and Gemfile.lock are used by bundler to manage your application dependencies. You declare your application dependencies in your Gemfile and then you run bundle install to create the Gemfile.lock file. When your Rails application starts it reads the Gemfile.lock file so it can figure out which gems to require. This work is done in your config/boot.rb file. You'll see something like:

require 'rubygems'
gemfile = File.expand_path('../../../../Gemfile', __FILE__)

if File.exist?(gemfile)
  ENV['BUNDLE_GEMFILE'] = gemfile
  require 'bundler'
  Bundler.setup
end

$:.unshift File.expand_path('../../../../lib', __FILE__)

Source: https://github.com/rails/rails/blob/3-2-stable/railties/lib/rails/generators/rails/plugin_new/templates/rails/boot.rb

The Bundler.setup message requires the gems in your Gemfile.

What is the Gemfile.lock then?

The Gemfile.lock file is created by bundler (normally when you run bundle install. This file contains a snapshot of all the gems (and their dependencies) used by your application. This means that if you share your application with someone else, when they run bundle install bundler will skip the dependency calculation process and simply install the Gems specified in the Gemfile.lock. *You should always commit both the Gemfile and Gemfile.lock to your source control system (you should be using git).

The bundler website has some great information on this and other topics. I'd recommend starting with: http://gembundler.com/rationale.html

2013 Resolutions

2013-01-09

Over the past few years I've been setting goals for myself, and achieving most of them but it becomes easy to weasle out of some goals if they aren't written down. So in the spirit of @awws's post 2013 New Year's Resolutions I've decided to write up my goals for the year and post them publically.

Health

Towards the end of last year I got a personal trainer (Rebecca of Becompetitive Fitness is amazing) and started my fitness journey. I lost ~6kg over 2 months and put on a bit of muscle. More importantly than the numbers I broke my Coca-Cola habit, and established new routines which involve me going to the gym multiple times per week. I found myself sleeping better and experiencing less ups and down related to sugar levels. I've still got a bit of work to do here (cutting out all sweets for instance).

So my goal for this year is to step it up a notch:

Lose 15kg. (Actually I'd like to hit this by June but it would be pretty close to my goal weight)
Keep an average of 4 exercise sessions per week.
Cut out the afternoon and after-dinner sugary treats. I'll be able to incoporate dessert into my weekly cheat meal instead.
Take up a martial art.

Overall I just want to be able to feel good about my fitness and health. One more thing that I'm doing is taking up a martial art. I've decided to give Krav Maga a go and have booked an induction session for later this month.

Personal

Last year I travelled through Western Europe with my lovely girlfriend, capping off the trip with 3 days snowboarding in the Dolomites.

This year I'd like to do some more snowboarding, and to plan (but possibly not execute) my next big trip.

Spend at least a week snowboarding.
Plan the next big proper holiday.
Get on top of my financial situation. This is already in progress with another housemate moving in shortly.

Professional

In 2012 I made a huge change in my career moving firmly into a Rails focused role as Lead Developer for Airtasker. I also taught a Ruby on Rails focused course for General Assembly at the end of the year.

Start contributing to an Open Source project.
Pick up some small contract work outside of work hours.
See where the Airtasker ride takes me over the year.
I'd really love to teach again.
Continue to participate in the Ruby community going to events like RoRo, Railscamp and the upcoming RubyConf.
Blog at least once a week.

Overall I feel that my fitness goals are going to be the most challenging, but they're also my most important ones for this year. Last year I really changed up my professional life with a big job change, focusing heavily on Rails and web development, and then teaching later in the year.

Cucumber

2013-01-08

Cucumber is a common tool used in many Rails applications to provide Acceptance/Integration testing. It uses a DSL called Gherkin which is a Business Readable DSL.

The idea is that you write the features of your application into Scenarios which are then executed by Cucumber. As a developer you also provide the steps to execute those Scenarios often using a tool like Capybara.

In the Rails community Cucumber can be slightly contraversial but I've come to believe that Cucumber is greatly misunderstood within the community.

@twe4ked I’ve come to really like Cucumber. I think it gets an unfair rap from a lot of devs.
— Leonard Garvey (@lgarvey) December 13, 2012

A really great conversation followed on Twitter, but I do often feel constrained by 140 characters.

Is love too strong a word?

I really like using Cucumber. Whenever I write a Feature I feel like I'm focused implementing that functionality, even if the very next thing I do is write a few specs for models and controllers. I feel that if I ever get sidetracked I can get right back into the flow by just running my Cuke.

It's for this reason that I completely agree with Chris Parsons when he says that Cucumber isn't a testing tool

A sales issue

Besides the whole web-steps awfulness, which is extremely well explained by Jonas Nicklas in his great post You're cuking it wrong, I think Cucumber has a sales problem.

@mwotton @lgarvey User stories are a great way to describe a system. Cucumber's problem is claiming that you can run them with little effort
— Warren Seen (@warren_s) December 13, 2012

Understanding how to set it up as a newbie can be difficult. It isn't well described on the Cucumber website
It promises a great deal: "The text is written in a business-readable domain-specific language and serves as documentation, automated tests and development-aid - all rolled into one format" The issue with this type of promise is that it doesn't seem to fit in with how most devs view the world. They have automated tests (written in Ruby!) so don't see the need for more wordy abstraction. They view the Features as something else to maintain in addition to tests and documentation.
Cucumber is often sold as a way for Business people and devs to communicate. I'd argue instead that it's a way for devs and anyone else to communicate, you don't get benefit from Cucumber only if you can convince your Business Analyst to read your feature files. You should get benefit from Cucumber ever time you on-board a new developer, every time you need to remind yourself, or another dev, how a complex feature works (at least at a high level).

If Cucumber was instead pitched as a tool that helped you keep (and communicate) simple system requirements and your app in-sync then I think it would get more acceptance.

Expected: [] Got: [] - Ruby Object Equality

2012-12-13

Recently while working on Airtasker we got the following error from our (pretty extensive) Rspec test suite:

Failures:

  1) UsersController#show with user actions gets user actions
     Failure/Error: assigns(:actions).should == actions
       expected: []
            got: [] (using ==)
     # ./spec/controllers/users_controller_spec.rb:51:in `block (4 levels) in <top (required)>'

Finished in 4.46 seconds
11 examples, 1 failure

The fix ended up being fairly simple, simply reverse the output which causes the test to pass. This led to one of my collegues commenting that:

One of the empty arrays was backwards.

Not sure if Serious

I'm fairly certain he was joking, but I wanted to know exactly what was happening. I was able to replicate the issue in the rails console just by manually executing the test (which is a fantastic way of understanding code)

irb(main):018:0> a
=> []
irb(main):019:0> b
=> []
irb(main):020:0> a == b
=> false

It turns out that the the a and b are ActiveRecord::Relation From there it was pretty easy to figure out what was happening, the Relation class implements the equality method == which Rspec was using. Because both objects are Relations it compares their sql rather than the result. When reversing the Relation it executes the sql and compares them as an array.

Lessons Learned

There are a few cool lessons here:

Ruby isn't magic, source code is pretty easy to read.
Try using irb or a debugger to figure out more about your particular issue.
In your custom objects implement the equality method, or even better include comparable and implement <=> which will give you all of the comparison methods for free!

Deleting you Softly

2012-12-11

A tale of woe

You're building a great new app and the product manager/business person says:

Hey we're going to need to be able to delete things, and then restore things that get deleted

You say:

Sure no problem, I'll just include this gem that lets me soft delete and make a few code changes to my models

You've already made a huge mistake, the damage is done just take your hands off the keyboard.

What's the issue?

The problem is that your database (and therefore ActiveRecord) is the source of all truth within your application. You've just made that truth inconsistent and more difficult to define, truth becomes truthish and much pain ensues.

I could leave this blog post there, but I think some justification is in order. First I need to explain what I mean by "soft delete" and explain roughly how that's implemented. Hopefully you'll soon agree with me that it's generally a bad idea.

Soft Delete

Soft delete is a process where instead of deleting records in your database you simply flag them as deleted. In Rails this might mean you use permanent_records which lets you use a deleted_at timestamp field in your models to mark the record as "deleted". Also you'll end up defining a default scope which filters out all those "deleted" records:

default_scope where(:deleted_at => nil)

What permanent_records does is overrides the standard destroy and destroy_all methods and instead just marks this records with the time they were "deleted". It's smart enough to follow dependencies such as has_many and so on.

The Problem Revealed

There are a few issues:

If you destroy an object that isn't soft deleted, but has dependant objects that are soft-deletable then the parent object is actually deleted, while the child objects are only soft deleted. You can't revive them or work with this like normal objects in this case.
Admin panels where you can view soft deleted objects by using unscoped become problematic, business people begin to abuse the deletion functionality.
If you're viewing a "deleted" record (for example in the Admin panel) none of the associations will work because they'll revert to using the default_scope of the association.

Ultimately the problem is that you're not letting your database manage truth. The database itself is fantastic at enforcing Foreign Key constraints and it's something that I think Rails developers aren't particularly good at.

Partial Solutions

Make sure you are extremely disciplined about setting up your dependant_destroy declarations in your models. This is a remarkably easy trap to fall into, and takes constant developer awareness in order to avoid. A different strategy is to use after_destroy (or before_destroy) callbacks to flag dependant records as deleted.

after_destroy :clean_up_your_mess

Train your business and administration staff to not think that deleting and restoring is a good tactic to try. This isn't a computer to be turned off then turned back on again in a futile effort to fix the issue. Because of caching bugs it can be easy to get into the situation where non-technical staff think that the record needs to be deleted and restored before it works properly (don't laugh). And because of timestamps on many models this deletion and restoration will actually cause a cache miss if you're using fragment caching like you should be!
There is soem horribly awful code you can put into your Rails application which will fix the scoped -> unscoped association issue.

class ActiveRecord::Associations::Association
  def scoped
    if owner.is_permanent? && owner.deleted_at?
      association_scope
    else
      target_scope.merge(association_scope)
    end
  end
end

Note: this code will need to be changed in Rails 4 where scoped has been deprecated in favour of scope.

Complete Solution

Don't use soft delete

More helpfully, don't use soft delete and propose an alternative solution when this requirement comes up. It's a common requirement to never actually "delete" anything from the database, so argue in favour of never deleting something. It's better for your business if you keep all that valuable data.

If you do need to delete something, simply serialize it to json and keep a record of it. I've thought of creating a deleted_things table where I can throw all the things that get deleted. This way it's not impossible to revive an object, but it is sufficiently difficult that you won't want to do it regularly.

Using a Primary Key Factory for Mongoid

2012-03-19

I'm developing a joke application for a bit of fun and to fill out my github with some more public code. So I'm creating Blitter which lets me do some playing and development with mongodb and rails using mongoid.

One of the things I wanted to support was querying of User objects by the name of the user rather than the _id. Obviously I could have just altered the show action in users_controller.rb to query by name:

@user = User.find(params[:name])

The problem with this is it breaks the standard rails url helpers. You get the following error: Mongoid::Errors::InvalidFind in UsersController#show. So instead what we can do is use the name of each user as the primary key of the class!

The mongodb ruby driver says:

A primary key factory is a class you supply to a DB object that knows how to generate _id values. If you want to control _id values or even their types, using a PK factory lets you do so.

The problem with Mongoid is that there isn't an easy way to use a primary key factory. No problems!

Create config/initializers/mongoid.rb and paste the following:

class PKFactory
  def create_pk(row)
    puts "creating #{row.inspect}"
    row['_id'] = row['name']
    row
  end
end

Mongoid.configure do |config|
  puts config.inspect
  config.master = Mongo::Connection.new.db(Mongoid.database.name, :pk => PKFactory.new)
end

Afterwards just change your user model:

field _id, :type => String

See https://gist.github.com/2072345 or the commit in blitter: https://github.com/lengarvey/blitter/commit/4cc76b7ede518c8d112e66a49e18ad1090081af7

Super easy!

MongoDB Sydney

2012-03-18

Yesterday I attended 10gen's first Australian MongoDB conference. While most of the sessions were relatively basic it left me burning to continue working on some MongoDB projects I've been working with lately. The most informative session of the day was on sharding. I don't expect to have cause to use sharding in production any time soon but I have a better understanding of how the sharding key works to balance data across the cluster. Certain assumptions I had previously had were completely wrong so it's great to get those cleared up.

The biggest project I've had is experimentation with the aggregation framework which is available in MongoDB 2.1 (2.2 for stable).

Installing MongoDB 2.1 on OSX

If you aren't using homebrew you should be. To install the current version of MongoDB it's as simple as

brew install mongodb

but if you want to experiment with the Aggregation Framework which won't be stable until 2.2 you can install 2.1 with the following instructions:

Update the mongodb recipe with the following gist https://gist.github.com/2006403 :

brew edit mongodb

Install mongodb 2.1!

brew install mongodb

Still Running with MapReduce

Even after playing with the Agg framework I've still found that my needs require map reduce. I'd love to use the Agg Framework since it's meant to be more performant, it's written in C++ so doesn't block single Javascript server thread, but I just can't figure out how!

Unfortunately I can't really share the concept since it's fairly work related.

MapReducing

Reading on stackoverflow and the mongodb-user google group definitely reveals that many people have issues with map reduce. I think it's pretty simple:

You loop over all the documents (map)
You emit some stuff from those documents (emit)
You combine all the stuff you emit down using some function (reduce)

The most common reduce is to just add things together. One little trick I noticed (and should have realised before) was that MongoDB provides an Array.prototype.sum function. So in the MongoDB shell the following code works:

Array.sum([1,2,3,4,5])
// => returns 1+2+3+4+5 = 15!