arjanvandergaag.nl

Working with Postgresql on the command line

2013-03-02T13:00:00Z

Unlike MySQL, PostgreSQL doesn’t have lots of nice-looking GUI tools available to it. But don’t let that hold you back, because its command-line client packs a lot of power that you’ll quickly come to love, once you get to know it.

Editing your queries

You can use the psql program to interactively send queries to your database. Although there are steps you can take to make editing in the interactive shell easier¹, nothing beats your regular editor.

To create a new SQL file in your editor and quickly send it to your database, you can use the --file (or -f) option. Assuming you use Vim, here are two example key mappings:

map <leader>r :w !psql -d mydb -1 -f -<cr>

This allows you to use \r in normal mode to execute the entire file, or in visual mode to execute only the current selection. Note the use of the -1 option (short for --single-transaction) to wrap your queries in a transaction, as if you had used BEGIN/COMMIT, and the - argument to -f to tell psql to read from standard input.

Alternatively, you can start your editor from the psql prompt using the \edit (or \e) metacommand. This will launch $EDITOR to edit a temporary file containing the last query you ran. When you quit the editor, psql will run the contents of that file as the next query.

Finally, you can use PostgreSQL variables and interpolation to make writing queries a little easier. For example:

psql> \set t my_long_table_name
psql> select count(*) from :t;

Note how psql will interpolate :t with my_long_table_name. You might use this to read a blob of data from a file, and insert it into a row:

psql> \set content `cat my_big_textfile`
psql> INSERT INTO posts VALUES (:'content');

The quotes in the placeholder name will escape the variable contents as an SQL literal.

Inspecting tables

The main reason I used to prefer GUIs to work with databases is how easy they make it explore the database schema. psql provides a handy family of metacommands to explore objects in the database, all starting with \d.² You can list all tables in your current schema using just \d. Specify an additional name and psql will tell you all about that named object. For example, use \d comments to list column information of the comments table:

psql> \d comments
                                      Table "public.comments"
     Column                 Type                                   Modifiers                       
---------------- --------------------------- -----------------------------------------------------
id               integer                     not null default nextval('comments_id_seq'::regclass)
body             text                        not null
commentable_id   integer                     not null
commentable_type character varying(255)      not null
user_id          integer                     not null
created_at       timestamp without time zone not null
updated_at       timestamp without time zone not null
Indexes:
    "comments_pkey" PRIMARY KEY, btree (id)
    "index_comments_on_commentable_id_and_commentable_type" btree (commentable_id, commentable_type)
    "index_comments_on_user_id" btree (user_id)

To zoom in on the comments_id_seq sequence, use \d comments_id_seq:

    Sequence "public.comments_id_seq"
   Column      Type          Value        
------------- ------- -------------------
sequence_name name    comments_id_seq
last_value    bigint  375
start_value   bigint  1
increment_by  bigint  1
max_value     bigint  9223372036854775807
min_value     bigint  1
cache_value   bigint  1
log_cnt       bigint  21
is_cycled     boolean f
is_called     boolean t
Owned by: public.comments.id

I’ve found these commands tell me all I need to know about my database, without the need to take my hands off the keyboard.

Working with results

When querying your data, psql might give you a lot of data — way more than a single terminal’s screen full. There are several ways to make working with such data a little easier.

Customize the pager

psql will page through the query results using $PAGER, usually defaulting to more.³ If you prefer less (and why wouldn’t you?), you can set the $PAGER environment variable or use \setenv PAGER less at the prompt.

Customize the presentation

psql will draw pretty borders between your query columns. You can turn them off using \pset border 0, or add even more borders with \pset border 2. It’s also nice to use pretty unicode characters to draw those borders by setting \pset linestyle unicode. Finally, use wrapped mode to wrap content in columns with \pset format wrapped to prevent your columns from running wider than your screen.

When there’s so much content and wrapped mode won’t cut it anymore, switch to vertical mode using \x. This will display a single column per row, making many columns or long text values much easier to read.

Inspect data in external programs

Sometimes you just want to open your results in your editor or a spreadsheet for further analysis. Use \o data in the interactive shell (or the command line argument --output data) to redirect all output to the file ./data. Note that data might also be a script that accepts query results on standard input.

Tip: for some quick and dirty CSV generation, issue \f ',' to set the field separator to ,, \a to switch to unaligned output mode and \t to show tuples but no headers or footers. Or, from your shell: psql -d mydb -t -A -F,.

Other tweaks and tips

One nice customisation to make is tweaking the psql command prompt. Mine looks like this:

arjan@mydb
=# SELECT count(*) FROM posts;

The prompt is defined by the special PROMPT1, PROMPT2 and PROMPT3 variables. You’ll usually see PROMPT1; PROMPT2 is used when entering queries across multiple lines. Some of the special substitutions you can use are:

%n: The current user name
%/: The current database
%#: > for regular users, # for database superusers.
%R: Expected input indicator, hinting when you have unbalanced quotes or missed a semicolon.
%x: Current transaction status: nothing when there’s no transaction, * if there is, ! if the transaction has failed.

Store preferences in .psqlrc

If you tweak your psql preferences it would be nice not to have to reapply them in every session. Store your customisations in ~/.psqlrc to issue them at the start of every session. My .psqlrc file looks like this:

\set PROMPT1 '%n@%/\n%R%x%# '
\set PROMPT2 '%R%x%# '
\pset border 1
\pset format wrapped
\pset linestyle unicode
\pset null NULL

You can then keep that file under source control, along with all your other dotfiles.

Running single commands

When you are really in a hurry, or are trying to practice some shell-scripting magic, you can use the --command (or -c) option to psql to run a one-off command:

$ psql -d mydb -c 'select count(*) from users;' -A -t
58

Using connection strings

If you have ever wanted to connect to a remote Heroku postgres database, you know how easy it is to get the connection details from Heroku:

$ heroku config:get DATABASE_URL -a myapp
postgres://username@password:very-long-hostname:5432/database

Try converting that to the appropriate command-line options to psql to do some manual prodding around. But, turns out, psql will accept such a connection URI just fine:

$ psql `heroku config:get DATABASE_URL -a myapp`

In this particular example, you might as well run heroku pg:psql as that does essentially the same thing. But it’s good to know that you can do this, if you want to.

Conclusion

Using the psql program is much easier than it may seem. Its a good Unix citizen and it is very simple to integrate into your regular editor. This interoperability also makes it easy to customise your workflow even further with scripts and aliases – for example, to automatically connect to the database described in ./config/database.yml if you’re in a Rails project, or to automatically download and import data dumps from remote production databases. Such scripts are left as an exercise to the reader.

For example, if you are a Vim user, try adding “set editing-mode vi” to ~/.inputrc to enable Vim key bindings in most interactive shells, including psql.↩
There’s a whole range of \d metacommands, so read the psql man pages for more information.↩
The exact program used differs per operating system, but on Mac OS X it seems to default to more.↩

Programming Awk for fun and profit

2013-01-05T13:00:00Z

I recently came to learn and love Awk, a “pattern-directed scanning and processing language”. If you find yourself working with text on a command line often, you owe it to yourself to learn some Awk.

Awk is valuable to learn, because in many cases it can replace complex Unix pipelines consisting of sed, grep, cut, wc and tr. It offers a concise language for text processing that makes it easier to use than Perl or Ruby. And above all, Awk is so small, it is easy to learn.

Awk’s main purpose is to extract information from structured text, and report it in newly combined or aggregated ways.

The components of an Awk program

Records and fields

The Awk programming language revolves around records and fields. Awk’s purpose is to parse input into records, parse records into fields, operate on those records to generate output records and fields.

In true Unix fashion, by default a record consists of a single line, with fields separated by whitespace. Naturally, you can customize delimiters using the special variables RS and FS for Record Separator and Field Separator and their output-specific companions ORS and OFS.

Note that FS is actually a regular expression, so you can tell Awk to split on various punctuation characters at once using a value like [| ,()].

Rules

An Awk program always loops over all the records from its input to execute rules. A rule consists of a pattern and an action:

pattern { action }

A rule’s action is executed when its pattern matches the current record. The default pattern matches any record; the default action is to print the current record.

Multiple rules in a single program are separated by newlines or semicolons.

Patterns

Patterns can match record by regular expression, by range or by boolean expression. Here are a few examples:

By regular expression: /foo/: Matches when the entire record contains “foo” somewhere.
By boolean expression: $2 ~ /foo/: Matches when the second field matches the regular expression foo.
By boolean expression: NR % 2: Matches all uneven records (lines), because NR is a special variable containing the current record number and 0 is falsy.
By range: /foo/, /bar/: Using the comma notation, this matches every record from the first to contain foo up to and including the next that contains bar. For example, you can print a table definition from a Ruby on Rails db/schema.rb file using awk '/create_table "users"/, /^ end/' db/schema.rb.
BEGIN and END: Run a block of code before or after looping over all the input. This is useful for setting up arrays, modifying special variables and presenting aggregated results.

Actions

While the default action of printing the current action already makes Awk an interesting alternative to grep, custom actions make Awk truly powerful.

Actions can contain entire programs using loops, variables, functions and conditionals – but in the end they usually come down to printing information to the output stream.

For example, you can print columns from the input record using the special $ function:¹

{ print $2, $1 }

This program prints the first two columns of every record in reverse order. To have a little more control over output formatting, you can use the familiar printf function:

{ printf "%6s: $%.2f", $1, $2 }

This will print the first input column as six characters wide, right-aligned string; and the second as monetary value.

Advanced Awk programming

Awk is a surprisingly complete programming language. Consider the following example program:

BEGIN { max = 0 }
{ if($1 > max) max = $1 }
END { print max }

This simple program demonstrates the use of variables and conditionals to find the biggest value in the first column.

Consider the following example:

{ categories[$2]++ }
END {
  for(key in categories) {
    print key, categories[key]
  }
}

This program builds a histogram of the second column, outputting each unique value found in the second column of its input, following by how many times that value was found. Note the use of an associative array and the for(key in array) loop.

Finally, Awk allows you to use a set of built-in functions and even define your own.

Running Awk programs

The simplest way to use Awk is on the command line, for example to print known usernames:

$ awk -F: '{ print $1 }' /etc/passwd

Awk will read its first non-option argument as the program to run, and any other arguments as files to read. Awk will gladly accept multiple files as one big input stream. Of course, it will also work fine in a Unix pipeline.

When your programs get too large to comfortably write at a command prompt, you could put it in a file and tell Awk to run it:

$ awk -f my_program.awk /etc/passwd

Alternatively, as Awk also functions as an interpreter, you can write your Awk programs in an executable file. In a file called usernames.awk:

#!/usr/bin/awk -f
BEGIN { FS = ":" }
{ print $1 }

Make the file executable with chmod +x usernames.awk and invoke it with ./usernames.awk /etc/passwd.

Example program

Here is an example (prettified) script that parses some server log files, containing information about products fetched from a remote server and the total amount of time that took. We want to create a histogram showing time intervals and the total number of products in that category. The logs contain some lines that look like this:

[CatalogItem] Retrieved 12 product, expected 46 (1837.8ms)

This program is composed of several Awk scripts piped together, with a sort thrown in as well.

$ awk -F '[ ()]|ms' '
  # Construct an array of categories of average request times
  # and the number of products in that category.
  /\[CatalogItem\] Retrieved/ {
    number_of_products = $6
    total_request_time = $8
    average_response_time = nummber_of_products / total_request_time
    categories[int(average_response_time)] += number_of_products
  }

  # When all lines have been read, print the results
  END {
    for(label in categories) {
      print label, categories[l]
    }
  }
  ' production.log
  |
  awk '
    BEGIN {
      max = 0
    }
    # Loop over all records to find the highest frequency
    # in the histogram
    {
      if($2 > max) {
        max = $2
      }
      categories[$1] = $2
    }
    # Print the histogram again, but relate all frequencies to the
    # maximum we found
    END {
      for(label in categories) {
        printf("%5d %d\n", label, categories[label] / max)
      }
    }
  '
  |
  awk '
    {
      # Construct a visual bar sized by the histogram frequency
      # in the second column. Note that Awk's string concatenation
      # operator is the space.
      bar = ""
      size = $2
      while(size-- > 0) {
        bar = bar "#"
      }

      # Print the original histogram labels and the generated bar.
      printf("%6d %s\n", $1, bar)
    }
  '
  |
  sort -n

This program will output something like:

     0 ##
   100 ######
   200 ###############
   300 #########

Conclusion

Search around the web and you’ll find dozens of pages with Awk one-liners to do all sorts of crazy text manipulation stuff. They’re a good resource to pick up new ideas from, but the power of Awk lies not in building libraries of text processing functions, but rather in the ability to quickly whip up custom scripts to handle unique circumstances. Master Awk and be creative!

Since $ is a function, you could use expressions to retrieve a dynamic column number, i.e. n = NR % 2; print $n.↩

Modular front-end code in Rails

2012-08-23T10:00:00Z

Maintainable stylesheets of non-trivial size need a modular design — so much has been established by now. How can you incorporate this style of front-end coding in a typical Rails application?

Modular CSS

With ‘Modular CSS’ I mean a range of new ideas in front-end web development aiming to solve some of the issues with giant catch-all .css-files, rife with duplication and specificity conflicts. These approaches typically revolve around defining “objects”, “modules” or “components” in your stylesheets, in order to reduce the cost of change by:

increasing style re-use and, by extension, consistency;
creating a shared language among team members;
decoupling markup from styles.

To learn more about these ideas, check out Nicole Sullivan’s Object-Oriented CSS, Jonathan Snook’s SMACSS, Yandex’ BEM and keep an eye out for Roy Tomeij’s book Modular Frontend.

Modular front-end code in Rails apps

Historically, Rails has “forced” many best practices on developers by providing conventions for trivial matters (see Yehuda Katz’s RailsConf 2012 talk for examples and background). But to properly modularize our front-end code, we have to fight Rails a little bit and rid ourselves of the “one controller, one stylesheet, one script” default file layout. At the same time, Rails does provide us with some nice tools to make our jobs easier: the asset pipeline, Sass included by default, and RecordTagHelper.

Note: in this post I’m ignoring the Javascript aspect of front-end code on purpose. That’s a story for another time.

Marking up objects

The modularized front-end revolves around identifying the building blocks of your web pages, like a post, a footer or a horizontal list. Rails’ content_tag_for is a great help to mark these objects up:

<%= content_tag_for(:article, @posts) do |post| %>
  <h2><%= post.title %></h2>
  <div class="content">
    <%= markdown post.body %>
  </div>
<% end %>

This might generate the following output:

<article id="post_1" class="post">
  <h2>My example post</h2>
  <div class="content">
    <p>Foo bar</p>
  </div>
</article>
<article id="post_2" class="post">
  <h2>Another example post</h2>
  <div class="content">
    <p>baz qux</p>
  </div>
</article>

Rails gives us a convention for naming and identifying objects. You would typically provide styles for the .post class, and attach javascript behaviour to individual posts using the #post_1 and #post_2 IDs.

Note: There is also a div_for tag, which does basically the same as content_tag_for but omits the first argument and always gives you a div. For more information, see the documentation on ActionView::Helpers::RecordTagHelper.

If Haml is more your cup of tea, you could write:

%div[post]
  %h2= post.title

If you are not dealing with instances of ActiveRecord::Base, you can still use these helpers by defining some extra methods on your objects, as these classes and IDs are generated as "#{object.param_key}_#{object.to_key}". Take a look at ActiveModel::Naming and ActionController::RecordIdentifier for details.

Object inheritence

You will usually find yourself needing to go one step further and apply object inheritence: a post can take many different forms on your site, ranging from full articles, excerpts in an index or links in a sidebar. Rails can still help us:

<%- content_tag_for(:article, @posts, :excerpt) do |post| %>
  ...
<% end %>

Will give us:

<article id="excerpt_post_123" class="excerpt_post">
  ...
</article>

Although I would have preferred dashes insteads of underscores, Rails still makes a trivial decision on how to name stuff for us, so let’s stick with it.

Partials, paths and STI

It is a natural fit to use a partial for every object you define. So, to render blog post excerpts on your front page, you will end up with a posts/_excerpt_post partial and a modules/posts/excerpt_post.css.scss stylesheet. It is now trivial to render an excerpt post object anywere in your application:

<%= render partial: 'posts/excerpt_post', collection: @posts %>

Should you create custom Ruby objects — such as decorated models or presenters that group multiple persistence objects into a single business object — you may want to define to_partial_path on it to tell Rails how to render them:

require 'delegate'
class ArticlePresenter < DelegateClass(Post)
  def to_partial_path
    'posts/article_post'
  end
end

Which would allow you to render an instance of ArticlePresenter like so:

<%= render article_presenter %>

…and have it use the posts/_article_post.html.erb template.

Finally, Rails’ Single Table Inheritance can get in your way. Consider you have set up your models so that both Post and Page inherit from a generic Node model. Using content_tag_for will give you .post and .page classes. If you want to use the same styles for all Node subtypes, you could apply the same styles to both classes in your stylesheet, or you could use ActiveRecord::Base#becomes:

<%= content_tag_for(:div, @post.becomes(Node)) do %>
  <h2><%= @post.title %></h2>
<% end %>

becomes will give you a new Node object with all the attributes of the @post object, so you will end up with the proper .node class on your object. You can also use this trick to render child class using the partial that would have been used for the parent class:

<%= render @posts.map { |p| p.becomes(Node) } %>

See the documentation on #becomes and read Henrik Nyh’s notes on using becomes with form helpers.

Styling objects

Now you’ve got your .post object and various sub types, such as .excerpt_post set up. These are now trivial to style using Sass and the @extend directive:

.post {
  h2 {
    color: #f00;
    margin: 20px 0;
  }
}

.excerpt_post {
    @extend .post;
  h2 {
    margin: 0;
  }
}

Which would result in something like:

.post h2, .excerpt_post h2 {
  color: #f00;
  margin: 20px 0;
}
.excerpt_post h2 {
  margin: 0;
}

With Sass 3.2, you might even keep the base .post class from the final style sheet — since you will most probably never use it on its own. Just use a placeholder selector:

%post {
  color: #333;
}
.excerpt_post {
  @extend %post;
  font-size: 12px;
}
.article_post {
  @extend %post;
  font-size: 14px;
}

This leads to the base class not being output, only the extensions:

.excerpt_post, .article_post {
  color: #333;
}
.excerpt_post {
  font-size: 12px;
}
.article_post {
  font-size: 14px;
}

Catches

Much like object-oriented programming, we can define generic styles for all post objects, and add special styles to special instances. Also much like object-oriented programming, note that that you ‘child’-objects should not differ too much from your ‘parent’-object; even though you are technically displaying a post object, it may be a better idea to classify your sidebar link item as a “link” rather than a “post”. Here, Rails leaves you on your own and you will have to write your markup yourself:

<li id="link_<%= post.id -%>" class="link">
  <%= link_to post.title, post %>
</li>

This is a nuisance, but it would be trivial to write a helper method for it. I haven’t come across it much in practice.

Don’t forget about mixins

Regardless, it is clear it makes sense to think about your object hierarchy before your build it. Also, as Ruby itself shows us with its modules, not everything is modelled best with a hieriarchical structure. Consider mixins to share styles across object types:

@mixin sidebar_link {
  font-weight: bold;
}
.link_post {
  @include sidebar_link;
}
.link_comment {
  @include sidebar_link;
}

Organising your files

Using the aforementioned techniques, you will quickly end up with a bunch of styled objects. It is probably best to abandon Rails’ convention of one-stylesheet-per-controller and keep each in a separate file using the module pattern:

app/
|--assets/
|  |--stylesheets/
|  |  |--application.css
|  |  |--layout.css.scss
|  |  |--elements.css.scss
|  |  |--modules/
|  |  |  |--post.css.scss
|  |  |  |--excerpt_post.css.scss
|  |  |  |--link_post.css.scss

Tip: When using Rails generators, you can provide the --no-assets or --no-stylesheets flag (along with --no-helper) to not generate the regular asset files you are not going to use anyway.

Your modules directory quickly clutters up so you had better group them in subdirectories:

app/
|--assets/
|  |--stylesheets/
|  |  |--application.css
|  |  |--layout.css.scss
|  |  |--elements.css.scss
|  |  |--modules/
|  |  |  |--posts/
|  |  |  |  |--post.css.scss
|  |  |  |  |--excerpt_post.css.scss
|  |  |  |  |--link_post.css.scss
|  |  |  |--comments/

Concatenating assets

With so many files lying around, it would of course be great if you could include them all in one fell swoop, rather than importing each by hand in the application.css manifest file. Sprockets can do that for you:

/*
 * in app/assets/stylesheets/application.css
 * require_tree .
 */

Done! But Sprockets, alas, will first compile every file and then concatenate them together. Now Sass will no longer be able to use mixins and variables from one file in another. This is a disappointment, so you will have to bypass Sprockets for concatenation and let Sass handle it for you. Rename application.css to application.css.scss and let the sass-rails gem handle the importing for your:

// in app/assets/stylesheets/application.css.scss
@import "modules/**/*"

This approach imports your modules sorted alphabetically, which is a great way to foce yourself to write truly independent modules that do not depend on the order in which they are defined. In practice, this might not work out, in which case you will have to revert to importing every module by hand in your manifest stylesheet.

Conclusion

There’s a little bit of friction when adopting these methods in your Rails application, but for the most part it is easy to get started and Rails has some nice convenience methods for us to use. With a little bit of discipline you can set up a very neat and organised front-end codebase — just make sure your approach is documented so all your team members are on the same page with you.

If you have any other tips, let me know on Twitter!

Vim advanced search and replace

2012-07-01T10:00:00Z

When I get a few days off from work, I do what every self-respecting geek would do: I dive into my text editor and try to learn some new stuff. Here’s a few tricks I learned recently concerning search and replace in Vim.

1. Indenting a multi-line search match

Say you have a Markdown document with code blocks marked up in Github-style ‘fences’:

Here is a code example:

```ruby
puts "Hello, world"
```

You want to convert these code blocks to standard Markdown code blocks – which have no fences, but instead are indented four spaces. This is desired end result:

Here is a code example:

    puts "Hello, world"

The example includes only a single code block with a single line of code, but let’s assume there’s several multi-line code blocks. How to go about this problem?

My first attempt was to use a regular expression search to match the entire code block, and then find some way to perform the indent command on its matches. The regular expression is not too hard, once you know the relevant Vim regular expression syntax:

/\v^```ruby\_.{-1,}```$/

Note the \v for very magic syntax, \_. to match any character (including newlines) and {-1,} to non-greedily match at least one of \_..

This search matches all the fenced code blocks alright, so we need a way to operate on every match. Here the global command (g) comes into play. Here’s my first attempt to indent every match:

:g/\v```ruby\_.{-1,}```$/>

The global command allows you to operate on every line matching a pattern, in this case using the > command to indent lines. Alas, only the first line of the match would indent, giving me:

Here is a code example:

    ```ruby
puts "Hello, world"
```

My misunderstanding was that a multi-line search would match multiple lines – instead, a search in Vim is found on a single line (which makes sense when you think about it).

After some Googling for search operations, I found the global command sets the current line in vim to the line of the search match. From there, you can start a new range up to the closing fence and operate on that range with the indent command. Here’s how:

:g/^```ruby/.,/```$/>

So, to recap: g/^```ruby/ searches for our opening fence and sets Vim’s current line to that line. The rest of the command is the operation on that line. .,/```$/ is a range. We might express ranges with line numbers (i.e. 10,20), but we can also use special symbols and searches. In this case, . is the current line (the one with the opening fence), and we end our range on the first match from our current line that matches our closing line. Finally, we can operate on a range using any ex command¹, in this case > to indent. Success!

2. Solve your problems with external tools

It later occured to me the above problem might just as well have been solved using Ruby. Here’s an example solution:

:%!ruby -pe "gsub /^/, '    ' if /^```ruby/../```$/"

We use %! to filter the entire file through the ruby program. The -p flag to Ruby tells it to loop over STDIN and print every line back to STDOUT. The -e flag to Ruby evaluates a string of Ruby code. Since -p put Ruby in the special command-line mode, gsub and and regular expressions are implicitly sent to the current line. The above string of Ruby code is equivalent to the following program:

while $_ = STDIN.gets
  if $_ =~ /^```ruby/../```$/
    $_.gsub! /^/, '    '
  end
  print $_
end

Finally, the regular expression range syntax (also known as the flip-flop operator) is one of those awesome Ruby tricks few people know: looping over several lines, it becomes true when the left-hand regex matches, and stays true on subsequent tests until the right-hand side regex matches.

Is the Ruby alternative more readable than the Vim one? You decide for yourself – just know there’s more than one way to skin a cat.

3. Adding an incrementing counter to search matches

A while back I had an HTML document with sixteen headers I wanted to prefix with an incrementing number. I wanted to turn this:

<h3>Heading 1</h3>
<p>Lorem ipsum</p>
<h3>Heading 2</h3>
<p>Lorem ipsum</p>

…into this:

<h3>1. Heading 1</h3>
<p>Lorem ipsum</p>
<h3>2. Heading 2</h3>
<p>Lorem ipsum</p>

I accomplished it with Vim’s search-and-replace capabilities and a little vimscript, taking advantage of \=, which allows us to use an expression in the substitue string.

The first step to solving this problem is matching the right characters for replacement. In this case, we want to insert our counter right after the opening heading tag. We can match that exact position like so:

/\v\<h3\>\zs/

We have to escape the < and > because in Vim’s regular expression syntax they represent left en right word boundaries. \zs is Vim’s positive look-behind syntax, causing Vim to look for our opening heading tag, but only starting our match after it.

Using vimscript to increment a counter is not too hard, either. We could do it like so:

let i=1
" insert i in the document somehow
let i=i+1

My first try to combine these was to just use a regular substitue command, combining it with the vimscript lines for the counter:

:let i=1 | %s/\v\<h3\>\zs/\=i/ | let i=i+1

You can use | to combine multiple statements into a single command. \=i will use the current value of i as the the substituion string. But here’s the end result:

<h3>1Heading 1</h3>
<p>Lorem ipsum</p>
<h3>1Heading 2</h3>
<p>Lorem ipsum</p>

Vim has inserted the same value of i on every match. Here’s why: %s is a single operation on a range (%) that includes the entire document. If only we could make a single replacement for every line that matched our search… The global command to the rescue, again:

:let i=1 | g/\v\<h3\>\zs/s//\=i/ | let i=i+1

I replaced the % range with a global command using the same search pattern. To not repeat myself, I removed the search pattern from the substitution command, causing vim to re-use to last used pattern – which in this case is the pattern from the global command. Miraculously, it works:

<h3>1Heading 1</h3>
<p>Lorem ipsum</p>
<h3>2Heading 2</h3>
<p>Lorem ipsum</p>

We just need to add a little extra text to the substitution string:

:let i=1 | g/\v\<h3\>\zs/s//\=i.". "/ | let i=i+1

As you can see, we can concatenate the string ". " to our counter using the . operator, giving us the end result:

<h3>1. Heading 1</h3>
<p>Lorem ipsum</p>
<h3>2. Heading 2</h3>
<p>Lorem ipsum</p>

Another great success!

4. Editing complex commands

Admittedly, we have created quite a monster command using vimscript, a global command, a substition command, the special expression register and positive look-behinds. But Vim is a developer’s editor, and developers (should) know this stuff. It is not the most readable of code, but it is not horrible either.

The only truly awkward aspect of typing complex commands like these is entering them on Vim’s command line. There’s a remedy for that, though: the command line window.

The command line window is a new window (a split screen) in the current tab, that contains a buffer with your entire command history, one command per line. You can move around and edit commands just like in any buffer, making it easy to make changes. But when you press Enter in normal mode, it will execute the current line as if you had typed it on Vim’s command line, against the previously active window.

The command line window is a great help in iteratively building up complex commands. It allows you to quickly revisit previously issued commands, make some changes, and execute it again. When done, you can simply close it again using :q. But Vim wouldn’t be Vim if it hadn’t some awkardness, and this time it’s the q: shortcut to open it. It just way too similar to :q, and I open the command-line window by accident all the time.

For a full list of available ex commands, search the Vim help system using: :help holy-grail.↩

FactoryGirl Tips and Tricks

2012-06-20T19:00:00Z

FactoryGirl is an awesome fixture replacement library that gives you a lot of power and flexibility, at the cost of more code to maintain and increased mental overhead. It pays get to know it, so you can wield its flexibility to improve your tests and your productivity.

Get the most out of FactoryGirl

1. Use traits for modular factories

Traits are reusable pieces of attribute definitions that you can mix and match into your factories. Traits are to factories what modules are to classes a much more natural and flexible way of sharing common behaviour.

For example, say you have a Post and a Page object that both have a publication date. You could use inheritance to create various combinations:

FactoryGirl.define do
  factory :post do
    title 'New post'

    factory :draft_post do
      published_at nil
    end

    factory :published_post do
      published_at Date.new(2012, 12, 3)
        end
      end

  factory :page do
    title 'New page'
    
    factory :draft_page do
      published_at nil
    end

    factory :published_page do
      published_at Date.new(2012, 12, 3)
    end
  end
end

FactoryGirl.create :draft_page
FactoryGirl.create :published_post

The repetition should be obvious. Traits can make this DRY:

FactoryGirl.define do
  factory :post do
    title 'New post'
  end

  factory :page do
    title 'New page'
  end

  trait :published do
    published_at Date.new(2012, 12, 3)
  end

  trait :draft do
    published_at nil
  end
end

FactoryGirl.create :post, :published
FactoryGirl.create :page, :draft

With a simple example the difference might seem trivial, but try to think how quickly the complexity of your inheritance chain would increase if you had not one but two, six or twelve different attributes (or sets of attributes) you wanted to be able to apply in different combinations.

Traits are awesome because they can define callbacks, ignored attributes and even nest other traits.

2. Use ignored attributes to tweak callbacks

FactoryGirl lets you define ignored attributes, which will not be set on your newly created object. This is surprisingly useful in combination with dependent attributes and callbacks, which do get to access them.

Consider an example of a blog post with comments:

FactoryGirl.define do
  factory :comment do
    author 'Anonymous'
    body 'Great post, man!'
    approved_at Date.new(2012, 3, 6)
    post
  end

  factory :post do
    title 'New post'
  end

  trait :with_comments do
    after :create do |post|
      FactoryGirl.create_list :comment, 3, :post => post
    end
  end
end

It is trivial to create a post with three comments by applying the with_comments trait to a factory invocation. But what if we wanted to adjust the number of comments? We can use an ignored attribute:

trait :with_comments do
  ignore do
    number_of_comments 3
  end
  
  after :create do |post, evaluator|
    FactoryGirl.create_list :comment, evaluator.number_of_comments, :post => post
  end
end

Note that a special second argument is passed to the callback block, the evaluator, which knows about the ignored attributes. Now you can simply pass in the ignored attribute like you do a regular attribute:

FactoryGirl.create :post, :with_comments, :number_of_comments => 4

Make trivial variations possible with ignored attributes removes the need for tons of almost-identical factories.

3. Create non-generic examples

Your factories are used in tests with as goal to make assertions about their state and behaviour. It can really help to use very specific example objects with well-known attribute values. So don’t create a person_with_three_comments_and_a_post; instead use a person_mike and person_john. Use your factories to let your tests tell a simple story about your objects.

4. Use aliases

FactoryGirl allows you to define aliases to existing factories to make them easier to re-use. This could come in handy when, for example, your Post object has a author attribute that actually refers to an instance of a User class. While normally FactoryGirl can infer the factory name from the association name, in this case it will look for a author factory in vain. So, alias your user factory:

FactoryGirl.define do
  factory :user, :aliases => [:author] do
    username 'anonymous'
  end

  factory :post do
    author # => populated with the user factory
  end
end

5. allow setting up common associations

When you have many business models with many associations to other business models, you quickly end up with tests that first have to employ a dozen objects before the object under test is in such a state that meaningful queries can be made about it. This might be a sign of bad design, but could very well be unavoidable. In case of the latter, consider creating traits and factories that preload such associations for you, as with the example of a post with comments:

FactoryGirl.define do
  factory :post do
    title 'New post'
  end

  trait :with_comments do
    after :create do |post|
      FactoryGirl.create_list :comment, 3, :post => post
    end
  end
end

FactoryGirl.create :post, :with_comments

Common pitfalls when creating factories

1. Do not use random attribute values

One common pattern is to use a fake data library (like Faker or Forgery) to generate random values on the fly. This may seem attractive for names, email addresses or telephone numbers, but it serves no real purpose. Creating unique values is simple enough with sequences:

FactoryGirl.define do
  sequence(:title) { |n| "Example title #{n}" }

  factory :post do
    title
  end
end

FactoryGirl.create(:post).title # => 'Example title 1'

Your randomised data might at some stage trigger unexpected results in your tests, making your factories frustrating to work with. Any value that might affect your test outcome in some way would have to be overridden, meaning:

Over time, you will discover new attributes that cause your test to fail sometimes. This is a frustrating process, since tests might fail only once in every ten or hundred runs – depending on how many attributes and possible values there are, and which combination triggers the bug.
You will have to list every such random attribute in every test to override it, which is silly. So, you create non-random factories, thereby negating any benefit of the original randomness.

One might argue, as Henrik Nyh does, that random values help you discover bugs. While possible, that obviously means you have a bigger problem: holes in your test suite. In the worst case scenario the bug still goes undetected; in the best case scenario you get a cryptic error message that disappears the next time you run the test, making it hard to debug. True, a cryptic error is better than no error, but randomised factories remain a poor substitute for proper unit tests, code review and TDD to prevent these problems.

Randomised factories are therefore not only not worth the effort, they even give you false confidence in your tests, which is worse than having no tests at all.

2. Test for explicit values

In addition to random values being bad, relying on your factories default values may be bad idea, too. Unless you create specific story-telling factories, such as “john” rather than “user1”, you should anticipate someone else (i.e. you, four weeks from now) changing the default factory values. When you are testing, you want to test for explicit values your test controls. A test like this is silly:

FactoryGirl.define do
  factory :post do
    title { Forgery(:lorem_ipsum).words(5) }
  end
end

describe 'Blog' do
  it 'should show the post title on the page' do
    post = FactoryGirl.create :post
    visit '/blog'
    page.should have_content(post.title)
  end
end

What happens when your post title randomly turns out to be an empty string, or a phrase that also happens to occur somewhere else on the page? If your test value is random or outside your control, how can you prove something about it? Consider this improved example and its increased readability:

it 'should show the post title on the page' do
  post = FactoryGirl.create :post, :title => 'My example post'
  visit '/blog'
  page.should have_content('My example post')
end

The explicit title and test value underline the intent of the test and leave far less room for false positives or future changes.

3. Do not use dynamic values by default

Dynamic attribute values are evaluated on invocation time rather than evaluation time. This allows you to use, for example, the current time in your factories:

FactoryGirl.define do
  factory :post do
    created_at Time.now # => will be the same for every object
    published_at { Time.now } # => will be updated for every object
  end
end

This does not mean, however, that you should alway use dynamic attribute values when a simple static value will do fine:

FactoryGirl.define do
  factory :post do
    title { 'My new post' } # silly!
    body 'Lorem ipsum'      # pretty!
  end
end

Although I’m sure there will be minor performance benefits to using static values, that is not the point. We want our tests to be readable and clear. Using blocks everywhere conceals the true meaning of the actually dynamic attributes between the static attributes.

4. Do not manually create associations

When using Rails, FactoryGirl is smart enough to know how to set up associations for you. So don’t do this:

FactoryGirl.define do
  factory :comment do
    post { FactoryGirl.create :post }
  end
end

Usually you can just use another factory name as an attribute, or use the association method to customise it:

FactoryGirl.define do
  factory :comment do
    post
  end

  factory :user do
    username 'anonymous'
  end

  factory :post do
    association :author, :factory => :user, :username => 'admin'
  end
end

Note how the comment factory simply uses the post factory to set its post attribute. The post factory however uses the user factory to populate its author attribute and provides some additional details on how it is to be built.

More tips?

If you have any pet peeves when using FactoryGirl, let me know on Twitter!

arjanvandergaag.nl

Working with Postgresql on the command line

Editing your queries

Inspecting tables

Working with results

Customize the pager

Customize the presentation

Inspect data in external programs

Other tweaks and tips

Store preferences in .psqlrc

Running single commands

Using connection strings

Conclusion

Programming Awk for fun and profit

The components of an Awk program

Records and fields

Rules

Patterns

Actions

Advanced Awk programming

Running Awk programs

Example program

Conclusion

Modular front-end code in Rails

Modular CSS

Modular front-end code in Rails apps

Marking up objects

Object inheritence

Partials, paths and STI

Styling objects

Catches

Don’t forget about mixins

Organising your files

Concatenating assets

Conclusion

Vim advanced search and replace

1. Indenting a multi-line search match

2. Solve your problems with external tools

3. Adding an incrementing counter to search matches

4. Editing complex commands

FactoryGirl Tips and Tricks

Get the most out of FactoryGirl

1. Use traits for modular factories

2. Use ignored attributes to tweak callbacks

3. Create non-generic examples

4. Use aliases

5. allow setting up common associations

Common pitfalls when creating factories

1. Do not use random attribute values

2. Test for explicit values

3. Do not use dynamic values by default

4. Do not manually create associations

Further reading

More tips?