arjanvandergaag.nl atom feed

A Ruby gem for the bol.com developer API

2012-02-02T12:30:00Z

It has bugged me for a long time that I could not access product information from bol.com the same way I could from Amazon.com. When bol.com released their public API some time ago, I still couldn’t. Not easily. So I wrote a Ruby library for the Bol.com API.

The bol.com API allows you to fetch detailed information for common products, such as books, DVDs and toys, search for products by keyword, or list bestsellers. The API is not too complex: it’s just a handful of endpoints, a couple of available query parameters and a request signature.

I built a wrapper around this API that exposes the available operations and handles the difficult parts of request signing and XML parsing. All you, as the developer, have to do is provide the access credentials. All operations give you nice, clean Ruby objects back, such as a Bol::Product.

Getting started

You need to sign up for a developer account first. When your account has been approved, you can request an API key. The key has two parts – one public, one secret – that will be used by bol.com to make sure you really are who you say you are when you make a request.

Once you have both your access key and its secret, you can get started by installing the Ruby gem:

$ gem install bol

An example application

Let’s create a very simple Sinatra application to search the bol.com store (see the completed app in one file). Create an application file app.rb:

require 'sinatra'

get '/' do
  erb :search
end

Add a simple form view:

<form method="post" action="/search">
  <label>Query: <input type="text" name="q"></label>
  <input type="submit">
</form>

Run the app to confirm everything works:

$ ruby -rubygems app.rb

When you browse to http://localhost:4567 you can see the search page, but it doesn’t do anything yet. Let’s add a search action:

post '/search' do
  @products = []
  erb :results
end

And a view to display results:

<ol>
<% @products.each do |product| %>
  <li><%= product.title %></li>
<% end %>
</ol>

Implementing search

We’ve added a /search route that will somehow set an array of products, and then render them to an ordered list on the page. Simple enough. Now to actually search some products:

post '/search' do
  @products = Bol.search params[:q]      
  erb :results
end

Searching the bol.com website is as easy as Bol.search(params[:q]). Restart the app, and try it out. You will get an error, complaining that the gem is not properly configured. We need to provide our access key and its secret, so they can be used to sign our requests:

Bol.configure do |c|
  c.access_key = '123456789'
  c.secret     = 'abcdefghi'
end

When you restart the application, you should be able to search bol.com by keyword and get a list of titles back.

Showing product details

Let’s go one step further and add some product details. We’ll create a new route for that:

get '/product/:id' do
  @product = Bol::Product.find params[:id]
  erb :product
end

We can use Product#find to find a particular product on bol.com by its internal ID. We’ll create a link to the detail page in our search results:

<ol>
<% @products.each do |product| %>
  <li>
    <a href="/product/<%= product.id %>">
      <%= product.title %>
    </a>
  </li>
<% end %>
</ol>

We also need a view for our new action:

<h1><%= @product.title %></h1>
<dl>
  <% @product.attributes.each_pair do |k, v| %>
  <dt><%= k %></dt>
  <dd><pre><%= v %></pre></dd>
  <% end %>
</dl>

Restart the app, perform a search and click on a result. You should see all available product details. Note that there are several sizes of cover image available, and that the author attribute is an Array – there can be more than one, after all.

Scoping search results to categories

Now we want to limit our search results to a particular category of products. Bol.com products are extensively categorized and we can look up those categories and search in them. Let’s add a drop-down list of categories to search to our search form:

get '/' do
  @categories = Bol.categories
  erb :search
end

post '/search' do
  @products = Bol::Scope.new(params[:category_id])
    .search params[:q]      
  erb :results
end

And in the view:

<form method="post" action="/search">
  <select name="category_id">
    <% @categories.each do |category| %>
    <option value="<%= category.id %>">
      <%= category.name %> (<%= category.count %>)
    </option>
    <% end %>
  </select>
  <input type="text" name="q">
  <input type="submit">
</form>

Once we’ve reloaded the application, we can choose a category to search in and get results limited to that category. Categories are nested, so you can get subcategories using Bol::Scope.new(some_id).categories. And there are also refinements, such as groupings by price or brand and other attributes, but they work similarly to categories.

Referral links

Then only referral links remain. It is a good idea to link to products on bol.com, so you can get a kickback on any sales resulting from the traffic you send over. This is not strictly a feature of the developer API, but it is so commonly used, I just threw it in here. You can simply ask a product for its referral URL, given a specific referral ID:

product = Bol.find(params[:id])
product.referral_link('my-site-id')
# => "http://..."

Other features

This example application showcases the basics of the bol gem, but it includes some more:

Ordering and limiting of number of results
Automatic pagination
Joining and subtracting categories and refinements to create complex scopes
List popular or bestselling products

See the project README file for more information.

Example application and source code

That’s all you need to know to get started. You can see the entire example application in this gist. You can find the source for this Ruby gem on Github, where I also keep the API documentation and track issues. The gem is still in beta stage, so it can be field-tested before getting an actual stamp-of-approval by releasing a version 1.0. Try it out and do let me know if you make anything awesome with it!

Shell scripting to the rescue

2012-01-19T19:00:00Z

I love Ruby and tend to use it for everything I can use it for. But I’ve reading up on Unix recently, and I decided to test my newfound knowledge by using standard unix programs to solve a problem. Those who do not know Unix are doomed to re-implement it badly (or so I have been told).

I needed to copy a lot of images from a remote server to my local machine. Since images were constantly being added to the remote server, I wanted to have a repeatable script to download only those images that were listed in a YAML file from another application. So I needed to read the YAML file, find the files listed inside it, and collect those in an archive for easy downloading.

01. Reading input

My input file was in YAML, so the first step is reading that. But since the file is several thousand lines long, we pipe it into head to just print the first few lines:

$ cat images.yml | head
---
- http://host.tld/images/image1.jpg
- http://host.tld/images/image2.jpg
...

The first problem was the first line of three dashes, which I needed to get rid of. Using sed you can actually issue ex commands like in Vim, so this was easy:

$ cat images.yml | sed '1d' | head
- http://host.tld/images/image1.jpg
- http://host.tld/images/image2.jpg
- http://flickr.com/images/image3.jpg
...

This deletes line one, but there’s a saying along the lines of: “if you cat a file and immediately pipe it into something else, something’s wrong”. So, I rewrote it like so:

$ sed 'd' images.yml | head

02. “Parsing” YAML

Then, I needed to get rid of the YAML array element indicators – the dashes starting each line. I could have used sed for that, but I chose cut, which extracts fields from a line, splitting the line on a given delimited into columns. I wanted the second column with a space as delimiter:

$ sed 'd' images.yml | cut -d' ' -f 2 | head
http://host.tld/images/image1.jpg
http://host.tld/images/image2.jpg
http://flickr.com/images/image3.jpg
…

This was starting to look useful.

03. Getting just the image path

There was a problem with the images: all images contained the full URL, and I wanted to get just the path. sed to the rescue, again:

$ sed 'd' images.yml | \
  cut -d' ' -f 2 | \
  sed 's|http://host.tld/||' |\
  head
images/image1.jpg
images/image2.jpg
http://flickr.com/images/image3.jpg

This time, I used a replacement pattern as we would in Vim, only replacing the standard / separator with a | to not have to escape every / in the search string.

04. Getting rid of externally hosted images

This left the problem of externally hosted images. I just gave up on those. Getting rid of those sounded like a task for grep, which can be used to exclude lines matching a pattern:

$ sed 'd' images.yml | \
  cut -d' ' -f 2 | \
  sed 's|http://host.tld/||' |\
  grep -v "flickr" |\
  head
images/image1.jpg
images/image2.jpg
http://amazon.com/images/image4.jpg

This gives a new problem: there are several different external hosts in the file. I only wanted our own. I decided to rewrite the command and use grep to filter out all lines that do contain our own host, and then remove the domain:

$ sed 'd' images.yml | \
  cut -d' ' -f 2 | \
  grep "http://host.tld" |\
  sed 's|http://host.tld/||' |\
  head
images/image1.jpg
images/image2.jpg
images/image5.jpg

05. Combining files into an archive

The next task was to zip up all those files into one big archive for easy downloading from the server to my local machine.

The first idea was to just dump the whole lot into zip, like so:

$ sed 'd' images.yml | \
  cut -d' ' -f 2 | \
  grep "http://host.tld" |\
  sed 's|http://host.tld/||' |\
  zip dump.zip

Alas, that doesn’t work. I started investigating possible solutions, such as using xargs – which mashes a bunch of lines into a single line and feed them as arguments to another program, with some intelligence about the number of arguments a program accepts. After some fiddling, I got frustrated that zip just didn’t read filenames from standard input, so I finally decided to open the zip manual with man zip. Searching the manual for stdin, I found out zip indeed does not read input filenames from standard input by default, but On Mac OS X, there’s the --names-stdin option, while on most other systems there’s -@. There you go, it pays to RTFM.

So, the entire command now looks like this:

$ sed 'd' images.yml | \
  cut -d' ' -f 2 | \
  grep "http://host.tld" |\
  sed 's|http://host.tld/||' |\
  zip dump.zip -@

This does what I wanted it to do quite nicely, but I figured I could do slightly better.

06. Duplicates and thumbnails

One problem was a lot of duplicate images; another was lots of different sizes of the same image – with the original one the only I care about.

Solving duplicates is easy enough using the uniq program:

$ sed 'd' images.yml | \
  cut -d' ' -f 2 | \
  grep "http://host.tld" |\
  sed 's|http://host.tld/||' |\
  uniq |\
  zip dump.zip -@

Then, I want to only use the original image, not the generated thumbnails. I happened to know that generated thumbnails have filenames like original-filename-150x75.jpg. Removing the dimensions at the end of the filename would give me the regular file. My list could very well contain that original file already, but uniq would sort that out. So, there’s one more sed to add:

$ sed 'd' images.yml | \
  cut -d' ' -f 2 | \
  grep "http://host.tld" |\
  sed 's|http://host.tld/||' |\
  sed 's/-\d+x\d+\.jpg/.jpg/' |\
  uniq |\
  zip -9 dump.zip -@

That gave me a dump archive file containing all my images. As I was happy with the result, I tacked on a -9 to enable maximum compression for the archive, shaving a couple of percentage points of the end result file size.

Conclusion

This post might seem long, but the process of developing this command chain was actually rather quick. Feedback is almost instant and there’s a rich collection of tools to get the job done. I’m pretty sure developing a Ruby script doing the same thing would have involved more manual tweaking and looking up documentation.

Generating fancy URLs with Rails

2012-01-13T19:00:00Z

There’s a problem with Rails resourceful routes. It is very easy to generate standard resourceful routes; it is equally easy to match incoming fancy URLs. But DRYly generating fancy URLs is a lot harder than suspected.

Matching fancy incoming routes is easy enough:

get '/posts/:year/:month/:id' => 'posts#show'

Our application will now correctly match URLs like /posts/2011/3/1-hello-world, but see what happens when we try to generate a polymorphic route:

link_to(post.title, post)
# => <a href="/posts/1">hello, world</a>

This is not what we want. Of course, we could set :year and :month parameters manually, but that’s not very DRY. We could also try to override the post_url and post_path methods, but this does not cover all use cases.

Hypothetical solution

Ideally, we would like to return a hash of parameters for the to_param method:

class Post
  def to_param
    { year: created_at.year,
      month: created_at.month,
      id: id }
  end
end

Rails would then have to interpolate these params into the generated URL. Alas, this doesn’t work.

Actual, hacky solution

We can hack around it, tough, by overriding url_for:

class ApplicationController
  def url_for(options = {})
    obj = options[:_positional_args][0]
    if obj === Post
      options[:_positional_keys] = [
        obj.year,
        obj.month,
        obj
      ]
    end
    super(options)
  end
  helper_method :url_for
end

This works fine for the most part – despite being a terrible hack. The helper_method :url_for line does, however, override ActionView’s normal behaviour of generating only paths instead of full URLs. To get around this, we can create a new helper method that uses the controller method:

module ApplicationHelper
  def url_for(options = {})
    return super unless options.is_a? Hash
    controller.url_for(
      options.reverse_merge(only_path: true)
    )
  end
end

Now all URLs are properly generated:

link_to post.title, post
# => <a href="/2011/2/1-hello,world">Hello, world</a>

A note of warning

The code in these examples is extremely simplified. Being a hack, you need to add some proper checks for argument types and the contents of the :_positional_args and :_positional_keys options. Also, you can probably not depend on this behaviour remaining on subsequent versions of Rails. But it still just might come in useful some time…

Customize your ZSH prompt with vcs_info

2011-12-23T07:00:00Z

You’re not cool if you’re don’t display some Git status information in your prompt. If you use ZSH – and you should – it’s quite easy to customize your prompt. ZSH comes with a nice package vcs_info that provides almost all the information you need.

vcs_info is a function that populates a variable for you. This variable can then be used inside your prompt to print information. What information is printed can be controlled by some configurations – which are essentially format strings.

Set up

Assuming you are using ZSH, setting things up is easy enough:

Enable vcs_info.
Call it in a pre-command.
Include the variable in your prompt.

To do so, modify your ~/.zshrc:

autoload -Uz vcs_info
zstyle ':vcs_info:*' enable git svn
precmd() {
    vcs_info
}

Then, include the output variable in your prompt:

setopt prompt_subst
PROMPT='${vcs_info_msg_0_}%# '

This example is a rather minimalist prompt, so make sure you customize it with additional information.

Configuring `vcs_info`

There’s not too much documentation on vcs_info available, but the official docs got enough to get you going. The defaults should be fine, but we wouldn’t be using ZSH and reading articles about vcs_info if we were satisfied with defaults, would we?

Format string

First, set the general format string of your vcs_info_msg_0 variable:

zstyle ':vcs_info:git*' formats "%{$fg[grey]%}%s %{$reset_color%}%r/%S%{$fg[grey]%} %{$fg[blue]%}%b%{$reset_color%}%m%u%c%{$reset_color%} "

This is, admittedly, a little hairy. Let’s remove the color codes:

zstyle ':vcs_info:git*' formats "%s  %r/%S %b %m%u%c% "

That’s a lot clearer. This would look like this:

git my_project/. master %

This prompt includes:

%s: The current version control system, like git or svn.
%r: The name of the root directory of the repository
%S: The current path relative to the repository root directory
%b: Branch information, like master
%m: In case of Git, show information about stashes
%u: Show unchanged changes in the repository
%c: Show staged changes in the repository

There’s a special format string for ‘actions’ – which in practice means a special format for when Git is currently performing a merge or rebase. Mine is identical to the normal format, but with the action string added:

zstyle ':vcs_info:git*' formats "%s  %r/%S %b (%a) %m%u%c% "

Which would result in something like:

git my_project/. master (rebase) %

This is all I need, although some people do write their own functions to display how many commits the current branch is ahead or behind its remote. Knock yourself out if you’re so inclined.

Flags

There are also a couple of flags you can set to influence the vcs_info’s behaviour:

check-for-changes: Disabled by default because it might slow things down, it tell the back-end to check for working-copy changes and staged changes. Enable it with zstyle ':vcs_info:*' check-for-changes true if you want to use the %c and %u sequences.
enable: List the versioning systems to enable. In the example above, I have enable Git and Subversion, but there’s many more – notably Mercurial.

The vcs_info format string, especially with lots of color codes, is not all that readable, but I nonetheless prefer using a ZSH library over hacking together my own functions that parse the output of various Git commands.

Clarify your Git history with merge commits

2011-12-06T19:00:00Z

The Git history is supposed to help you understand what has happened over time in your project. In practice, how stuff happened tends to obscure what happened. Here are two tips for Git merges to help keep your history clean.

Do not create redundant merge commits

Suppose you are working on a branch with a colleague, be it master or some feature branch. You make some commits in your local repository:

O [master, HEAD] Added comments to post
O Added comment model
O [origin/master] Created post model
O Initial commit

Your colleague has also made some in his local repository, and pushed them:

O [origin/master, master] Added author to post
O Created user model
O Created post model
O Initial commit

When you want to push your changes, Git will reject the non-fast forward merge. So you pull the remote changes, which will merge origin/master into your local master branch. Although you can now safely push your changes, you end up with the following history:

O [origin/master, master, HEAD] Merge branch 'master' of remote-repo.git
|\
O | Added comments to post
O | Added comment model
| O  Added author to post
| O Created user model
|/
O Created post model
O Initial commit

This makes it seem there are two different branches of development here, while actually this would have worked fine as a single linear history. This would have been the result had Git used rebase rather than merge to combine the remote and local commits. Simply use:

git pull --rebase

…and it shall be done. Here’s what happens under the hood:

reset your master branch back to origin/master
fetch new commits from the remote master branch
fast-forward master to origin/master
replay all your original commits on the new master branch

This gives you:

O [master, HEAD] Added comments to post
O Added comment model
O [origin/master] Added author to post
O Created user model
O Created post model
O Initial commit

So now you can safely push your changes and be all in sync again. The history now demonstrates all these commits were all part of a single feature.

This behaviour is usually what you want to do, so you might want to make it default:

git config branch.autosetuprebase always

This will ‘always’ set up remote tracking branches to pull using --rebase. There are more options though, so make sure to read the docs (man git-config):

When never, rebase is never automatically set to true. When local, rebase is set to true for tracked branches of other local branches. When remote, rebase is set to true for tracked branches of remote-tracking branches. When always, rebase will be set to true for all tracking branches.

Do create redundant merge commits

Usually, Git prefers to create a linear history, as usually that’s neat. But when you develop your project with feature branches, and you want to look back through your history to study what features are included, when and by whom, you will have to start parsing a lot of commit dates, authors and log messages.

Suppose the following history, where you have added commenting to a post model in a feature branch called comments:

O [comments, HEAD] Added comments to post
O Added comment model
O [master] Added author to post
O Created user model
O Created post model
O Initial commit

You’re done and you want to merge the changes back into master

git checkout master
git merge comments

This will give you:

O [comments, master, HEAD] Added comments to post
O Added comment model
O Added author to post
O Created user model
O Created post model
O Initial commit

Git figured out it only had to move the “master” pointer to point at the same commit as “comments” does – a fast-forward merge. This is neat, but you lose the information that the last two commits were a stand-alone feature. You want to instruct Git to not perform a fast-forward merge, but create an explicit merge commit:

git checkout master
git merge --no-ff comments

This gives you:

O [master, HEAD] Merge branch 'comments'
|\
| O [comments] Added comments to post
| O Added comment model
|/
O Added author to post
O Created user model
O Created post model
O Initial commit

You could now remove the comments branch (git branch -d comments) and push your commits.

If you like the --no-ff behaviour, you might want to configure Git (version 1.7.6 and later) to use it by default (although I personally don’t do that):

git config --add merge.ff false

Although the end result is more noisy, it does convey useful information: both that there was a group of related commits that constitute a feature; and what it was named. You might even consider not immediately creating a commit, and adjusting the commit message:

git checkout master
git merge --no-ff --no-commit comments
git commit -m "User story #4831: As\
a visitor I can comment on posts"

Conclusion

Git merge commits can both clutter your history or help structure it. Create them to indicate groups of related commits; omit them to avoid the illusion of unrelated commits.

arjanvandergaag.nl atom feed

A Ruby gem for the bol.com developer API

Getting started

An example application

Implementing search

Showing product details

Scoping search results to categories

Referral links

Other features

Example application and source code

Shell scripting to the rescue

01. Reading input

02. “Parsing” YAML

03. Getting just the image path

04. Getting rid of externally hosted images

05. Combining files into an archive

06. Duplicates and thumbnails

Conclusion

Generating fancy URLs with Rails

Hypothetical solution

Actual, hacky solution

A note of warning

Customize your ZSH prompt with vcs_info

Set up

Configuring vcs_info

Format string

Flags

Clarify your Git history with merge commits

Do not create redundant merge commits

Do create redundant merge commits

Conclusion

Configuring `vcs_info`