Matthias Nehlsen

Software, Data and Stuff

Weekly Update: AngularJS book, BirdWatch and Clojure, Web Components, Upstart and Play

In this weekly update, I will be talking about Clojure and ClojureScript, an upcoming article on AngularJS, Grunt, Karma and Protractor and how I use Upstart to run my Play applications as services on Ubuntu.

AngularJS book available for pre-order on Amazon, Meetup

I am very excited that the book that Amit Gharat and I wrote about AngularJS UI Development is now available for pre-order on Amazon. Interesting experience to see it being listed already while still working on the final stages.

We will have content available for preview soon. On my end, I will publish an article about setting up the environment for AngularJS with a build system consisting of Grunt, Bower, Karma and Protractor. Last week I also gave a talk on the subject at the Hamburg AngularJS meetup. Or rather a live coding session in which we coded up an application that is being tested in Chrome, Firefox, PhantomJS and also in Mobile Safari. Android is still missing due to some difficulties but should be added soon. Two interesting questions came up during the meetup: a) How do you test swipe gestures? b) When selecting elements to click by id, will Protractor catch errors if an element, though visible, is covered by another element and thus cannot be clicked? Tests should also be added for cases such as these. The project the article will be based upon is already available on GitHub. You may find it to be a good starting point for an AngularJS application that includes a test and build system. Besides the planned article I also want to do a screencast on the subject. Stay tuned. By the way, you could help with testing the application on Android or adding the additional test cases.

Tweet stream analysis with Clojure and ClojureScript/Om

I recently rewrote the client side of my BirdWatch application using ClojureScript and Om. While I enjoyed the process, I also noticed some performance issues when I tried to keep the application responsive while ingesting and transforming thousands of previous tweets as quickly as possible. While optimizations certainly could have been done on the client side alone, this was a good reminder that the information flow architecture was far from ideal. The previous version was also a bit of a Frankenstein’s patchwork of programming languages. I acknowledge that it may be a little bit of a tough sell to have to understand both Scala and Clojure in order to wrap your head around a single application. Totally unnecessary, too.

So I rewrote the server side using Clojure. That already works nicely, this time making use of WebSockets instead of Server-Sent Events. Turns out WebSockets are a nice fit conceptually for CSP-style channels.

A couple of things still need to be resolved. In order to fully achieve the previous functionality, there needs to be an auto-reconnect if the connection to the Twitter Streaming API is lost. That should be really simple, given what is there already. Also, matching new tweets with the clients’ queries using ElasticSearch’s Percolator and only delivering a filtered stream needs to be implemented. This is slightly more involved, but also seems totally doable.

Above and beyond, it seems useful to partially perform the computation (ranking by retweets, word counts) on the server in order to reduce the load on the client. With Clojure and ClojureScript being so similar, my idea is to use a part of the code base on both sides and split the computation. Then, the amount of data having to traverse the potentially slow network connection would be reduced drastically, leading to faster loading and a more mobile-friendly memory footprint of the client.

A new article on this application rewrite will follow soon. The work in progress code is on Github already, currently in a separate branch.

Web Components / Polymer / X-Tag resources

As already mentioned last week, I find some of the ideas behind Web Components brilliant, in particular Shadow DOM and Custom Elements. Now that I am learning these concepts anyway, I thought I might as well share useful resources I come across, so I have created a list over on GitHub. Check it out and please add your links too. I expect this list to grow substantially in the next couple of weeks.

Upstart scripts for Play

I am running live instances of my Play applications (BirdWatch, sse-chat, amzn-geo-lookup) on an Ubuntu server. There are hardly any disruptions, something like once every few months. But whenever they occurred, I had to restart the applications manually. Not terrible when all is running smoothly, but not great either. So how could that be done better? Upstart provides the answer. After some googling I found this blog post by Adam Evans. I have only slightly modified it and now, I have all of my applications running as services that I can start and stop the way one would expect, e.g.:

# status birdwatch
birdwatch start/running, process 947
# stop birdwatch
birdwatch stop/waiting
# start birdwatch
birdwatch start/running, process 30453

Also, the services start automatically after a system reboot.

So what needs to be done? First of all, we need a standalone instance of the applications using the dist command, e.g.;

# play dist
[info] Loading project definition from /home/bw/BirdWatch/project
[info] Set current project to BirdWatch (in build file:/home/bw/BirdWatch/)
[info] Wrote /home/bw/BirdWatch/target/scala-2.10/birdwatch_2.10-0.3.0.pom
[info] Your package is ready in /home/bw/BirdWatch/target/universal/

We can unpack the zip file where we choose and then simply adapt the script from the blog post mentioned above, in this case:

description "Upstart script for, modified from"

env USER=bw
env GROUP=www
env APP_HOME=/home/bw/apps/birdwatch-0.3.0
env APP_NAME=birdwatch
env PORT=9000
env EXTRA=""

start on (filesystem and net-device-up IFACE=lo)
stop on runlevel [!2345]

respawn limit 30 10
umask 022
expect daemon

pre-start script
    #If improper shutdown and the PID file is left on disk delete it so we can start again

    if [ -f $APP_HOME/RUNNING_PID ] && ! ps -p `cat $APP_HOME/RUNNING_PID` > /dev/null ; then
        rm $HOME/RUNNING_PID ;
end script

exec start-stop-daemon --pidfile ${APP_HOME}/RUNNING_PID --chdir ${APP_HOME} --chuid $USER:$GROUP --exec ${APP_HOME}/bin/$APP_NAME --background --start -- -Dhttp.port=$PORT -Dhttp.address=$BIND_ADDRESS $EXTRA

Et voilĂ  - after a restart of the server, all services come up as expected. Much nicer. The script is also on GitHub. For more information, check out the upstart cookbok and the getting started guide.


Last week was fairly productive, I got some really cool stuff done that had been on my mind for a while. I hope to continue this flow in the week that has just started. I’ll let you know next week. Cliffhanger: I recently increased the Google PageSpeed Insights score of this blog by a lot, from 58/100 to 83/100 for mobile and from 77/100 to 90/100 for desktop.

It also feels like the pages are loading a lot faster. Next week I’ll let you know what I did. Would you like to get notified when the next article is out? Just sign up for the mailing list.

Have a great week, Matthias

« Weekly Update: git vs brain fart, Octopress, Sony A7, my audio setup Weekly Update: Pomodoro, all-Clojure BirdWatch, Income »