Distributed Systems: Getting Started Resources

The very first software project I was ever involved with required us to write a financial portfolio management system for a London based company. After the system had been running for a few months, we were asked to make it available to their other branches across the globe too. As a web application this was easy enough, or so we thought. First challenge: system should be fast wherever you are accessing it from. Our journey with distributed computers was about to begin.

The landscape has changed dramatically since then. Cloud services like AWS and Azure have now made it possible for any small company to build systems that are distributed across many machines and even across the world. Furthermore, users are now expecting that their software should always be available, reliable and fast.

That’s the journey I find myself on to this day. Here are some resources that have helped along the way

Distributed Systems for Fun and Profit

What is a distributed system in the first place? This guide lays the ground work I find myself going back to over and over again to understand some of the core principles and problems that one is faced with when dealing with distributed systems. Read More…

A Note on Distributed Systems

A paper written over 20 years ago, but is still valuable for today. Trying to make the developers job easy, by abstracting away the underlying issues is not a good solution. Don’t drink the latest Koolaid. Dig a bit deeper and find out what we are actually trying to solve. Read more…

Designing Data-Intensive Applications

Data Intensive ApplicationsMartin Kleppmann has written a few great articles /talks (Turning Database inside out, Streams). And his book is no exception, even though it’s still in pre-release.

The book explores “what we’re actually trying to achieve: reliability, scalability and maintainability — how we need to think about them, and how we can achieve them.” Read more…

Dist-Systems: Slack Channel

Distributed System Slack ChannelA slack channel (created by Henry Robinson) for people who work on this stuff every day. I have found people on this channel extremely helpful. I have also learnt a lot, by just eaves dropping on the conversation that happen here.
Get your invite here: https://dist-sys-slack.herokuapp.com/

Academic Papers to read

Academic papersIn my past developer life, I never understood why one should read academic papers. They are outdated, difficult to read, and only work in “theory”. Well, turns out I was wrong. As an industry we tend to ignore things we have learnt a few decades ago. This blog post highlights some papers you should read. These papers lay the groundwork and discuss the challenges distributed systems need to overcome. Read more…

Show Your Work

Show Your Work
Rating: 9/10

Show you work is a book that has made a big impact on me. Austin Kleon advocates sharing your work, your process and your ideas with the world and lays out a blue print on how to do it. I have certainly been inspired to share more of my work after reading this book (this blog is a direct result of it). I hope it inspires you to share your work too.

Here are some of my notes on the book.

You don’t have to be a genius
Geniuses don’t work in isolation and good work is not created in isolation. Instead, we geniuses are constantly inspired by other people’s work that drives us to hone our own craft
Be an amateur, because in the beginner’s mind, there are many possibilities. In the expert’s mind there are few. They are just regular people who get obsessed by something and spend a ton of time thinking out loud about it.
The best way to get started on the path to sharing your work is to think about what you want to learn, and make a commitment to learning it in front of others.
You can’t find your voice if you don’t use it: If your work isn’t online in this day and age, it does not exist.
Read obituaries: take inspiration from those who have walked this earth before us.

Think Process, Not Product
Take people behind the scenes: human beings are interested in other human beings, how they work, how they solve problems. Show them this world.
Become a documentarian of what you do (start a work journal, photograph album, scrapbook and share insights).

Share Something Small Everyday
Send out a daily dispatch (could be something very small tweet, photo, blog post etc.)
The “So What?” Test: does the work matter? Ask the So What question after reading the tweet or blog post.
Turn your flow into stock. Collect small pebbles of your work everyday, and by the end you can build a wall, house, play park or however you want to arrange them with it.
Build a good domain name. You need to own your share on the internet (and not on facebook, twitter etc.) Your own name. Don’t think of it as a self-promotion machine, think of it as a self-invention machine.

Open Up Your Cabinet of Curiosities
Don’t be a hoarder: Your influences are all worth sharing, because they show people who you really are.
No guilty pleasures. When you find things you genuinely enjoy, don’t let anyone else make you feel about it.
Credit is always due. Attribution is key on the internet.

Tell good Stories
You need to know what a good story is and how to tell one.
Structure is everything. You stories will get better the more you tell them.
Talk about yourself at parties. Have an answer to the question: So What do you do? at parties. For example: “I am a writer, who draws.” Don’t ever use Ninja, rockstar, guru in your bio. Ever.

Teach what you Know
Share your trade secrets: the minute you learn something, turn around and teach it to others. Share your reading list. Point to helpful reference materials. teaching people doesn’t subtract value from what you do, it actually adds to it.

Don’t Turn into Human Spam
Listen, Be thoughtful. Don’t send out spam. People see straight through your selling techniques.
You want hearts not eyballs: stop worrying how many people follow you online. have a few followers that are really committed to you and your cause. Make stuff you love and talk about it and you’ll attract people who love that kind of stuff too.
Vampire Test. Don’t stay around people who suck the life blood out of you and drain your energy.
Identify your fellow knuckleballers. Meet with people who share a common passion as you do. Meet at meetup.

Learn to take a Punch
Let em take their best shot. As soon as you put your work out there, there will be critics. Roll with the punches. Don’t feed the trolls.

Sell Out
Even the renessaince had to be funded. People need to eat and pay the rent. Sell the work you are proud of and be proud of it.
Pass around the hat.
Keep a mailing list. Make more work for yourself: try new things, be ambitious.
Pay it forward: You just have to be as generous as you can, but selfish enough to get your work done.

Stick Around
Don’t quit your show: every career is full of ups and downs. Just keep going.
Chain smoke: Artists who’ve managed to achieve lifelong careers: they all have been able to persevere, regardless of success or failure.
Go Away, so you can come back: You need rest periods too in order to recover and put your best work forward.
Begin Again. Look for something new to learn, and when you find it, dedicate yourself to learning in the open.

3, 2, 1 … Shipped

This post is part of the 30 day app challenge. An overview of the journey and progress can be found here.

30 Day Challenge Status
Download Treasure Hunt Challenge on App Store

30 days ago, I challenged myself to write an app using the Ionic Framework in 30 days. I had lots of doubts at the beginning of this process, whether I would have enough time, what app I would write, etc etc.
Well, 30 days have now passed. Coding time is up. The app has been sent to Apple to review. I did finish writing an app. And the two users who I wrote this app for, are very busy using the app already (my two sons) Here are some screenshots and lessons learnt.

App Overview

Allow users to create treasure hunts, by taking pictures and recording riddles which will reveal where the next clue is to be found.


What went well

Test the app with actual users as early as possible
Whenever I was unclear if I should add a button here, or should I add a listbox there, I would ask whoever was closest to me, to test the app (hallway usability testing). I had to suppress my need to show them how it works, and rather observe how they discovered functionality. Exclamations like the following were very common:”Oh that’s how they interpret that!”, “Whoops. I forgot about that bug.” “Tapping vs clicking seems to be different”

My sons loved testing the app. They are included in the credits because of this very reason. The main thing they loved was recording “You are a poo poo Pirrate, Arrr!” and playing that over and over again, laughing hysterically. Come to think of it, I should have just created a sound recording app! They do occasionally use it to hunt for treasures too though.

Having fun and keeping it simple
Creating a new app was a whole lot of fun. This whole experience reminded me of creating my first webpage on geocities: learning, coding, playing, testing. A whole new playground! I did not have many technical mishaps along the way. I do think this is because I kept the feature set simple and achievable.

What didn’t go so well

I am a perfectionist. There are a thousand things I would still like to add to the app, designs I would like to polish off, screens I want to change. But shipping in 30 days is more important than this. And I know my perfection hunger is never satisfied. It’s good to have a deadline and constraints (time, time, time).

It took me a while to settle in on a design that I felt comfortable with. I don’t love the design, but it is good enough for this challenge. If I were to sell this app, I would invest more money into the design aspect of the app and get a professional designer. I love the concept behind 99designsand would make use of their service.

Major feature left out
I had to make a tough call early on in the project. The geo location plugin wasn’t working to the accuracy that I needed it to. I really wanted to have a compass that showed the way to the next treasure and how far away you were. I got 90% of the way there. Unfortunately, I could not get it to work accurately enough for clues that were close to one another, and for in-house treasure hunts. I did not have much time, so I decided to leave it out entirely and get the rest of the features working. Perhaps in version 2 (if there is ever going to be one), this will become part of it.

Resources Used

Ionic Framework
The ionic team have done a great job. They have taken away a lot of the pain you have to deal with when building mobile, hybrid apps. This has helped me tremendously in order to ship something in 30 days. It’s more than just an open source framework.

Most of the assets I used for the app came from here. Great marketplace for music, sound effects, graphics, stock photos etc. At great prices too!

Whenever I browse the library here, I am inspired. I love what photographers and artists are doing at iStockPhoto. You do pay a hefty price for many of the items. But who can put a price on beauty, right?


I really enjoyed writing this app. It’s great to see it being used already by my own family. If you have been wanting to write an app for a while read on…

Getting Lost Adventures

Being Lost

My son’s snapshot of us being lost

One of the things that we as a family love to do while on holiday is to get lost. We are often forced out of bed at a far too early time by our two boys, who are both eager for a new adventure. After quickly packing a picnic, we will choose to walk down a random street (asking our 4 or 2 year old where to go helps getting lost quickly too). We try and spot a few locals and ask them which coffee shop or bakery is their favourite. At the coffee shop, old men and women, who are always eager to talk and the only ones with enough time, will tell us stories of what this place used to look like; and where we should go next. It’s fun and one of the best ways to discover a new city, or indeed a place you have been to many times before.

Programming Adventures

I have tried to adopt this same philosophy in my programming journey recently. I have been lost in NodeJS caves and took a ride on the ruby Rails. I speak to the locals in these communities and ask them for advice: “How would they tackle this problem?” “What do they recommend?”. I am always surprised how helpful everyone is in every community.

I know the alleys and nuget packages of the .NET space quite well, but joining a .net open source community allowed me to talk to people from different countries and different experiences all together.

A beginner’s mindset

I feel like a complete beginner when I enter these foreign lands. It forces me to adopt a beginner’s mindset. It forces me to be lost for a while; and that’s a great place to be sometimes.

be your name Buxbaum or Bixby or Bray
or Mordecai Ali Van Allen O’Shea,
you’re off to Great Places!
Today is your day!
Your mountain is waiting.
So…get on your way!
– Dr Seuss, Oh the places you’ll go

Gulp-Inject and Ionic

This post is part of the 30 day app challenge: I am building a mobile treasure hunt app using the Ionic Framework.

As your app starts to grow, you will be creating more controllers, services and stylesheets for your app. Every time you create a new file, you have to remember to manually insert this link into your index.html file. I keep forgetting about this last step and often wonder why my new service is not working. If you are in the same boat, read on. Gulp-inject allows us to automate this last step.

Gulp Inject: What it does

Whenever you create a new file or change a file, gulp watch will pick up on this change and call the gulp-inject task.

Gulp Inject Ionic

Gulp Inject Ionic

Get it setup for Ionic

This is how to get it setup for your ionic project.

1) npm install -g gulp
This installs gulp globally, which means you can use gulp with all your projects.

2) npm install —-save-dev gulp-inject
This adds gulp-inject as a dev dependency to your project. Dev-dependendies won’t be installed automatically when you run npm install, because it is not needed to run the project. However, anyone who wants to use the project for development needs to have dev-dependencies installed.

3) Add a new task in your gulp file: index
Next we are going to add a task to the gulp file, that will instruct gulp to inject any js and css files into the index.html file

In the gulp.js file add the following task:

var inject = require('gulp-inject');

gulp.task('default', ['sass', 'index']);

gulp.task('index', function(){
return gulp.src('./www/index.html')
.pipe(inject(gulp.src('./www/components/**/*.js', {read: false}), {relative: true}))
.pipe(inject(gulp.src('./www/components/**/*.css', {read: false}). {relative: true}))

I also want to make sure that whenever a new file is added or changed this task gets run.

var paths = {
sass: ['./scss/**/*.scss'],
components: ['./www/components/**/*.js', './www/components/**/*.css']

gulp.task('watch', function() {
gulp.watch(paths.sass, ['sass']);
gulp.watch(paths.components, ['index']);

5) Add gulp watch task to Ionic Project

"name": "treasureHunt",
"app_id": "",
"gulpStartupTasks": ["sass", "index", "watch"]

This means that whenever you start up ionic on the command line using, ionic serve, the above gulp tasks will be run.

6) Add html comment to inject js and css files
Add the following lines into your index.html file to allow gulp-inject to insert the new links:

<!-- inject:js -->
<!-- endinject -->
<!-- inject:css -->
<!-- endinject -->

If you are looking for a different setup or want to read up more on how gulp inject works, the gulp-inject documentation covers a bit. To see what this setup looks like within an ionic app,

This setup took me about 10 minutes but is already paying dividends. It’s one less thing to worry about in the development cycle.

Time, time, time

This post is part of the 30 day app challenge: I am building a mobile treasure hunt app using the Ionic Framework.

“I am just so busy at work at the moment.” “I am too tired for this”. “I need to spend time with my family.” “I need to contribute to other Open Source projects” “So many things to do, not enough time left”. “If anything is worth doing, it’s worth doing it right and I can’t do it now.”

All true and all thoughts that have gone through my head in the last two weeks. There is so much I want to throw into this app, so many design ideas and features I have in mind. So what do I do?

Here are my options:

A) Throw more money at it
With services like odesk, freelancer.com I could easily outsource this problem. Freelancers are amazing now and will get it done quickly for me. I can get all the features delivered in no time. They aren’t that expensive either.

B) Do it all, fast!
I could still complete all the features I had in mind. The screenshots would look great in the app store. All the features I had in mind would be showcased. The only sacrifice I would have to make is that the product would be useable only some of the time. Bugs are not a priority.

C) Screw the schedule
Who said 30 days? 30 days? Nothing can be done in that time. I will secretly extend this time to 40 60 days. That ought to fix it.

D) Just give up
I could just throw in the towel at this point. Enough. I have learnt what I needed to. Next project.

The answer: KISS

What’s the right answer here? Every option is very appealing. And I have probably thought of using each one of them at various points.

But for me the actual answer is answer E. Yes the one that’s not listed above: It’s KISS.

Keep it Simple. really, really, really Simple.

For me this means throwing out all the super cool features I had in mind, and making the app only do the most essential things. Anything that is not serving this purpose or that is a nice-to-have, is not part of this 30 day challenge.

What’s was the actual goal with this challenge?

For me, it is to learn more about Ionic and to have fun while learning it. I also wanted to see what I can deliver within these extreme time constraints in 30 days.

When daily life keeps me busy and my own high demands and wanting to impress thoughts keep knocking on my door, it’s easy to lose sight of what I set out to do. Reminding myself about my initial goals: fun and learning, puts this whole challenge into perspective again.

Rabbit Holes: Getting Unstuck

I spent two hours last night figuring out why a plugin for the treasure hunt app kept timing out. It worked in the emulator, on the web, but not on my phone! I scoured stack overflow, forums, documentation, trying numerous variations. I finally packed away my laptop in frustration and tried to get some sleep.

Today, I had to face the question, should I head down that rabbit hole again and dig even deeper? That’s what I, as a software developer, am destined to do: digging deeper to solve these difficult problems.

Pair programming with imaginary friends

In the past, this is when pair programming has helped me the most. I would be head down deep in a rabbit hole and someone on my team would join me.

P: “Where you at?”
Me: “I have been trying to solve this now for THREE hours. Well you see this variable here, I cannot figure out why it is not getting a new value from our web service. Our codebase is so fragile! We should be doing this, that and the other. But there aren’t enough tests and [blah blah blah]. If only we had [shiny new framework] I would not have this problem!”
P: “What problem are you trying to solve?”
Me:”Ok […heading out the rabbit hole…]. I am developing this [ new feature description ]. When the user clicks on the button, this method is called and then this method will …. hmmm… OH that’s where the problem is! I have not done [stupid mistake here]!”.

Ever had this happen to you? Even though I did not have a pair programming partner join me, I talked it through with my imaginary friend (try a rubber duck, if that feels too weird 🙂 ).

Solving the wrong problem

rabbitwarrenIn my case, it turns out, I was actually solving the WRONG problem and did not need that plugin in the first place. Last night, I was so deep in the rabbit hole and kept on digging deeper and deeper. I am glad I forced myself to take a break (mainly due to lack of sleep!) and backtrack out the rabbit hole again.

So, what problem are you trying to solve today*?

* Hat tip to Kevin Trethewey, who would always ask me: “What problem are we trying to solve?”. I have found that it’s one of the most powerful questions we can ask ourselves.

Crappy First Drafts

This is the third post in the 30 Day App Challenge series

The best advice I have ever received about writing was to “Write Crappy First Drafts”. I believe it is relevant for the app I am about to write too. See I am a perfectionist by nature and can easily get stuck in the details. I can rewrite a single sentence about ten times and still not be happy with the result. The major problem with this kind of perfectionism is that I can easily get stuck in an endless circle, obsessing about the details, and not focusing on the big picture.

In order to combat this in writing, I write really crappy first drafts. They are terrible. I keep that voice in my mind quiet, that keeps telling me: “I am sure we can express that in a better way”, “That doesn’t make any sense”. If I am not embarrassed about my first drafts, I have not done a good enough job to keep that voice quiet. Writing this way helps get all the words that you possibly want on paper. Getting to a flow-state is much easier that way. The editing can always happen at a later stage.

I have applied the same principle here to the app: writing crappy first screens. My goal is to get to an overall picture quickly (like writing a chapter outline for a book) and then obsess about the small details later on in the process.

What does this process look like?

1) Code up each screen in HTML

From each sketch that I drew in the previous step, I create a very quick HTML version of it. This shouldn’t take more than 3min per page.

For example for the Clue Editing page:

The Sketch



The Result

Clue Edit Page


HTML version

<div class="bar bar-header">
<a class="button button-clear button-positive" href="#/edit”>Cancel</a>
<h1 class="title”>Edit Clue</h1>
<a class="button button-clear button-positive" href="#/edit">Done</a>
<ion-content class="has-header">
<h2>Clue Details</h2>
<li>add Picture</li>
<li>add Riddle</li>
<li>add GPS location</li>


2) Place each HTML into components folder

Component Folder Structure
I create a new folder called components under www and for each HTML page I now have, I create a feature folder for it. i.e. For the clue page I create a clue folder.

Later on, this becomes easier to manage the application as a whole, because the controllers, any css, any directives will also go into this feature folder. Google has released how to best lay out your angularjs app and the same principles apply to a mobile app too.

3) Add Routing

Next, I add each screen as a state to the following file: app.js. Ionic uses ui-router to know which screen to show for each url or state. In past projects, I have spent a few hours debugging to find out how I messed up this router, so it’s good to get this out of the way early on in the project. After this step is done, all your screens should be accessible, right from the start.

For the treasure hunt app, this is what the routing looks like:

app.config(function($stateProvider, $urlRouterProvider) {

.state('welcome', {
url: "/welcome",
templateUrl: "components/welcome/welcome.html"
.state('list', {
url: '/',
templateUrl: 'components/list/list.html'
.state('details', {
url: '/details',
templateUrl: 'components/details/details.html'
.state('edit', {
url: '/edit',
templateUrl: 'components/edit/edit.html'
.state('play', {
url: '/play',
templateUrl: 'components/play/play.html'
.state('finish', {
url: '/finish',
templateUrl: 'components/finish/finish.html'
.state('clue', {
url: '/clue',
templateUrl: 'components/clue/clue.html'




Once this is all done, I have an app that I am too embarrassed to show anyone, but I have a very good starting point and know what my screen flow looks like.

Next Task

I can’t stand these screens for very long, so the next task will be to make it a little less embarrassing by making use of Ionic’s CSS components.

Setup Dev Environment for Ionic

TL;DR: use the guide here to get Ionic up and running on your machine

Setting up your computer to develop for mobile apps is a bit of a process. There are so many dependencies that need to be installed. That’s why I really appreciate that the ionic team has made and is continuing to make the on boarding process easy.

Here are a couple of options to get you on your way.

Manual Installation

Follow the guidelines how to install it on your machine on the ionic website. I used these guidelines both on a windows machine, using Visual Studio and Telerik App Builder, and on my Mac.

When developing on my Mac, I tend to just use the exact guidelines as posted by the ionic team.

If you are using Visual Studio and Telerik App Builder, download one of the starter templates from the github repo, create a new telerik app builder project, add the folder to the project and you should be good to go. Telerik App Builder.

If you are having troubles getting started, it’s a good idea to look on the ionic forum, post a question. I have found the ionic community to be very helpful!


If you don’t want to install all the dependencies that ionic needs, you can make use of ionic-box. In summary, ionic-box contains everything you need to develop in a vagrant box. Once you have vagrant and virtual box installed all you need to do is boot up vagrant (vagrant up from the command line) and you are good to go.
The source and instructions on how to get started can be found on the github project page.

Next Task: writing some HTML for the first few screens.

App Idea and Sketches

This is the first post of the 30 Day App Challenge series.

What will I be building?

Treasure Hunt App
treasuremapI have two young boys and they LOVE to play treasure hunt and they LOVE to play on my phone. So why not combine the two together? Using this app I want to easily be able to create a treasure hunt for them. In order to get to each checkpoint on the treasure hunt map they will get some clues; a picture, riddle or GPS hot and cold compass. The app would provide visual clues to say how close they are to the checkpoint. Once they have completed all the checkpoints, they will receive the treasure. … Bribing them to do their chores will be so easy now.








What I learnt

Choosing an idea
There are always a ton of app ideas out there. How do you choose an idea? Seth Godin said that we should treat it like choosing lunch. We do it every day because we have to. We do not delay the decision, because otherwise we will go hungry. I like the fact that this 30-day challenge is forcing me to choose an idea… any idea and then to go ahead with it.

Sketch out ALL screens
I find it easiest to use a tool like balsamiq to flesh out what each individual screen should look like. I do like the fact that right from the get go I am forced to think about what goes on each screen and not to overcomplicate the whole thing. It’s also a great way to get instant feedback. For example, I showed these screens to my wife and she was confused about a few items. After a few more iterations these confusions got ironed out. Because it is in Balsamiq this is quick to do and to change. This exercise is also very helpful when building an app for someone else. They might have a general idea about what to build, but putting it down on paper/mockup tool, in detail screen by screen, always reveals questions that have yet to be answered.

Constraints is a good thing
Having less time and not a lot of screen estate to put lots of different buttons on can be a good thing. It forced me to think about the most essential features. I am nervous that this app is too big for 30 days, but let’s see.

Next Task:
1) Get my dev environment setup for Ionic