Behind the scenes @ Coolblue HQ

What is Coolblue?

Last week I was invited to go visit Coolblue behind the scenes in Rotterdam, NL. Coolblue has always been one of my favorite online dutch/belgian webshops because of their great customer service, their clear but simple websites, and most important, their great range of products at very affordable prices.

They roughly have about 50k unique products in 150-200 different webshops (they launch 2 new webshops per week on average), which they all have in stock at all time in one of their huge warehouses. Their biggest warehouse is more than 40.000 square meters which is about 10-20 FIFA football fields! How big is that! Ordering before 23.59 means it’s delivered at your doorstep for free the next day, or the same day when paying additional shipping costs. Also on sunday!

Why would they organise behind the scenes events?

The main idea behind this ‘Behind the scenes’ is that they are looking for talent programmers. They’re looking for roughly 100 new developers. They have about 40 right now so it seems they have big plans. Are they going international? Who knows! But their CEO didn’t really deny it. Their global employee basis is about 750 people.

Since I’m a freelancer I won’t be able to work for them so why did I go to this event? Well.. because their main dev talk was about scalability. I find this a very interesting and challenging topic so getting insights from one of the biggest webshops in the Benelux was a great opportunity for me to learn.

So what did I learn at Coolblue?

  1. use microservices / hypermedia api’s to flatten out your infrastructure. This allows you to easily isolate bugs and/or maintain those API’s without having to retest stable services on every deploy.
  2. every microservice uses an isolated datastore. CouchDB for customer data, PostgreSQL for order+payment information, ElasticSearch for the product catalog.
  3. RabbitMQ as a central state change notification mechanism. This is the glue between your microservices.
  4. use git pull requests to review & validate your team members changes
  5. CentOS RPM packages to distribute everything
  6. Puppet labs or Chef to install & maintain your (virtual) servers
  7. Statsd + Graphite for advanced reporting
  8. Nagios for alerting
  9. Create a “Chaos Monkey”. It’s single purpose is to once in a while kill your live environment. Your devs should then come with solutions to auto-fix these downtimes. This is how Netflix stayed online during the massive AWS outage a while ago: http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html

Have a look at the slides from their software architect.

Why should you attend one of these meetings?

  1. If you want a developer job @ Coolblue obviously.
  2. If you want to learn more about scalability.
  3. If you want to network with hundreds of other devs.

Inform yourself on this page: http://www.coolblue.nl/behindthescenes

Here’s a nice dutch article about the same event: http://www.dailybits.be/item/coolblue-behind-scenes/

QNAP 412 Turbo Nas – no more disk space

I have a QNAP 412 Turbo Nas and ran out of disk space. No problem you say?
Well.. apparently QNAP’s are very fragily when it comes to 100% disk space usage.

Symptoms could be:

  • • console: ls freezes, rm’s freeze
  • • winscp/ftp/web admin: cannot delete files (times out)
  • • reboot: NAS becomes unavailable, doesn’t show up in your network anymore, QFinder can’t find your nas
  • • even my Windows 7 box became unstable because I mapped a few network drives to my shares

15-10-2013 10-09-11

I’ve tried upgrading my firmware (remove your disks first! use 1 old hdd) but this doesn’t help. As soon as you insert your old drives, it returns to slowmotion mode.. (aka. timeouts).

I then contacted QNAP via their support forum and here’s their solution:

  1. Turn off your QNAP NAS
  2. Pull out all HDD’s
  3. Restart QNAP without the HDD’s
  4. Use QFinder to detect the IP
  5. Put back the HDD’s
  6. Don’t follow the installation wizard (web)! Instead, connect via SSH (use putty?), login credentials will probably be admin/admin
  7. Execute these commands (mount the raid):

 

mdadm -A /dev/md9 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm -A /dev/md0 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3
cd /mnt
(Under /mnt)
mkdir HDA_ROOT
cd /share
(Under /share)
mkdir MD0_DATA
 
mount /dev/md9 /mnt/HDA_ROOT
mount /dev/md0 /share/MD0_DATA -t ext4
(or mount /dev/md0 /share/MD_DATA -t ext3)
  1. Your shares should now be accesible again in /share/MD0_DATA so you can delete some data. QNAP recommends atleast 3GB free space.
  2. Reboot your NAS with this command: ‘reboot’
  3. Your NAS should now be accesible again.

 

So all in all this fixes the problem and I still have all my data. But I do think QNAP should have added an extra fail-safe to prevent this from happening. We’re not all technical people and most people won’t even know what SSH is..

Raspberry Pi PLC/Domotica testcase

Here’s a quick tutorial on how to build a hardware on/off switch which sends this signals to a RESTful web API using Raspberry PI with Raspbian. This is in fact a small PLC testcase (proof of concept). The possibilities are in fact endless!
I’m planning to use this to monitor certain events around my house. E.g. is a door open/closed? Is a device on/off?

Download & install Raspberry

Download the latest version of Raspbian onto your Raspberry PI SD card:
http://www.raspberrypi.org/downloads

Updates & depencies

Do some updates + install extra depencies:

apt-get update
apt-get install python, py-pycurl

Setup the hardware

In order to know the GPIO’s pins you’ll have to find the input/output pins. Here’s a map:

raspberry-gpio
Connect your Raspberry’s GPIO (the big black serial thing) to some switch or toggle. Here’s how I did it (testcase):

raspberry-pi-gpio-switch

We actually need 3 pins. One for I/O, one for power, and one for grounding (safety first!). Make sure you solder the right cable to the right GPIO pin (see map above).

Now you might experience the naming of these pins are confusing. That’s because there’s 3 type’s of naming conventions used here..

Pin Numbers RPi.GPIO Raspberry Pi Name BCM2835
P1_01 1 3V3
P1_02 2 5V0
P1_03 3 SDA0 GPIO0
P1_04 4 DNC
P1_05 5 SCL0 GPIO1
P1_06 6 GND
P1_07 7 GPIO7 GPIO4
P1_08 8 TXD GPIO14
P1_09 9 DNC
P1_10 10 RXD GPIO15
P1_11 11 GPIO0 GPIO17
P1_12 12 GPIO1 GPIO18
P1_13 13 GPIO2 GPIO21
P1_14 14 DNC
P1_15 15 GPIO3 GPIO22
P1_16 16 GPIO4 GPIO23
P1_17 17 DNC
P1_18 18 GPIO5 GPIO24
P1_19 19 SPI_MOSI GPIO10
P1_20 20 DNC
P1_21 21 SPI_MISO GPIO9
P1_22 22 GPIO6 GPIO25
P1_23 23 SPI_SCLK GPIO11
P1_24 24 SPI_CE0_N GPIO8
P1_25 25 DNC
P1_26 26 SPI_CE1_N GPIO7

Anyway, let’s move on and try & catch the GPIO’s input using python.

Read GPIO signals using Python

plc.py (daemon script)

import RPi.GPIO as GPIO
import time
import os
 
buttonPin = 07
GPIO.setmode(GPIO.BOARD)
GPIO.setup(buttonPin,GPIO.IN)
 
while True:
  if (GPIO.input(buttonPin)):
    os.system("sudo python /home/pi/plc_handle.py")
    #print "button called"

plc_handle.py

import time
import RPi.GPIO as GPIO
import datetime
import pycurl, json
 
buttonPin = 07
GPIO.setmode(GPIO.BOARD)
GPIO.setup(buttonPin,GPIO.IN)
 
# reset state
last_state = -1
 
while True:
  input = GPIO.input(buttonPin)
  now = datetime.datetime.now()
 
  # check if value changed
  if (input != last_state) :
    	print "Button state is changed:",input, " @ ",now
	api_url = "webserver.com/api/input.php"
	data = "location_id=1&status=%s" % input
	c = pycurl.Curl()
	c.setopt(pycurl.URL, api_url)
	c.setopt(pycurl.POST, 1)
	c.setopt(pycurl.POSTFIELDS, data)
	c.perform()
 
  # update previous input
  last_state = input
 
  # slight pause to debounce
  time.sleep(1)

You can run this script doing this:

sudo python /home/pi/plc.py

Or add it to /etc/rc.local (so it runs after each reboot)

python /home/pi/plc.py
exit 0

Web API

Here’s a quick (and unsafe) ‘API’ script for receiving the Raspberry signals:
input.php

<?
header('Cache-Control: no-cache, must-revalidate');
header('Expires: Mon, 26 Jul 1997 05:00:00 GMT');
header('Content-type: application/json');
 
$dbh = new PDO('mysql:host=localhost; dbname=database', 'username', 'password');
 
$response = array(
    'status'    => 'nok'
);
 
if(!Empty($_POST['location_id']))
{
    $status = $_POST['status'];
    $location_id = $_POST['location_id'];
 
    // create log
    $sql = "INSERT INTO status_log (location_id, status, created_at, updated_at) VALUES (:location_id, :status, NOW(), NOW())";
    $q = $dbh->prepare($sql);
    $q->execute(array(':location_id' => $location_id,
                      ':status'      => $status));
 
    // update location  
    $sql = "UPDATE location SET status=:status, updated_at=NOW() WHERE id=:location_id";
    $q = $dbh->prepare($sql);
    $q->execute(array(':location_id' => $location_id,
                      ':status'      => $status));
 
    // output
    $response = array(
        'status'    => 'ok'
    ); 
}
 
echo json_encode($response);
?>

 
Now I’m very curious what sort of applications you guys are building with this Raspberry Pi “plc implementation”. Feel free to post them in the comments section.

Usefull links

 

A/B split testing with PHP

What is split testing?

A/B split testing is the art of setting up multiple random variants/tests in a controlled experiment. Example tests are different call to actions, different types/locations of buttons, different images, etc.
Your goal? Finding the best converting test, learn from this, and implement this further in your website, thus getting more & more conversions out of existing visitors. Learn why people click and don’t click.

ab-split-testing

Why should you use split testing?

When using split testing, you’ll learn how your visitors think, what works best on your site, and how you can get the most out of all visitors. Optimising websites takes time, but this time may be cheaper then putting another 1000$ in advertising. First optimise, then do some more advertising.

How can you split test in PHP?

There are several PHP libraries which you can use for split testing, and I tested most of them. Here’s my favourites:

Must we use PHP libraries?

Nope.. There are several other ways to integrate split testing in your projects. Here’s some more:

What should we test? Example tests:

And here’s some more articles to help you get started..

 

Any feedback or tips & tricks regarding split testing are more than welcome in the comment section. Good luck!

Laravel 4 released

Great news! My favorite PHP framework Laravel finally released version 4 after 5 public beta’s and months of waiting. Why am I so happy? Well.. it makes my job as a developer a lot simpler compared to my Zend Framework days.

Here’s why you must try Laravel

  • – Smart framework: write less code (faster + easier)
  • – Works with composer, the PHP plugin tool
  • – Filters, events, rest, built-in auth, … it has it all out of the box
  • – Elequent ORM: the smartest and easiest PHP ORM today!
  • – Unit testing for all components

So what’s new in Laravel 4?

  • – L4 now runs on PHP >= 5.3.7
  • – PSR-0 code standards are used
  • – Composer (packagist.org) instead of Laravel bundles
  • – Lot’s of small but convenient little tweaks and options
  • – Mail is now built in
  • – L4 is built with Unit Testing in mind
  • – Great community!
  • – The command line tool (artisan) is now more powerful

 

Here’s what changed in L4:
https://github.com/laravel/framework

Laravel 4 introduction

Download Laravel @ http://laravel.com/, and make sure to follow them on Twitter: https://twitter.com/laravelphp

Facebook: the numbers (infographic)

fb_infographic

Google Universal Analytics: the next big thing?

Today I came across a video about Universal Analytics, Google’s self-proclaimed next big thing.

But is it really? In short? Yes! Here’s why.

What is the difference between Google Analytics and Google Universal Analytics?

Universal Analytics tracks much more than just visits to a website likes Google Analytics does. Where Google Analytics (based on Urchin) tracks visits, Universal Analytics tracks visitors. This allows us to do cross-device tracking which mean even more detailed statistics!

google-universal-analytics

Offline conversions

With the Measurement Protocol you’re now able to ‘plug in’ extra user data, provided by offline/external appliances such as an RFID chip, apps, sensors, etc. Have a look at the YouTube video in my introduction. As a geek, I absolutely love this. The more data the merrier.

Customisable session timeouts

You can now alter this ourselves. Previously this limit was a hard fixed 30 minutes.

Analytics.js

Ga.js becomes analytics.js. The include code also changes.

universal analytics code

Custom dimensions/metrics

Custom dimensions/metrics allows you to create specific metrics in combination with your CRM’s database. See this as custom variables 2.0.

External cost structure

External input, external output. We’re now able to import external costs structures so we can calculate our conversions and ROI even more accurate. Back in the old days.. we could only import Google Adwords costs.

And much, much more..

Here’s some more useful and interesting links I found about Google Universal Analytics:

Did I forgot to mention something very important about Universal Analytics? Please leave a note in the comments 🙂

 

Popular websites 10 years ago

Here’s some screenshots of popular websites (Linkedin, Facebook, Google, Twitter, eBay, YouTube, Wikipedia, Amazon, Hotmail, Blogger, Apple), but 10 years ago.. Pretty neat how things evolved right? Who can still remember this?

websites