Author: 1

no longer 100%

interdubs

Until yesterday we had a perfect uptime history. 1,375 days online, and no interruptions. Today that changed: Starting from 9:16am PDT INTERDUBS.com was not reachable for 19 minutes.

Absolutely our fault. Of all possible scenarios it is actually our preferred one since we can fix it. We did and this problem will never occur again. Still sucks to having lost our outage virginity. All outages are avoidable, and so was this one. And -as usual- it was the lack of imagination that caused us to not see this coming.

What happened?

An upload with 550Mb/s triggered an automated protection system that took a network interface offline. It should have only impacted the one address using that amount of traffic. But it was doing it’s job wrong and shot in both directions: Rendering us unreachable.

This system is meant to guard INTERDUBS against malicious brute force attacks. Not a bad idea, if implemented right. 550Mb/s is of course still very very far away from the limit of our network capabilities. It was the volume and specific traffic pattern that caused the emergency shut off. We did neither envision nor test this specific condition. No two ways around this: Our fault. Embarrassing. Please accept our apologies (and money, if you like – see below for details).

We re-configured the system and are confident todays outage will never happen again.

Since we were unaware of the bug in the system configuration it took us a little while to identify the cause. Since the system is responsible for security we also had to spend a couple minutes testing the changes that then became the fix. 19 Minutes is a long time for an outage. Looking at what needed to get done to bring us back online we feel that we did OK. Not great, but OK. Of course there is also room for improvement, and we started to implement those changes today.

INTERDUBS overall uptime dropped now from 100% to 99.99904%.
For the month we are down to 99.95%. Well below the 99.999% we promise. None of our clients has to pay for INTERDUBS this month. If they choose so. A simple email is enough: we will discount the whole month.

Since counting nines of uptime is not really what most people want to think about, we decided that effective immediately we change our policy: we now guarantee 100% uptime. No longer ‘only’ 99.999%. If a client feels that INTERDUBS wasn’t there for them when they needed it, then they don’t pay. Simple as that.

This page documents all outages and service disruptions we ever experienced

paging don norman

internet

Allot of work has probably gone into the creation of this comparison chart of Javascript UI libraries. Which is great and appreciated.

Tragically the author didn’t specify the meaning of the X axis. With benchmarks both can be the case. (Here it is apparently shorter is better). The comments on the post do point this out. Oddly the author did not fix the page. Would have taken 1 minute.

It might also be that not having a meaning on the axis’ caters to the biggest audience: People looking for this page might already have a favorite. So, let’s say you embrace YUI (shivers) then you come to this page, look at the long bars and click away just having confirmed that YUI is best. (Firmly ignoring what should have been a give away that protoype does even have longer bars).

People probably roam the Internet looking for ‘information’ that confirms what they were thinking all along. Since the Internet is pretty vast this works better than it ever did in history. There are 22,000 pages about unicorns and postage stamps.

Wiedervereinigung

history

There will always be West- and East-Germany. I was convinced of that. All political parties had ‘a unified Germany’ listed on their official agendas. But nobody took that serious.

It was nice to be wrong about this. It was nice to see something change 20 years ago.

The thought that many people were born the GDR and died before their countries ill fated system is a weird one: All these people knew was this version of the world.

hft

free of any reason internet misc

A new cable gets dropped into the atlantic to save 5ms on a 60ms delay. And High Frequency Trading will pay for that. You know that it really has taken off when they start considering a straight tunnel between London and New York. As impossible as it is, it WOULD save at least another 15-20 ms.

Is Wiredrive using Isilon news?

internet marketing technology

Wiredrive is a system that operates in a similar space as INTERDUBS. Naturally their press releases get my attention.

The last one announces that
Wiredrive recently dumped its open source, clustered storage system in favor of Isilon.

I don’t think that it would be appropriate to go into detail here why INTERDUBS uses a different storage solution. Or why I think that such details do not matter for the clients, as long their data is 100% protected. Isilon works, I would not exactly title it meaner and leaner myself, but people can feel about what they do in any way they want and express it accordingly. Would not be worth the blog entry.

The question that is worth being raised is how Wiredrive using Isilon is newsworhty at all. This Wiredrive document outlines how Isilon is in use. While it itself is not dated it references the 2006 Olympics and 250GB hard drives.

fun spam

daily life

“you look great in your profile picture” the mail said. When I realized that I actually checked the link (not by clicking it obviously) I had to laugh about myself. Never thought I’d derive some fun out of spam and the way I happen to look.

what if

free of any reason

Currently facebook is down. Funny since Mr Zuckerberg just eclipsed Steve Jobs in terms of wealth. While it is gone I find the following thought amusing: What if this is it. They just close it, turn it off,
erase the hard drives and be done. That would be a pretty awesome exercise. The users certainly could not do anything about it. I don’t think they would have any rights on ‘their content or contacts’.

And imagine how quickly the economy would recover if more people would stark working again while being at work.

Facebook came eventually back. Facebook engineering posted what had happened. Reading the comments on that entry is quiet depressing. 99.9% have no clue what it takes to run a system that can deal with 80Gb/s of their ramblings. That seemingly does not stop hundreds of them them to still come up with comments on how to ‘improve’ things.

picture of the Internet

internet

one picture

reels looking nice

interdubs

This is Scott Salinas Music reel. It is an awesome reel. Scott used INTERDUBS for it. His company Critical Mass switched to INTERDUBS just a couple of weeks ago.

We are very happy to see how quickly they are able to make custom looks with our system. This one here, he made to match his personal site..

They save a bundle too we hear.

datacenter fumes

technology

Bringing decommissioned hardware from the data center in the office in order to break it down. Turning the machines on it is amazing what kind of stench gets emitted from the machines. Not the disaster forbearing smell of burnt wire. Just that pure ‘cheap electronic factory’ odor.