Andreas Wacker

while burning in a new machine with a 3ware 9560 adapter I got twice the following error messages:


kernel: 3w-9xxx: scsi0: WARNING Character ioctl (0x108) timed out, resetting card.
kernel: 3w-9xxx: scsi0: AEN: INFO Cache synchronization completed:unit=0.


Looking at the source it seems that 0×108 is TW_IOCTL_FIRMWARE_PASS_THROUGH

I had these errors only when smartd was configured to probe the drives on the 3ware card. This was under max load
during the burn in of the array. Not sure if there were any other errors etc. It seemed harmless enough, but then again
no errors are better than spurious warning etc. Once I have a couple of days of a clean running machine I probably will
turn smartd on again to see if the connection indeed exists.

A new problem that is looking for solutions: Computers are worth replacing while they are still work fine.

Some solutions

Interesting is the theme that the OS is often the reason to kick it to the curb: It is outdated (OS9), just got slower and slower (windows used to do that, does it still? Luckily I have no idea) or it just filled with malware (That WOULD be a windows feature). The hardware might do some good. I am always surprised how little harddrives have been. “Back in the day”.

people alerted me that they got an email bounce saying:

it turned out that my local mail file that I keep as a backup was bigger than 1000 Megabyte. Seems to be that postfix (or whichever program delivers the mail locally to /var/spool/mail) does not like to write to files that are bigger than that number. Scary the file grew to that size within one year.

Since years I work on a couple of computers via command line. Since they are real unix computers it all works remarkably well. For a specific solution I need to run osacompile. AppleScript needs to get compiled. I did not find a way to distribute it as text. So finally I got a hold of an OS X machine in the internet. More on that part later. osacompile really wants to run the application that it will talk to later. Also rather odd. But, hey, we talk Apple here. A sect in disguise of a technology company. So everything is possible. Or rather impossible. Like adding a development environment. The Box happened to have no Dev Tools installed. Usually that’s maybe a bit timely but overall straight forward. Installing development tools on a unix computer.
With Apple OS X 10.4.11 it turns out that doing so via ssh is not as trivial. You can download the source code. But first you need to create a developer account with ADC. It’s free. It’s annoying. They keep forgetting my password. Once you logged in,
you could download the dmg file to your local machine. I could have done that and waited only a couple of weeks for my DSL to upload the 900+ MB file to the final server I need it on. Downloading the dmg directly did not work. I had to fake a login. Which is easier as it seems. In the browser that is logged in (firefox I assume) you look for a cookie called ADCDownloadAuth. This you copy paste into the following command line:

curl -b "ADCDownloadAuth=SomeVeryLongCookieString" -O \
http://adcdownload.apple.com/Developer_Tools/xcode_2.5_developer_tools/xcode25_8m2558_developerdvd.dmg

At least that’s the valid file of today.
Once you have the file you attach (aka mount) it via:

hdiutil attach xcode25_8m2558_developerdvd.dmg

and navigate into

/Volumes/Xcode Tools/Packages

to then run:

sudo installer -verbose -pkg XcodeTools.mpkg -target /


Don’t run this against XcodeTools.mpkg in /Volumes/Xcode Tools directly. This results in the error message:

2008-01-09 03:47:43.889 installer[2843] IFPkg::_parseOldStyleForLanguage - can't find .info file (XcodeTools)


which does not google very sucessful.

The install seems to work, from what I can tell so far. I have gcc and make. And that’s all I cared for.

Autodesk aka Discreet Flame Flint Inferno applications run under irix or linux. Which is great. Unfortunately it is a long standing practise of those people in montreal to seperate different versions of their application by giving them a different user. Of course that’s just plain wrong and stupid. But if you pay north of 40,000 US$ for a single software seat you stop making reasonable demands. Discreet / AutoDesk does this since more than 14 years, why should they stop?

A couple of simple commands can fix the biggest issues with this. The first one is that each install creates a new user id. The fix is to edit /etc/passwd and give the new user a common id (100 in this example). We assume it was 101 for the new install. Running the following command as root:


find /usr/discreet/NEWLYINSTALLEDVERSION -user 101 -exec chown 100 {} \;


will fix the permissions.

Another annoyance is that they set the umask in the .cshrc of each login. If you run a couple of versions side by side it’s pretty tedious to fix these flags manually. The following does so for all installed versions.

Under Linux you can use sed for this:


cd /usr/discreet
sed -i.bak.umask "s/umask 002/umask 000/" */.cshrc


For Irix you would need to turn to perl:

cd /usr/discreet
perl -i.bak.umask -p -e "s/umask 002/umask 000/" */.cshrc


This will make the umask wide open for the user running flame or one of the other Discreet products. Some people might like that everybody can now delete
and overwrite files. Others don’t.

With a full mailbox mail might die with:

fseek: Invalid argument
panic: temporary file seek
Aborted


of course it’s all spam. That aside it seems that

mutt


can deal with these big mail boxes.

For historical reasons I run a Fedora core 4. It turns out that php5 is configured NOT to use mysql in this default install. Which is stupid, but what can you do?
Well, for starters you could follow: helpful howto.
The php source Version that it uses is 5.1.2. And it turns out that this is needed: Trying the current 5.2.1 resulted in

configure: error: The PCRE extension requires PCRE library version >= 6.6

In other words php 5.2.1 insists on having perl compatible regular expressions with a version number higher than 6.6. We ignore the fact that Perl Regular Expressions probably have not changed in the last ten years. So I am not sure why, -o- why, php 5.2 should now be insisting on a new Version.
Over at Centos there is a how to to get pcre 6.6 installed. Only problem is, that it does not fly on fedora core 4. To new for it I suppose.

So in the end I got php 5.1.2 from here and the beforementioned how to worked like a charm. Here my mirror of that specific php Version.

I am not a sysadmin. Ok, I adminstrate machine, but mostly so that I can go back and write some more horrible code. Coding for unix system is the most fun. I still remember when the Sun spark pizzabox showed up in the adjacent office and it came with a huge box of documentation: “They want you to know all this? Awesome!” On DOS PCs I was used to an environment where equipement makers would not share any knowledge.
After working on Intergraph work stations I then spent years with Irix. Which was nice at the time, since SGI had lots of money and even used some of it wisely. Of course the writing was on the wall. And running a web server on basically free hardware (except for electricity) was intruiging enough to try to deal with Linux. I am still trying to do that. Redhat was what came my way first. It worked ok, but at some point I got sick of rpm dependency stacks. Debian looked good with apt-get. So I built two boxes running that, and they drive me crazy. No chkconfig, ‘just’ use ’sysv-rc-conf’. Once in a while I have to deal with Suse, but new machines I build with Fedora. Yum is pretty much making me happy these days. I simply don’t understand why somebody thought it would be a great idea to rename httpd to apache (or vice versa). And there are lots and lots of these differences. You don’t notice them when you stay with one system. But switching back and forth makes this annoying. Comes with the concept of free and open software I guess. But somebody I would like to have the cake and eat it too.

The quality of software is quiet interesting: the core of things seems to work really well for linux. Not so much ‘core’ as in ‘kernel’ but rather functionalities. The fringes, the configurations, the interface to adminstrate these things is pretty horrible. The babylonic /etc/init.d/ confusion is only one example. Another one would be that sar is by default off after you installed it on debian. You have to go into /etc/default/sysstat and enable it. Trickier to find than it should be.

simple, yet powerful


W: GPG error: http://security.debian.org etch/updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY XXXXXXXXXXXXXXXX
W: GPG error: http://ftp.debian.org etch Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY XXXXXXXXXXXXXXXX
W: You may want to run apt-get update to correct these problems


get’s fixed by

gpg --keyserver wwwkeys.eu.pgp.net --recv-keys XXXXXXXXXXXXXXXX
apt-key add /root/.gnupg/pubring.gpg
apt-get update


as stated here

Substitute the missing key for XXXXXXXXXXXXXXXX

To find which rpm provided the file example you simply enter:


rpm -qf /path/to/example



So far botnets have predominantly infected Windows-based computers, although there have been scattered reports of botnet-related attacks on computers running the Linux and Macintosh operating systems.


That’s the NY Times being clueless about Botnets. Good that they write about it. As it is a problem.
Bad that they write so badly about it. The author seems to like ot cover his bases here. “Scattered reports”? God, there are scattered reports about Ant’s playing doom in mongolia. This is as covering. Not more. The reality is that 100% of all botnets are run on Windows machines. There are still no Viruses for OS X. There are MS Office infections that affect the OS X flavor of the product. But the Operating system has been save.

It’s as binary as that. Don’t get me wrong: Apple sucks in some areas. But their OS has had no real life virus infections. People seem to shy away from such binary truths. Easier to throw in a ’scattered reports’ here and there. Pseudo Balance. It’s actually much more harmful than it seems: It leaves loopholes. It kills the truth: Somebody with an intention could quote now the New York Times that there have been Botnets on Linux and OS X. Which is a lie. Not true. The big question that needs a real answer is, if Vista can join the club of predomiantly safe operating systems or not. Unfortunately journalists will not help in finding this out.

The only real weapon against malware is the truth.
Too bad that the New York Times is too afraid to avoid it.


mkdir /mnt/dvd
mount /dev/dvd /mnt/dvd


The simple things that I do every couple of weeks I keep forgetting.
(added slash, thanks Bernard!)

People see the shiny rack mountable Apple Servers. And then they make the mistake to buy them. Don’t! They are way to expensive. But the worst is, that they are impossible to administrate: It’s pretty much all different. And then when you spent hours and hours to go through the documentation you just have to realize that what you need is simply not there.

I wanted to change the default umask for a user that creates files on the afp server. Too bad. Not possible.

Using mac laptops since years I get to appreciate that the computer is not getting into the way when I want to do things.
Linux is a bit different, you have to google a bit for what you want to do. But 80% of it is pretty damn easy. OS X is just plain obscure. It really is a waste of time. Nobody should be believe that you actually got a unix box when you buy OS X server. What you get is some perverted BSD box that no sane sysadmin would like to deal with it. Phew.

Just try to use a device under OS X (server or not) if you have any doubts.

quick howto

how to turn old PC into something useful.

I like the idea of putting the boot CD back into the tray upside down for keeping it handy, for whenever it should be needed.

Running a postfix server with virtual mailboxes. It’s all pretty nice. Vacation however was sketchy. So say the least. One thing that breaks “Virtual Vacation” (implemented via virtual.pl in /var/spool/vacation, and, yes, googlebot, I am writing this for you) are < > brackets around the recpients email address. Entourage does that for instance. I do not think that there is anything wrong with that. Postfix does not deliver the ‘out of office’ reply in those cases though the mysql table called vacation gets the name of the sender added. Tricky and odd bug. My fix was to add:

if($from =~/< ([^\s]+\@[^\s]+)>/){$from = $1 ;}


right after the line:

if ($from =~/([\w\-.%]+\@[\w.-]+)/){$from = $1;}

which was already in there.

Free and open software is great. Just sometimes it’s the odd things that get you.

I just positively confirmed that you NEED to run a linux kernel with ‘noapic’ like this:

...
kernel /vmlinuz-2.6.17-1.2142_FC4smp ro root=LABEL=/ noapic
...


Omitting noapic resulted in an array that would loose a harddrive for a split second in the very same midplane position after 12 - 14 hours of heavy use. The array state would switch to DEGRADED and it would rebuild for a couple of hours. Never lost any data. Just allot of time in trying to fix hardware that was not broken in the first place.

I remember people swearing and cursing getting bonding to work under linux. Things have changed. Following a decent and simple set of instructions for network bonding under linux (yes this is anchor text for google ) it only takes a few minutes. And even works.

Nice.

Needed to build a new NAS server with safe raid storage. It’s more or less a near line storage solution, so I tried to go for best price per Terrabyte. Just before it dissapears into what will be hopefully years of uninterupted service I snatched it’s keys and took it for a spin on the weekend. So to say. I am still tweaking things, but right now I get just a hunch more than 600 MBytes a second sustained writes xfs.

Which is actually quiet awesome, considering that there is not a single SCSI disk to be found in the case. We paid a very reasonable price for the net 6Terrabytes we got. In theory this machine could record 3 streams of 1920×1080x23.98 10bit dpx frames. For 3 hours.

What do I know about computers? I mean, really. So I build this rather big machine, to read along a couple of million weblogs. Needs storage. Sure. I get a 3ware raid controller. Works like a charm btw. There are more blogs, there is more spam, nothing surprising or new. The machine start to have a load of a solid 95-100%. Well, linux can deal with that, and it can. Today I tune another server, and look at parameters. One of them is the scheduler that is used to do the actual IO. The default for my kernel was anticipatory. I changed that, so that
cat /sys/block/sda/queue/scheduler

reads now

noop anticipatory [deadline] cfq

And, what a surprise, the IO load starts to decline and the CPU is idle for 10-15% again! Of course this makes only sense on a server with lots of IO and database activity. That poor machine had to do stupid things for years.

As I said: what do I know about computers? Academic, yet interesting question would be how many CPU cycles are actually wasted on things like this, how many are needed?

Some days things have the feel of a ‘techno groundhog day’. Once again I set up a computer. Once again it has a considerable sized disk system. It used to be that 30Megabyte (no typo) Winchester disk. Today it’s that 8TB raid. And the problem remains the same: The tools choke on the size. I forget what it was twenty years ago. It was not as easy as it should have been. And that did not change. To cut to chase of the technical knowledge that might be helpful now and will certainly be laughing stock in the future (30MB to big: hahaha):
Getting a 3ware 9550… with 16×500GB drives is a good idea. Fits in one nice case and in a Raid 50 config you end up with 6.3TB usable capacity. Historically it needs to run Fedora Core 4. Which is happy to find the array after the installer has been launched with linux dd and a proper floppy drive (!!) has been inserted with the 9550 drivers. The next mistake one can make (and I sure did) is to let the installer automatically partition the drive it found. Knowing that big disk systems can be trouble to start the OS from I already had seperated a 80GB boot partition in the 3ware bios. The Installer went along, formatted the whole thing and did it’s install. Which take some 6 hours I would guess.
Only problem was, that the poor thing could not boot from what it had made. The automatic partition manager was utterly confusde by the size of drives it found, but didn’t let that stop it from trying and failing hours later anyway.
Manual partition of the 80GB boot drive got me over that part. Having an OS to boot: priceless.
The data partition only started working after using parted and a crucial ‘mklabel gpt’. Only then it would accept the size of the partition correctly. Otherwise it was silently reducing it, and then would fail to mount after a reboot.

Sofar the gory technical details.

The bigger problem is:

Disks have become bigger. Ever since computers are around. Everybody knows this, is exposed to this, and benefits from it. The big question is, how can you write a software that deals with the nuts of bolts of disk systems and not be freaking prepared for that? Of course that 30MB harddrive I dealt with 20 years ago would have been a bit overwelmed to run a partition scheme that would be ready to hold 6 Terrabytes. First question is: Would it be really? Sometime people are scared of wasting 3% but waste the future of something. This side of the equation can be argued with.

There can not be ANY execuse for the way systems fail on bigger hard drive: Numbers roll over, systems report -1600% free space. Shit like this is unacceptable. Tremendously stupid. If you code like that, then you should not code. Period.
Disks will be bigger tomorrow. Deal with it. At least create an error message along the lines of “Can not create partition bigger than 2TB” etc. Fail gracefully. You might have not the money to buy enough disks to test it, but you CAN put in checks for these limits. Nobody will slip in an extra 10% ‘integer boost’ to help your code out. The limits are what they are today. Shame on the authors of the tools for the lack of imagination. If physical harddrives can catch their code only after a few years like they do I am actually surprised that y2k did so little damage …

Maybe someday I will find the time to read this interesting yet long article about the difference of BSD and linux.

Of course 1.50 US$ a GB is ridicolous

But the whole concept of a 1U quad drive cheap-o system seems intruiging: Raid cards are still expensive. They certainly deliver in many cases the best solution. But 3TB (4 x 750) cheap ’scratch space’ for data that can be recreated could certainly exist in a 1U box for a pretty sweet price point. Sacrificing 25% storage and you have save space.

And as long Moors Law keeps deflating disk and system prices it is still the best strategy to buy as little storage as late as possible. To paraphrase Einstein just not to late or to little.

the sudo command

wikipedia’s entry for sudo

Microsoft likes more people to develop games for their consoles. In their press release it sounds like a Windows XP machine is all you will need to develop games for their consoles.

The range of impact goes from ‘flash in the pan’ to ‘Sony is finished’. It all depends on the details of the implementation and capabilities. Nobody has ever opened game consoles to a wider development community. It might or might not take off. Trying it is a bold and innovative move.

Microsoft is a funny companies these days: Some of their divisions do all the right things, while others are as stupid as the Ottoman empire in 1907.

Trolltech makes a phone now. Trolltech got big with a toolkit for graphical user interfaces called “Qt”. I used it years ago, and it is not bad. Now they make a phone that runs embedded linux, and their user interface on top of it. In other words it is an open source phone.

From the pure aspect of technology these developments had to happen. The very interesting question is, what will come out of it. Content is a very tricky thing to predict. Hollywood survived despite constant failures in this area. As long the movie industry existed they tried to mechanize and control creativity and content creation, so that they can churn out products like a nuts and bolts manufacturer. And it never worked.

One the other side of the argument one could see Microsoft and Trolltech shipping typewriters to a million monkeys.

And, of course reality will fall somewhere in between. And once the revolution happened, it will be so clear why it did. Same in the other outcome.

Games could really use some injection of innovation. Roaming the show floor of what was the last E3 of it’s kind I was pretty surprised how alike most games looked. I don’t play. But I care about the technology and business side of this industry. There are racing games and first person shooters. Lot’s of those.
With production costs high new content development is tricky. That’s why I liked Rockstar’s Table Tennis.

Tetris was written by a russian programer when there was still a country called “Soviet Union”.

The situation with phones is similar. They don’t suck, but I never saw a phone that made just sense. Of course all Apple fan boys hope that Steve Jobs will come down Moses like with a phone on his arm. They hope so, since phones are ok, but definetely not as useful as we want them to be. And as they could be. If open software can fix this is to be seen.

I moved Method Software to a new server. Licenses always could be generated automatically and sent by email.
It was seven years ago that I did set up the original host. Things have changed. So nothing worked. Thanks to all that spam it is a bit trickier to set up a mail server in the internet in 2006 than it was 1999.

First thing to get right is to have reverse DNS set up right. Otherwise you get something like:


SMTP error from remote mail server after initial connection:
host actual.name.removed.com [1.2.3.4]: 5actual.name.removed.com You Must have reverse DNS setup in order to relay mail.


In other words: The ip address you are sending from must resolve to something. With

whois ip-address

you should get a domain. This is something that your hosting provider can set up for you. They have the authority over the IP range that they gave you one from. Took only an hour with my hosting provider. Another sign that they are decent.

Even after this fix the return-path was set to something stupid like ‘www-data@hostname-I-gave-the-machine’
I took a bit of googling and a couple of pointless detours to /etc/hosts and dpkg-reconfigure exim4-config before I found this blog entry that pointed to /etc/email-addresses which indeed did the trick.

Moving a development project form OS X to a Suse linux machine. Trying to install XML::LibXML.
Of course there is not the right lirbary. Which in itself is not a big deal. But Novell still is stuck in the last century: they have a page for the lib in question. But they have no download link for it! They really point to the CD. Which simply means one thing: Whenever I have the chance to recommend a linux distro it will not be Suse. Sorry, but there is neither the room nor the time for stupid crap like this page. What a tease! They say they have it, just that there is no way to get to it. Crap.

Somehow I ended up being a ‘redhat boy’. Just happened, in my former job, that lastest almost as long as linux was on the rise, it was nothing that I needed to do: Install and configure linux. I ‘just’ wrote software for it. Being freelance I know get to pick what I want to do and learn. Which is very nice. For the next two machine that will to clients I have decided to switch to debian. It’s all different, but ‘the head is round so that the thoughts can change direction’. At least that’s what Picabia said.

Debian appeared on my horizon once I had to move a site of a client to a hosting solution of their choosing, which happened to be Debian. They had to drag me there kicking and screaming. Everything was different. /etc/httpd became /etc/apache and so forth. It’s too early to tell if I really like debian. But things that are different seem to be better. Of course I missed


chkconfig


But a quick


apt-get install sysv-rc-conf


took care of that.

I actually have been bouncing around debian quiet a bit, and did horrible things to it (like compiling kernels that ought not to run, messing with raid, initrd and so forth, and so far it has been remarkably robust.
With redhat I would have not gotten that far so quick, and would have cursed allot more.

to be continued …

As nice as a working default install can be as terrible can things go with the very same software. I just wasted three hours trying to get lm_sensors to work on a Fedora Core4 install on a different, yet not exotic hardware. While I am convinced it can be made work, I just don’t feel the need to spend 4 days on something that took 4 minutes on another machine. What I really hate about Linux is that it is almost impossible to find out how long something will take. The range is usually between four minutes and four weeks. Just that some things are worth 7. Minutes or Hours, it really depends.

The other thing that I hate about Linux is that people just assume that the whole god damn world knows about their specific prerequisites. Of course you have this, that and the other thing installed and configured so that all these modules do indeed find each other. There needs to be a definition of ‘mainstream’. You go on a web page and see: Supported on mainstream, not supported on mainstream. As simple as that and you already could know if you are in the 4 minute range or not. Of course there is always a fix. Of course it can be all done easily if you just learn this, and install that. And oh, yes, of course you need to get some other things as well with it. Nightmare, that’s what it is.

The fix will be that a couple of ‘distros’ dissapear into obscurity. Would be also nice if the same configuration files would live in the same place on different distros.

BlogsNow started crashing. I wonder if it is the CPU temperature. Since I am a 5000 miles away from the computer I needed something to measure the temperature. This was harder to google to than it should be. In the end it was as simple as:


yum install lm_sensors
/etc/rc.d/init.d/lm_sensors start

now I need to copy this to another machine and then I know what happens when the machine dies.

A super simple monitoring way is to create an executable cgi file in your webservers path like this:

#!/bin/bash
echo Content-type: text/plain
echo ""
date
sensors


And then you wget / cron on a different machine to get the status and pipe it to one file …

If you try to mount a samba 3.0.14 volume from an OS X 10.4 client then the mount will never finish / connect and you will get an entry in the syslog of your server like:


smbd[30115]: [2006/02/14 18:15:46, 0] rpc_parse/parse_prs.c:prs_mem_get(537)
smbd[30115]: prs_mem_get: reading data of size 2 would overrun buffer.
smbd[30115]: [2006/02/14 18:15:46, 0] rpc_server/srv_pipe.c:api_pipe_bind_req(919)
mailhost smbd[30115]: api_pipe_bind_req: unable to unmarshall RPC_HDR_RB struct.


The fix is easy: just upgrade samba to 3.0.20 and things work again. Fedora Core 4 comes with
this ‘bad’ samba, and only OS X 10.4 barfs on it according to the net. 10.3 is supposed to be fine.