Andreas Wacker

On a centos machine ldapsearch was not giving me much love when accessing a Microsoft Global directory server via ldaps and a given port. The error message I got was:


ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)


When turning up debug level via -d 1 as in


ldapsearch -d 1 -v -H ldaps://servername:portnumber


I got the bit more revealing error message:


TLS certificate verification: Error, unable to get local issuer certificate
TLS trace: SSL3 alert write:fatal:unknown CA
TLS trace: SSL_connect:error in SSLv3 read server certificate B
TLS trace: SSL_connect:error in SSLv3 read server certificate B
TLS: can't connect: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (unable to get local issuer certificate).


It turns out that a simple line like


TLS_REQCERT never


in ldap.conf makes things better. In my particular install a simple ‘locate ldap.conf’ was a bit misleading. The true location of your config file can be revealed via:


strace ldapsearch -v -H ldaps://servername:portnumber 2>&1 | grep ldap.conf


If a machine sports edac then


find /sys/devices/system/edac/ \( -name mem_type -o -name size_mb -o -name mc_name \) -exec cat {} \;


will display quickly what kind of memory modules are visible to the kernel, and in which state they are in.

Often I work with a couple of shells simultaneously in the same directory. One may be the editor with a program in it, and the other one running it.
When I add the following lines to .bashrc


alias sd='pwd > /tmp/ddd'
alias d='cd `cat /tmp/ddd`; pwd'


I just need to type ’sd’ (for Set Directory) in a shell that is already in the right directory. When I then log in with the other shell a simple ‘d’ gets me where the other shell already is. Extra benefit: When I want to continue where I was last I just type ‘d’ again. Just a little thing. But the world is made out of little things. Lots and lots of them.

When using via the wrtg54 ssh connections timed out after a while.
Which is was mildly annoying. The problem that with mildly annoying things is they are mildly annoying.
So one does not go and fix them soon enough. In this case it was terribly easy to cure errors like:


Read from remote host 1.2.3.4: Connection reset by peer
Connection to 1.2.3.4 closed.


All that it needed was to create a file called .ssh/config in the home directory and add something like these lines:

ServerAliveInterval 60
ServerAliveCountMax 5000


Nice that it didn’t require any changes on the other end.

Over the last years I accumulated quiet a bit of mail in Gmail. It works, and I find it very inspiring to see its features grow while I keep all my data. But I also grew worried: What would happen if my mail should go away? I have paid google exactly zero for keeping all my email. There would be nothing I could do.

Turns out that it is possible to make a copy. Googles own Matt Cutts described it well

I found that these getmail parameters worked well for me:


[retriever]
type = SimpleIMAPSSLRetriever
server = imap.gmail.com
username = EMAIL@gmail.com
password = PASSWORD
mailboxes = ("[Gmail]/All Mail",)

[options]
read_all = false
verbose = 2
received = true
delivered_to = true
message_log = /root/.getmail/gmail.log


It took a while. Actually days. It seems that you only get mail out at a slow data rate. Then there is a bandwidth limit. getmail failed after a while with:


getmailOperationError error (IMAP error ([ALERT] Account exceeded bandwidth limits. (Failure)))


Just waiting a couple of hours took care of this. Having had the mail not backed up for 5 years it was quiet alright to wait 5 hours.

Another error occured with 5 mails. Getmail for instance would end with:


getmailOperationError error (IMAP error (command FETCH ('3049', '(RFC822)') returned NO))


And it would do so repeatedly with the same number. I assumed that something had gone awry with those mails. After pretending that the mail already had been retrieved via the oldmail-imap file getmail soldiered on.

Tragically at some point my connection went away. I had downloaded around 120,000 mails during that session.
Getmail updates the oldmail-imap file only when done (or cancelled via ctrl-C). So the next time it started I went to download the same mails again.

Even with that glitch things worked out. And I feel pretty good about having a copy of my mail now.

Having a secure copy of your data is never a bad idea.

just found

iptraf

and it is real nice and handy tool so see what is going on the network ports.

Very helpful.

(sorry if this should not make any sense to you. This is a note for me to go back to. Even though I bring new machines online regurlarly I forget the exact steps for this)

on X
ssh-keygen -t dsa

add content of .ssh/id_dsa.pub in the end of .ssh/authorized_keys2 on Y. Only thing we need to do on Y.

after boot of X run

ssh_info_file=~/.ssh-agent-info-`hostname`
ssh-agent >$ssh_info_file
chmod 600 $ssh_info_file
source $ssh_info_file
ssh-add ~/.ssh/id_dsa


before login from X to Y
source /root/.ssh-agent-info-hostname

sar from the sysstat modules is nice. I think it keeps about a weeks worth of history around. I’d like to have more than that. There might even be a command lines switch to do that. But often it is just faster to write what you need when you can type with reasonable speed. This script will copy all sa files into a directory called /var/log/allsa in the form saYEARMONTHDATE. So today’s sa file I can access forever via


sar -f /var/log/allsa/sa20090822


The script only cares about files that are older than a day. So it will take between 24 and 48 hours that the files appear in their final destination.


#!/usr/bin/perl

foreach my $file (@allfiles){
if ($file =~ /^sa[0-9]+$/){
my $completefilepath = "$sourcedir/$file";
my $mtime = (stat $completefilepath)[9];
my $dayage = (time() - $mtime ) / ( 3600 * 24 ) ;
if ($dayage > 1){
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime ( $mtime);
my $datestring = sprintf ( "%d%0d%02d" , $year + 1900, $mon + 1, $mday);
my $targetfilepath = "$targetdir/sa$datestring";
if (! -f "$targetfilepath"){
#print "$file $dayage $datestring\n";
system ("cp -p $completefilepath $targetfilepath");
if (! -f "$targetfilepath"){
die "tried to copy from $completefilepath to $targetfilepath and it did not work. This is a very bad sign!";
}
if (((stat $completefilepath)[7]) != ((stat $targetfilepath)[7])){
die "file sized for $completefilepath and what should have been a copy $targetfilepath did not match. Not good!";
}
}
}
}
}


shells keep a history. The default seems to be to keep 1000 lines. I have not found a reason to make this huge. And while at it time stamp it as well:


HISTFILESIZE=2000000
HISTSIZE=100000
HISTTIMEFORMAT='%F %T '
export HISTTIMEFORMAT HISTSIZE HISTFILESIZE


in your .bashrc will keep 100,000 lines (2MB “omg” ) of history around and will also time stamp it nicely. That and the grep command make for some nice shortcuts on memory lane.

When I wanted to build php with imap it did complain about:


configure: error: Kerberos libraries not found.


Turns out that this some fallout from lib64 vs. lib differences. So I found
this blog explaining exactly what was going on. Very helpful. Especially the


sh -x ./configure ...options go here...


trick can be very very helpful in the future.

If your ftp client reports

500 OOPS: bad bool value in config file for: config_name_here


then that might mean that you have a “YES ” or “NO ” with a trailing space in your config file. Easy enough to fix. I got lucky and found it quickly. Just in case somebody needs to google for this.

I have no freaking clue what I am doing. So be careful just following this blindly.
When I initially googled for this issue I did not find this specific solution:

When tried to install vsftpd.2.0.7 from source on a centos 5.2 64 bit machine I get the error:

sysdeputil.c:162: error: expected declaration specifiers or '...' before 'capset'


yum install libcap-devel.x86_64


At which point the linker complained:

/lib/libcap.so.1: could not read symbols: File in wrong format


I had to actually commend out the line

locate_library /lib/libcap.so.1 && echo "/lib/libcap.so.1";


in vsf_findlibs.sh. After that it compiled and seemed work.

While trailing the log files this messages showed up:


Jan 12 16:49:13 andreaswacker vsftpd(pam_unix)[20094]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=74.141.98.100


turns out some bot/script etc from 74.141.98.100 was trying to find an ftp user with a stupid name. Would have had no luck, but I don’t like my log files to be cluttered. So it turned out that a simple


iptables -I INPUT -s 74.141.98.100 -j DROP


blocks that IP address from now on. Nice. I think I will use that often now. There are lots of misconfigured systems out there. Like that Windows 98 computer in the philipines downloading the same file 5000 times yesterday. Thank you iptables.

while burning in a new machine with a 3ware 9560 adapter I got twice the following error messages:


kernel: 3w-9xxx: scsi0: WARNING Character ioctl (0x108) timed out, resetting card.
kernel: 3w-9xxx: scsi0: AEN: INFO Cache synchronization completed:unit=0.


Looking at the source it seems that 0×108 is TW_IOCTL_FIRMWARE_PASS_THROUGH

I had these errors only when smartd was configured to probe the drives on the 3ware card. This was under max load
during the burn in of the array. Not sure if there were any other errors etc. It seemed harmless enough, but then again
no errors are better than spurious warning etc. Once I have a couple of days of a clean running machine I probably will
turn smartd on again to see if the connection indeed exists.

A new problem that is looking for solutions: Computers are worth replacing while they are still work fine.

Some solutions

Interesting is the theme that the OS is often the reason to kick it to the curb: It is outdated (OS9), just got slower and slower (windows used to do that, does it still? Luckily I have no idea) or it just filled with malware (That WOULD be a windows feature). The hardware might do some good. I am always surprised how little harddrives have been. “Back in the day”.

people alerted me that they got an email bounce saying:

it turned out that my local mail file that I keep as a backup was bigger than 1000 Megabyte. Seems to be that postfix (or whichever program delivers the mail locally to /var/spool/mail) does not like to write to files that are bigger than that number. Scary the file grew to that size within one year.

Since years I work on a couple of computers via command line. Since they are real unix computers it all works remarkably well. For a specific solution I need to run osacompile. AppleScript needs to get compiled. I did not find a way to distribute it as text. So finally I got a hold of an OS X machine in the internet. More on that part later. osacompile really wants to run the application that it will talk to later. Also rather odd. But, hey, we talk Apple here. A sect in disguise of a technology company. So everything is possible. Or rather impossible. Like adding a development environment. The Box happened to have no Dev Tools installed. Usually that’s maybe a bit timely but overall straight forward. Installing development tools on a unix computer.
With Apple OS X 10.4.11 it turns out that doing so via ssh is not as trivial. You can download the source code. But first you need to create a developer account with ADC. It’s free. It’s annoying. They keep forgetting my password. Once you logged in,
you could download the dmg file to your local machine. I could have done that and waited only a couple of weeks for my DSL to upload the 900+ MB file to the final server I need it on. Downloading the dmg directly did not work. I had to fake a login. Which is easier as it seems. In the browser that is logged in (firefox I assume) you look for a cookie called ADCDownloadAuth. This you copy paste into the following command line:

curl -b "ADCDownloadAuth=SomeVeryLongCookieString" -O \
http://adcdownload.apple.com/Developer_Tools/xcode_2.5_developer_tools/xcode25_8m2558_developerdvd.dmg

At least that’s the valid file of today.
Once you have the file you attach (aka mount) it via:

hdiutil attach xcode25_8m2558_developerdvd.dmg

and navigate into

/Volumes/Xcode Tools/Packages

to then run:

sudo installer -verbose -pkg XcodeTools.mpkg -target /


Don’t run this against XcodeTools.mpkg in /Volumes/Xcode Tools directly. This results in the error message:

2008-01-09 03:47:43.889 installer[2843] IFPkg::_parseOldStyleForLanguage - can't find .info file (XcodeTools)


which does not google very sucessful.

The install seems to work, from what I can tell so far. I have gcc and make. And that’s all I cared for.

Autodesk aka Discreet Flame Flint Inferno applications run under irix or linux. Which is great. Unfortunately it is a long standing practise of those people in montreal to seperate different versions of their application by giving them a different user. Of course that’s just plain wrong and stupid. But if you pay north of 40,000 US$ for a single software seat you stop making reasonable demands. Discreet / AutoDesk does this since more than 14 years, why should they stop?

A couple of simple commands can fix the biggest issues with this. The first one is that each install creates a new user id. The fix is to edit /etc/passwd and give the new user a common id (100 in this example). We assume it was 101 for the new install. Running the following command as root:


find /usr/discreet/NEWLYINSTALLEDVERSION -user 101 -exec chown 100 {} \;


will fix the permissions.

Another annoyance is that they set the umask in the .cshrc of each login. If you run a couple of versions side by side it’s pretty tedious to fix these flags manually. The following does so for all installed versions.

Under Linux you can use sed for this:


cd /usr/discreet
sed -i.bak.umask "s/umask 002/umask 000/" */.cshrc


For Irix you would need to turn to perl:

cd /usr/discreet
perl -i.bak.umask -p -e "s/umask 002/umask 000/" */.cshrc


This will make the umask wide open for the user running flame or one of the other Discreet products. Some people might like that everybody can now delete
and overwrite files. Others don’t.

With a full mailbox mail might die with:

fseek: Invalid argument
panic: temporary file seek
Aborted


of course it’s all spam. That aside it seems that

mutt


can deal with these big mail boxes.

For historical reasons I run a Fedora core 4. It turns out that php5 is configured NOT to use mysql in this default install. Which is stupid, but what can you do?
Well, for starters you could follow: helpful howto.
The php source Version that it uses is 5.1.2. And it turns out that this is needed: Trying the current 5.2.1 resulted in

configure: error: The PCRE extension requires PCRE library version >= 6.6

In other words php 5.2.1 insists on having perl compatible regular expressions with a version number higher than 6.6. We ignore the fact that Perl Regular Expressions probably have not changed in the last ten years. So I am not sure why, -o- why, php 5.2 should now be insisting on a new Version.
Over at Centos there is a how to to get pcre 6.6 installed. Only problem is, that it does not fly on fedora core 4. To new for it I suppose.

So in the end I got php 5.1.2 from here and the beforementioned how to worked like a charm. Here my mirror of that specific php Version.

I am not a sysadmin. Ok, I adminstrate machine, but mostly so that I can go back and write some more horrible code. Coding for unix system is the most fun. I still remember when the Sun spark pizzabox showed up in the adjacent office and it came with a huge box of documentation: “They want you to know all this? Awesome!” On DOS PCs I was used to an environment where equipement makers would not share any knowledge.
After working on Intergraph work stations I then spent years with Irix. Which was nice at the time, since SGI had lots of money and even used some of it wisely. Of course the writing was on the wall. And running a web server on basically free hardware (except for electricity) was intruiging enough to try to deal with Linux. I am still trying to do that. Redhat was what came my way first. It worked ok, but at some point I got sick of rpm dependency stacks. Debian looked good with apt-get. So I built two boxes running that, and they drive me crazy. No chkconfig, ‘just’ use ’sysv-rc-conf’. Once in a while I have to deal with Suse, but new machines I build with Fedora. Yum is pretty much making me happy these days. I simply don’t understand why somebody thought it would be a great idea to rename httpd to apache (or vice versa). And there are lots and lots of these differences. You don’t notice them when you stay with one system. But switching back and forth makes this annoying. Comes with the concept of free and open software I guess. But somebody I would like to have the cake and eat it too.

The quality of software is quiet interesting: the core of things seems to work really well for linux. Not so much ‘core’ as in ‘kernel’ but rather functionalities. The fringes, the configurations, the interface to adminstrate these things is pretty horrible. The babylonic /etc/init.d/ confusion is only one example. Another one would be that sar is by default off after you installed it on debian. You have to go into /etc/default/sysstat and enable it. Trickier to find than it should be.

simple, yet powerful


W: GPG error: http://security.debian.org etch/updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY XXXXXXXXXXXXXXXX
W: GPG error: http://ftp.debian.org etch Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY XXXXXXXXXXXXXXXX
W: You may want to run apt-get update to correct these problems


get’s fixed by

gpg --keyserver wwwkeys.eu.pgp.net --recv-keys XXXXXXXXXXXXXXXX
apt-key add /root/.gnupg/pubring.gpg
apt-get update


as stated here

Substitute the missing key for XXXXXXXXXXXXXXXX

To find which rpm provided the file example you simply enter:


rpm -qf /path/to/example



So far botnets have predominantly infected Windows-based computers, although there have been scattered reports of botnet-related attacks on computers running the Linux and Macintosh operating systems.


That’s the NY Times being clueless about Botnets. Good that they write about it. As it is a problem.
Bad that they write so badly about it. The author seems to like ot cover his bases here. “Scattered reports”? God, there are scattered reports about Ant’s playing doom in mongolia. This is as covering. Not more. The reality is that 100% of all botnets are run on Windows machines. There are still no Viruses for OS X. There are MS Office infections that affect the OS X flavor of the product. But the Operating system has been save.

It’s as binary as that. Don’t get me wrong: Apple sucks in some areas. But their OS has had no real life virus infections. People seem to shy away from such binary truths. Easier to throw in a ’scattered reports’ here and there. Pseudo Balance. It’s actually much more harmful than it seems: It leaves loopholes. It kills the truth: Somebody with an intention could quote now the New York Times that there have been Botnets on Linux and OS X. Which is a lie. Not true. The big question that needs a real answer is, if Vista can join the club of predomiantly safe operating systems or not. Unfortunately journalists will not help in finding this out.

The only real weapon against malware is the truth.
Too bad that the New York Times is too afraid to avoid it.


mkdir /mnt/dvd
mount /dev/dvd /mnt/dvd


The simple things that I do every couple of weeks I keep forgetting.
(added slash, thanks Bernard!)

People see the shiny rack mountable Apple Servers. And then they make the mistake to buy them. Don’t! They are way to expensive. But the worst is, that they are impossible to administrate: It’s pretty much all different. And then when you spent hours and hours to go through the documentation you just have to realize that what you need is simply not there.

I wanted to change the default umask for a user that creates files on the afp server. Too bad. Not possible.

Using mac laptops since years I get to appreciate that the computer is not getting into the way when I want to do things.
Linux is a bit different, you have to google a bit for what you want to do. But 80% of it is pretty damn easy. OS X is just plain obscure. It really is a waste of time. Nobody should be believe that you actually got a unix box when you buy OS X server. What you get is some perverted BSD box that no sane sysadmin would like to deal with it. Phew.

Just try to use a device under OS X (server or not) if you have any doubts.

quick howto

how to turn old PC into something useful.

I like the idea of putting the boot CD back into the tray upside down for keeping it handy, for whenever it should be needed.

Running a postfix server with virtual mailboxes. It’s all pretty nice. Vacation however was sketchy. So say the least. One thing that breaks “Virtual Vacation” (implemented via virtual.pl in /var/spool/vacation, and, yes, googlebot, I am writing this for you) are < > brackets around the recpients email address. Entourage does that for instance. I do not think that there is anything wrong with that. Postfix does not deliver the ‘out of office’ reply in those cases though the mysql table called vacation gets the name of the sender added. Tricky and odd bug. My fix was to add:

if($from =~/< ([^\s]+\@[^\s]+)>/){$from = $1 ;}


right after the line:

if ($from =~/([\w\-.%]+\@[\w.-]+)/){$from = $1;}

which was already in there.

Free and open software is great. Just sometimes it’s the odd things that get you.

I just positively confirmed that you NEED to run a linux kernel with ‘noapic’ like this:

...
kernel /vmlinuz-2.6.17-1.2142_FC4smp ro root=LABEL=/ noapic
...


Omitting noapic resulted in an array that would loose a harddrive for a split second in the very same midplane position after 12 - 14 hours of heavy use. The array state would switch to DEGRADED and it would rebuild for a couple of hours. Never lost any data. Just allot of time in trying to fix hardware that was not broken in the first place.