(This entry was adapted from an IRC monologue I made on the fly yesterday.)
I have a problem with KDE that some of the colours get reverted to bad settings. And as I found out (by reading the KDE source) it is stored in kdeglobals, which keeps getting over-rided to one without the appropriate keys.
So I was looking for a way to find out which process overwrites kdeglobals. Now I knew about FUSE which allows one to write File Systems in user-land. And on the FUSE filesystemspage I found loggedFS, which is a logging file system that takes a directory, over-rides it and logs all actions (using the over-rided filesystem as the backend). I decided to give it a try.
Well, it took me some time to get it up and running with my ~/.kde/share/config directory. It required some change to the build system because there was a new FUSE API version. And I needed to add a flag (after trying to find how). I also had to compile and install the fuse kernel module for my kernel. After all that it was up and running.
So I tried to use it with KDE. KDE loaded but konqueror did not run. After some stracing I found the problem was with the KLockFile class and a lock file that was created in the config/ directory.
So after some time of inspecting the code without knowing why exactly it failed, I decided to add some fprintf's there to see what's wrong. I had to compile the apporpriate "kdecore" library, with the changes I added and then use LD_LIBRARY_PATH with konqueror to load the custom library.
And as it turned out the temporary lock file and its permanent hard link did not have the same inode. This seemed like a problem with the loggedFS mounted filesystem as ls -i returned very low numbers and not the real inodes. So I wanted to see what's wrong.
I added debugs to the loggedFS driver. And debugged the kernel using User-mode Linux. As it turned out the kernel received 0xFFFFFFFF as the inode number. This took me a lot of time to find out. So I decided to compile libfuse with debugging information and find out. After stepping into the fuse functions, I found out that there was a line there that set the inodes to this 0xFFFFFFFF value, because it didn't receive the use_ino option. After I added the use_ino option to loggedFS, konqueror worked with the config directory mounted using it.
I then felt like I wanted to faint.
After running loggedfs, I realised kdeglobals was changed, but the change was not logged. As it turned out it was not done by a write call, but by moving a temporary kdeglobals on top of the permanent one (Using a rename system call). And is it turned out, this was done many times, and only at certain times resulted in the faulty contents. Therefore, I decided to create my own modification of loggedFS, to check for such renames. After doing that, I received the appropriate notice today, and as it turns out it is a kio_smtp kdeinit process. Which means, I'll have to look at its source code.
All of this took more than an entire day of work, while I neglected other priorities. I ended up with 177 unread items in my feeds' aggregator (which I've gone over since), and also neglected my email and had no IM presence. But at least I was successful in getting the problem traced.