The Hive Archive

Discovered that the bash shell has a variable called $RANDOM, which outputs a pseudo-random number every time you call it. Sweet! Allowed me to randomize the lines in a file for a process I needed to do, thusly:

for i in `cat unusual.txt`; do echo “$RANDOM $i”; done | sort | sed -r ’s/^[0-9]+ //’ > randorder.txt

In other words, put a random number on every line, sort the file, then take off the random numbers. Worked like a charm.

# uname -a
Darwin Matt Michie’s iPhone 9.0.0d1 Darwin Kernel Version 9.0.0d1: Fri Jun 22 00:38:56 PDT 2007; root:xnu-933.0.1.178.obj~1/RELEASE_ARM_S5L8900XRB iPhone1,1 Darwin
# python
Python 2.5.1 (r251:54863, Jul 27 2007, 12:05:57)
[GCC 4.0.1 LLVM (Apple Computer, Inc. build 2.0)] on darwin
Type “help”, “copyright”, “credits” or “license” for more information.

Mmmmm tasty! iPhone with SSH, Python, BSD subsystem. Life is good.

I’ve been trying to code some more on Project Shelob (my web server) in my spare time. I’m to the point of needing a configuration file, so I can start up the server using different ports and directories for testing. Speaking of testing, I’m also to the point of needing automated test suites. I was refactoring some of the HTTP code, and when I got done, it was far more readable, and there was much rejoicing! Unfortunately, two days later I discovered I had introduced a subtle bug in keep-alive handling during a 404 event. Oops.

Anyway, I decided to use JSON as my configuration language. Simple, accommodated everything I needed, and later I would be able to easily write an AJAX GUI front end to configure the whole thing. Should be slick, right? Not as easy as it might sound. Though I have written parsers by hand, I’d rather not. Ok, so I’m using C++, surely someone has written an easy to use open source library that I can just stick in my rules and get out a nice data structure, right?

Well, kind of. There is Boost Spirit which would do everything that I want it to do, but it also required me translating the EBNF grammar of JSON into Boost’s strange amalgamation of YACC and C++. Okay well and good, but surely there is something better? After some more searching, I run across ANTLR which seems to be the spiritual successor to LEX and YACC/Bison. It even has a nice Java GUI and someone had kindly done the ANTLR rules for JSON. Check out the graphical goodness:

Still, the C++ backend wasn’t fully supported and required installing libraries and was complicated. Not 100% what I needed or wanted. All of which got me thinking about domain specific languages. Most programmers don’t consider it, but SQL and Regular Expressions are good examples of Domain Specific Languages (DSL), as are lex and yacc/bison. Up till now, I’ve frowned on the whole idea of DSLs in general. It had always seemed like bad software engineering practice to invent a new language for each problem. After all, did we really want to learn an entirely new programming language with each assignment? Who is going to maintain the code?

However, the facts point out that you have to learn an entire API anyway, and the API really just layers over what you’re really trying to do with a language that wasn’t quite expressive enough to do the job natively to begin with. Which of course leads me to LISP and through Martin Fowler who makes some good points here:

“One of the most obviously DSLy parts of the world is the Unix tradition of writing little languages. These are external DSL systems, that typically use Unix’s built in tools to help with translation. While at university I played a little with lex and yacc - similar tools are a regular part of the Unix tool-chain. These tools make it easy to write parsers and generate code (often in C) for little languages. Awk is a good example of this kind of mini-language.”

While I’ve been using SQL, regular expressions, awk, lex, and yacc for years, I’d never really classified them in my mind as DSLs. I’ve been well aware of the power of small specialized utilities aggregated together to perform a bigger task and why UNIX has been so successful at this, but I hadn’t made the leap to apply this to my programming. Fowler continues:

“Lisp is probably the strongest example of expressing DSLs directly in the language itself.. Symbolic processing is embedded into the name as well as practice of lispers. Doing this is helped by the facilities of lisp - minimalist syntax, closures, and macros present a heady cocktail of DSL tooling. Paul Graham writes a lot about this style of development. Smalltalk also has a strong tradition of this style of development.”

I’ve heard “grey-beards” and academics talk about the power of Lisp for years, and though I did some trivial functional programming in college, I’ve dismissed the rants of the Lisp guys as nothing more than rants. Today though, the ideas are crystallizing in my head, and I’m excited to explore this more.

Wes: check out introducing pipes
Matt: “Those who do not understand Unix are condemned to reinvent it, poorly.”
Matt: I hear vista finally has symlinks. Wake me up when they invent mount points and finally kill drive letters
Wes: I think you can do that somehow.
Matt: yeah sure, and break everything *nerd rage*
Wes: yeah, junction point. junction points (technet)
Matt: “Those who do not understand Unix are condemned to reinvent it, poorly.”

Update, Wes says, if you want to know more see his blogs at:

• www.brokenbuild.com/blog/category/windows-unixism
• www.brokenbuild.com/blog/2006/12/11/is-there-a-windows-equivalent-of-mount

Although I’ve had an Amazon Simple Storage Service account for awhile, I haven’t used it. For those of you who aren’t familiar with S3, Amazon has opened up their resources for everyday people to use. In this instance, you can use their servers as a place to dump your files online. Currently they charge $0.15 per gigabyte of storage used as well as a fee for the bandwidth to transfer it back and forth.

With this setup, they take care of the administration, backup, redundancy, troubleshooting, and the storage scales to whatever you need automatically. I’ve been searching for a good backup script so I can backup all the stuff I have running on this web-host, but most of them have been beta to this point or a pain to setup. Today I finally installed Brackup through CPAN, along with all the requisite Perl modules. I’ve already tested a backup and restore and it seems it will fit my needs well.

1 342 625 990 / 105 314 712 = 12.748703

Yes, log files contain a lot of redundant information. I enjoy seeing over 12X compression on a file!

Simple unix tools written in Haskell.

This is intended as a beginners tutorial for learning Haskell from a “Lets just solve things already!” point of view. The examples should help give a flavour of the beauty and expressiveness of Haskell programming.

I’m really happy that Microsoft employees are blogging more. Though I miss Robert Scoble. Microsoft really lost a lot of public relations points when Scoble left. Today, I came across a post by Raymond Chen, one of the great Microsoft guys that keeps new versions of Windows compatible with older applications. Truly, compatibility is a heroic task, one that most programmers don’t want to deal with. However in recent discussions on Windows blindly overwriting the master boot record (and in the process screwing everyone with alternate operating systems), he says:

In the discussions following why Windows setup lays down a new boot sector, some commenters suggested that Windows setup could detect the presence of a non-Windows partition as a sign that the machine onto which the operating system is being installed belongs to a geek. In that way, the typical consumer would be spared from having to deal with a confusing geeky dialog box that they don’t know how to answer.

The problem with this plan is that not everybody with a non-Windows partition type is necessarily a geek. Many OEM machines ship with a hard drive split into two partitions, one formatted for Windows and the second a small non-Windows partition to be used during system diagnostics and recovery. The presence of this small non-Windows partition is typically not well-known, and it comes into play only when you boot from the manufacturer’s “system recovery CD”.

I would challenge Raymend Chen to install Linux, because this problem isn’t difficult to solve and has been solved by every major Linux distribution years ago. This has been one of my biggest all time gripes with Microsoft. They put on blinders and ignore everything not invented at Microsoft (except when they steal Apple’s GUI, but that’s another entry). I’ve reproduced the common system partition types that Linux fdisk knows about. If Microsoft took this list and detected the top ten most common ones, they could solve this problem. If they decided to spend another couple hours implementing all of them, they would make installing Vista a breeze for those of us who know there is more than one Microsoft way. However, they won’t because why would Microsoft care if they overwrite your grub/lilo boot record? That just means you will only be using Windows, right? I think they forget that I am a customer too, and I don’t appreciate it when a product destroys my setup.

When are we going to start making Operating Systems install secure by default?

$ netstat -a|grep -i listen|awk '{print $1}'
*.sunrpc
*.32771
*.lockd
*.32772
*.32773
*.32774
*.32775
*.32776
*.telnet
*.ftp
*.finger
*.login
*.shell
*.fs
*.32777
*.ssh
*.5987
*.898
*.32778
*.5988
*.32779
*.9010
*.32780
*.32782
*.32781
*.smtp
*.smtp
*.submission
*.telnet
*.ftp
*.finger
*.login
*.shell
*.fs
*.ssh
*.smtp

Found the follow handy .screenrc posted in a forum. If you are a heavy UNIX user and you haven’t used GNU Screen, you are missing out on one of the handiest applications around. Highly recommended. The following adds a nice status line and F5 and F6 to switch through windows, F7 to rename the active window, and F8 to create a new window.

bindkey -k k5 prev
bindkey -k k6 next
bindkey -k k7 title
bindkey -k k8 screen
msgwait 1
autodetach on
nethack on
sorendition 04 43
hardstatus alwayslastline "%D %M %d %c | %-w %{+r} %n %t %{-} %+w"
vbell on
defscrollback 1024
startup_message off
defutf8 on
term xterm