I've almost finished a www spider that's written in Java. It uses multi-threaded networking code to utilise full bandwidth. The architecture is highly modular so code with various functions can easily be plugged in for different effects.
I will be posting code soon.
sample uses