ToddG
Dedicated LVC Member
Do NOT use a brush! They're just as bad as sponges or rags, and the bristles will scratch the paint. I would know, I used to use them, and I've learned the hard way.
I guess we have different experiences. I've used a brush for years and the finishes on my LS and the wife's Lexus are completely free of swirl marks or scratches. Choosing the incorrect brush could be the issue. You are correct, there are some very abrasive brushes out there, but I'm referring to a soft wash brush like this:
http://www.autogeek.net/boar-hair-brush.html
I would suggest that the OP experiment with mitts and brushes and determine for himself which does a better job for him.