I have to ask - how is the chamois getting dirty? It should only be touching the clean, wet surface of the paint and nothing else.
If the chamois is getting dirty then you've got to be doing something wrong. Without question, dirt on a chamois, or anything else used on the finish, will scratch the paint.
I can't stress how important it is to be meticulously clean with anything that comes in contact with the paint.

dave and xtoy - dirt = scratches, a pain to remove

------------------