If I remember correctly, the grounds on the factory stereo's harness/HU aren't even wired in.
When your stock head unit is screwed into the metal dash bracket, that's what grounds it. I grounded my aftermarket HU's to the metal brackets/screws behind the stereo.