I had the exact same symptoms - and was going crazy in the engine bay trying to figure out where the noise was coming from.
My problem was a loose section on the exhaust - It wasn't the heat sheild but rather part of a clamp around a enlarged part of the exhaust under the drivers seat.
The loose part would rattle until the exhaust got warm and expanded enough to prevent the loose part from moving.