That's the only way party matchmaking works on Live. Bungie created it with Halo 2, expanded on it in Halo 3, and COD4 does it now too. Very few other games do (you have to invite for every single game, which sucks).

Make your party first. Go grab a beer. Then stay grouped all night. If someone comes on in the middle of a game, open a voice chat with them and keep them entertained until your game ends. When it does, the leader drops out, and everyone follows, then add the new guy.