Remove/.Swap Rider

Hi, I'm the team captain, and one of the invitees has now declined - can I add a new rider to replace him? I sent the new rider the secret word, but I'm wondering if the other rider (who never accepted the invite) is removed, or if I'm at my limit.