Why can only certain people reply to my digitally signed emails?

Only the people who have your Public Key stored in their address book can reply to you.