This can be explained by Newton's Third law: The mutual forces of action and reaction between two bodies are equal, opposite and collinear.

Jumping off the boat is the action, and as a result the boat moves backward is the reaction. This is the reason why boat moves backward when the man jumps out of the boat.