Yes, it is necessary that the center of mass of the system of

particles lie in the X-Y plane.

Consider a two particle system:

Both particles have a mass ‘m’. The position of first particle is

and that of the second particle is . Now, the

center of mass of this system is given by

.

This can be generalized for an n-particle system to show that

the center of mass always lies in the X-Y plane.