It is a genetic mechanism which prevents the pollen (from the same flower or other flowers of the same plant) from fertilising the ovules by inhibiting pollen tube growth in the pistil

This prevents the fusion of the male and female gametes along with the formation of embryo. As a result, seed formation doesn’t take place.

In the self-incompatible species, when self-pollination takes place then the pollen tube doesn’t grow and fertilization doesn’t take place. So seed formation doesn’t take place.