The single major memory that comes to the poet’s mind is the memory of his mother.

The ‘darling dreamers' refers to are all lovely dreamers who remember their mothers in the same way as the poet does.