Given

Let 5x = 3x + 2x

Then

[using sin (A+B) = sin A cos B + cos A sin B]

[using the formulae :

sin 3x = 3sin x – 4 sin³x

cos 3x = 4 cos³x – 3 cos x

cos 2x = 2cos²x – 1

sin 2x = 2 sin x cos x ]

= (3 – 4 sin²x)(2cos²x -1) + (4 cos³x – 3cos x)(2cosx)

= (6cos² x – 3 – 8 sin² x cos²x + 4 sin² x) + (8 cos⁴x - 6 cos²x)

[using sin²x + cos² x = 1]

= – 3 – 8 (1 - cos²x) cos²x + 4 (1 - cos²x)+ 8cos⁴x

= – 3 – 8cos²x + 8cos⁴x + 4 - 4cos²x+8 cos⁴x

= 16 cos⁴x – 12 cos²x + 1

Therefore the answer is option A.