I think the problem is that the phase difference between the input signal and the LO, at the mixer, will determine the amplitude of the DC output mixing product. If you have a transmission line (or delay) somewhere between the input port and mixer then the phase difference will be a function of frequency. This is why you see periodic nulls and broad peaks close to your expected conversion gain (I assume).

If the input signal was offset from the LO then the phase difference would not matter other than to cause some proportional phase shift to the output frequency tone, but for the DC mixing product the output must be a constant and is proportional to the cosine of the phase difference.

I think you'll simply need to use a different analysis option to skirt around the issue. One possibility is to use PSS and PAC analysis which will let you sweep the conversion gain for individual LO frequencies. Other options include QPSS with the input signal being the moderate level tone or envelope simulation (but envelope is a bit more complicated to set up).

I will be interested in hearing if others have solutions for synchronizing the LO to the mixer input signal.