Hi.

It seems that nobody answers your question in both this board and EDAboard.
http://www.edaboard.com/ftopic274169.html

Do you surely read ADS's manual or http://eesof.tm.agilent.com/docs/adsdoc2004A/pdf/adshbapp.pdf ?

Why don't you simply begin with same setting as DesignGuide(Order[1]=9, Order[2]=3, Order[2]=3, MaxOrder=6) ?
Then see differences of results by increasing MaxOrder and Order[n].
This is common procedure in multitone HB analysis.
[Note] IM3 terms are LO-(2*RF1-RF2) or LO-(2*RF2-RF1). These are expressed with MaxOrder=4.

Since you are designing resistive mixer, LO power is very large, so more order and more oversample might be needed.
Why do you set large power(0dBm) as RF although you are evaluating IIP3 ?
If you set two RF input power as 0dBm, their combined peak envelope power could be 6dBm which is lesser than LO Power(15dBm) only by 9dB. I can't expect linear input condition for RF.

Although I don't know your thought and circuit's topology, I set RF power lesser than -6dBm.
Anyway you have to sweep RF input power.
I begin with two tone(LO and RF1 are large, RF2 is small) HB analysis using ADS's HB Small Signal Analysis which is same as PSS/PAC of SpectreRF.
Then I proceed to three tone(LO, RF1 and RF2 are large) HB analysis.

Hi.

  Order[1]=9 ; LO
  Order[2]=3 ; RF1
  Order[3]=3 ; RF2
  MaxOrder=6

are reasonable as initial setting, I think.

In this case, largest MutualOrder could be 15. But 15 is too large.
In general, IM3 and IM5 are important, so MaxOrder of 6 is reasonable since IM5 is expressed as Mutual Order of 6.

If you would like to input large RF power like 0dBm, you might be needed to increase Order[2] and Order[3].

Anyway you should sweep RF power.

> Yes, I did follow the DesignGuide and gradually increased MaxOrder.
> The point is simulation results seem not argree each other.

What do you see as results ?

Your frequencies are following.
  LO = 2350 MHz  
  RF1 = 2400 MHz-0.5MHz  
  RF2 = 2400 MHz+0.5MHz  

You should plot output powers of RF1-LO, RF2-LO, (2*RF1-RF2)-LO and (2*RF2-RF1)-LO with RFpower sweep.

If you see appropriate outputs I expect, your circuits is too nonlinear that it can't converge to true results with initial setting.

Could you upload simulation results curve ?

> As my thought, increasing Order results more accuracy with the trade-off of simulation time.

correct.

> More over, increasing order is supposed to decrease calculated linearity.
> However, simulation results disagree with the "calculated linearity" statement.

What do you mean by "calculated linearity" ?

> One more question: Whole design includes a mixer core and a LO buffer amplifier (with DC couple caps at input and output).
>  When I simulated without the buffer, there is no problem but when I added buffer,
> then it was very difficult to converge (not a single time the simulation was succeeded.
> I didn't have the same problem if I followed simulation setup in Examples/RFIC/Mixers_prj/MixerTOI.dsn .
> I have no idea why it was. BTW, could you comment on the differences between DesignGuide setup and Examples setup?

I checked "Examples/RFIC/Mixers_prj/MixerTOI.dsn".
 Order[1]=5 ; LO
 Order[2]=3 ; RF1
 Order[3]=3 ; RF2
 MaxOrder=4
This is minimum setting for IM3 evaluation. Usually this results could be inaccurate.

The followings are part of netlist for HB(TAHB) control for \examples\RFIC\Mixers_prj\networks\MixerTOI.
I don't have license of DesignGuide, so show me the netlist of DesignGuide.

================================================================================
============

HB:HB2 MaxOrder=4 Freq[1]=LOfreq Freq[2]=RFfreq+fspacing/2 Freq[3]=RFfreq-fspacing/2 Order[1]=5 Order[2]=3 Order[3]=3 StatusLevel=2 FundOversample=1 \
Restart=no UseAllSS_Freqs=yes UseOutFile=no UseInFile=no OutputPlan="HB2_Output" ConvMode=2 MaxIters=10 ArcLevelMaxStep=0.0 MaxStepRatio=100 MaxShrinkage=1.0e-5 ArcMaxStep=0.0 \
UseKrylov=yes SamanskiiConstant=2 \

Tran:HB2_tran HB_Sol=1 SteadyState=1 StatusLevel=3 \
Freq[1]=LOfreq Order[1]=5 \
OutputPlan="HB2_Output"

Component:tahb_HB2 Module="ATAHB" Type="ModelExtractor" \
Tran_Analysis="HB2_tran" HB_Analysis="HB2"