Based on your results, I think you need to just concentrate on the sample and hold action. After that, you can start worrying about comparator problems (which I'm pretty sure is NOT the issue). You are already seeing distortion with a simple switch and capacitor (no ADC effects).
What about switch charge injection? Are you doing bottom plate sampling? Does this help fix your ideal ADC model, and you get back close to the ideal ENOB?
Slide 3 is how a your ADC should be sampling. Two switches in series with the capacitor. The one connected to gnd or vcm should be open'ed first, then the input sampling switch. This will reduce charge injection going into the ADC.
https://inst.eecs.berkeley.edu/~ee247/fa09/files07/lectures/L18_2_f09.pdfSimilar thing on slide 33 of the sample and hold pptx
http://www.utdallas.edu/~yxc101000/courses/7327/handout.htmlNote, just for future reference, in faster interleaved ADCS, this technique is not always used due to speed considerations (there are 2 switches in series now instead of 1 for the time constant). But the charge injection is not a major error in these designs.