The phrase, "Well, you're comparing apples and oranges" is absurd. Apples and oranges can be easily compared: they're both fruit, they have seeds, an outer protective skin, they are both basically round, about the same size, grow on trees, can be eaten by themselves, they're similar in classification and usage, orchards and groves are basically two words for the same thing....
You arrive at the color orange by mixing yellow and red, two of the most popular colors for apples.
Intelligence can be used to compare all things, though intelligence can also be used to discern when it is perhaps best to restrain from doing so. I won't bother again with attempting to dictate or define the uses and limitations of intelligence. I've embarrassed myself enough for one month's posting, even if it was last month.
I suppose the phrase is meant to be used when the focus should be on comparing two things of the same kind, rather than two separate kinds.
But again, if you're using the same criteria or basis of comparison, then what's the problem? Isn't that the very essence and usefulness of analysis? To find and define corollaries, analogous attributes, differences and distinctions? To investigate the component parts of the whole, and their relationships?
It is precisely this sort of talk that has gotten me into trouble most of my life. Well, things like this and the illogical extension of things like this. Comparing beer to whiskey, etc. I always seem to question the idioms at just the wrong time, or just the right time, depending on how you look at enjoyments and advancement.
Or is that comparing apples and oranges again?
Or is that just another abstraction...