Assalamu alaykum ukhti,
Yes, you're right in your perception, but let me clarify the reasoning, because it's not as simple a matter as it appears in mere statement.
Before I was married for the first time, I probably would not have considered a divorcee. It wouldn't have been because of the attached stigma, I've never understood that, but it would have been because of the all important "virgin factor" which is a disease that plagues young men.
Having had the experience of women, I can now say with a bit of authority that the virgin factor is very much over-rated, and indeed a girl is only a virgin once! Naturally I advise others of this, in much the same manner as I was advised in my youth... and naturally the younger brothers ignore it completely, much as I did too! Nothing has changed, but I don't think the stigma for a divorcee is attached to the divorce itself, it's just that she's not a virgin any more, and guys have too much pride to accept "second-hand goods" as their first, so to speak.
And of course, if I were to have children, I'd naturally consider women with children because my requirements would have changed. It's implied that I have the ability to accommodate children (i.e. room within the home, sufficient change to working pattern to be able to drop them and pick them up from school, etc). With my own child, I would also have experienced the full aspect of parenthood, including what is supposed to be the sweetest age of a child (approx 1-2 years old), and I wouldn't feel I've lost out on anything. Not only that, I'd also be seeking a spouse that's ready to accept me with children, and women without children are unlikely to do so.
I think the statement "like attracts like" is important. The reasons for refusal may not be the obvious ones, and it's nothing to do with stigma, just the circumstances of the group as a whole.
Assalamu alaykum wa rahmatuLlah