I wouldn't say you are weird. Familiarity breeds ignorance, I guess.
It is only once for each time of Salah that one is required to answer and say the dua. That is five times a day. After that, one is supposed to get busy into preparation for the actual Salah. In the Muslim countries, since adhan from different mosques sometimes overlap, the answering may be confusing, but I do manage to get through it. For me hearing it again after many years was like being set free.
It is indeed the declaration which sets you free:
Woh ik sajda jisay tu graan samajhta haey
Hazaar sajdon sey deta haey nijaat
That one prostration that you consider difficult (to perform)
Gives you relief from a thousand prostrations (to false gods)
P.S. I thought you were in the US. Is the adhan said over the loudspeaker in your mosques?