When I come off the highway, my trans downshifts noticeably and pulls
on the car like it would if I manually downshifted, but the car is still in OD.
Also, when I manually downshift, OD-D for a steep hill, sometimes the trans
shifts down very smooth and other times it shifts kinda quick and harsh.
Even at the exact same speeds, the downshift is sometimes good, other
times hard.

Should I be worried?

you mentioned nothing unusual....but if that is a 1984 AOD, i wouldn't plan on it lasting over 100,000, unless it already has....then I would be worried :)

Silver creates mass panic....