No one “makes” you feel guilty. If you feel any kind of guilt, then you just feel guilty. If that bothers you enough to want to do something different then that’s on you.
Everyone else can say all the shit they want, immature, passive-aggressive, or not, but you feeling guilty over it isn’t their problem, it’s yours. If you can’t hold fast or say no when you clearly want to, then, again, that’s on you.
It’s not anyone else’s job to emotionally or morally regulate you, no matter how crappy their attitude, because you are not a child that requires that, allegedly. Adults who make a big deal out of being “made” to feel guilty are children.
This idea that we can simply curate ourselves as being only positive and helpful in other people’s lives because it’s bad if they “make” us feel guilty is shit.
P.S. And yes, this means that “rape” by passive-aggression or repeatedly asking a mature, grown adult (past 18 for me) is not rape because it’s not by force, intimidation, or threat. It’s just you not wanting to be bothered, annoyed, or feel bad, and definitely making a choice that you own whether you want to or not. Adults know that sometimes they’ll feel bad about saying no, and that’s okay, and it’s not anyone else’s problem.