Higekiri is weird and knows it, Papika is kinda weird and embraces it, so there's a commonality thing where he just rolls with what she says and he finds her fun to chat or hang around with? Someone who says what's on their mind whatever it is, that's refreshing for a simple old guy. If he wasn't a bandit they'd probably hang out lying on the floor together and call the sheriff stupid. Also, because he's used to the company of devoted people towards a person... it's a sword thing in other swords and it's in his brother... he totally finds it endearing, and thus Papika is endearing!