Can you be fired for being friends with the people you work for?

Say you work in a high cl residential building with a lounge, full gym with pool, etc. as their concierge/doorman/lifeguard/porter. Now, say you become close friends with one or more of the residents and they invite you on outings, to their apartment or just to simply hang out. Can you actually be fired for merely being their friend and spending time with them? You are not, although, letting the relationship interfere with your job. You complete all your duties and do not let the person distract you while you are on duty.