People you CAN NOT literally joke around over 50% of the time with someone you barely know and then all of a sudden say something you're serious about in the same joking tone and expect them to know you're actually serious.

If you intentionally choose to be a character known for never being serious then you cannot blame other people for seeing you as that.