How do i stop this from happening ?

you need to tell those guys to just lay off and get over the fact that yall are talking. sounds to me like jealousy. also ask him about it. if he is fine with yall talking then he shouldn't take that crap from them. but you know your still in 6th grade. its normal immature boy crap.