I haven’t tested this, but I think this will work. Follow someone, and turn device updates on, then unfollow them. Their tweets will still show up on your phone, but not in your public timeline. I’ve had two cases of users whom i once followed still sending tweets to my phone because i didn’t turn device updates off before unfollowing.