(I am not going to drive away those who have believed.) This alludes to the fact that they (the disbelievers) requested Nuh to disassociate himself from the believers, because they were averse to them and felt themselves too important to sit with them. This is similar to the request of disbelievers to the seal of the Messengers to disassociate himself from a group of the people who were considered weak in their social status. They wanted the Prophet to sit with them in a special gathering of the elite. Therefore, Allah revealed,
(And turn not away those who invoke their Lord, morning and afternoon.) Allah also says,
(Thus We have tried some of them with others, that they might say: “Is it these (poor believers) whom Allah has favored from among us” Does not Allah know best those who are grateful)6:53