They are forced to stay due to your higher population, if their population votes to stay and they are still forced to leave then they should get a vote to leave you. Thats the concept of self determination.
Don't think the US cared about self-determination when the North stopped the South from seceding, or when they annexed Hawaii, or when they invaded Korea, Vietnam, Panama, Cuba, Haiti, Grenada and the Dominican Republic in the last century.