In our stake, the Singles branch will not keep members on their records who don't attend. They will send their records back to the ward where they live. (If singles attend occasionally, they have hold the branch clerk at arms length to keep him from pulling their records in, LOL.) It might be useful for the family and YSA unit bishops/branch presidents to discuss which unit might best for the member and who can best meet their spiritual needs.
For us, one twin chose to go to the singles branch and he is loving it. Our other twin chose to stay in our family ward where she felt more needed and useful. As russellhltn wisely advised, the needs of the member should be paramount in the decision.