Ok, clear. So it tries to insert a new junction record while one already exists. This suggests that breaking the relation fails, which would mean a major bug.
Can you also post your models and the table creation code, so I can replay your situation here and debug it?
Sorry for the late response, had a firewall panic yesterday. ;-)
Tried your code here on 1.9/dev, and no problem. It removes the junction record on the unset, and re-inserts it again on the assignment. See https://bin.fuelphp.com/snippet/view/OA
So perhaps you've bumped into an issue that has already been adressed in 1.9/dev?