Labour Party rules stipulate you must adhere to "Labour values", which has been used to suggest that persons active in other political parties are not properly eligible for membership, UKIP was very proud of not officially permitting BNP/FN members and ex-members from joining.
I wouldn't imagine this is "against" the freedom of association, since you're still permitted to associate freely - obviously, who you associate with is considered mutually exclusive from each group.