May also want to consider how topic moving would work regarding this. If a moderator of a forum moves a topic, should they be allowed to move it to a forum they do not moderate?

Currently this would be handled using usergroups. There may be some case for having an easy way to create a group just for one member and have that in the UI, although I verge on simplicity being better (i.e. don't try and over-complicate an already complicated permissions model with extra options that aren't needed).