Based on a patch from Adamantin and cleaned up, corrected and extended with feedback from GriffonHeart, Ntsc, Laizerox, Triply, hunuza and others.

The documentation for this patch follows. The larger part describes how the game mechanics of instances work. First of all if anyone finds some feature of the instance system that is either incorrectly described or missing from the doc, do tell. Please test the patch and report bugs only when the system does not perform as described. Please report any console errors (that doesn't happen without the patch) at server startup or while the server is running. Crash dumps/backtraces and server log snippets are also quite welcome.

Some of this may still change but this is what I have so far..

Instance System

An `instance` (which is used as a shorter form of `dungeon instance`) is represented in code by the InstanceMap class. This inherits from Map but has many specific properties. Any map can be instantiated (made copies of) but not all maps are dungeons (`battleground instances` for example).
There are three kinds of instances: normal instances, raid instances and heroic mode instances. Raid groups may enter normal and heroic instances (there's a maximum size though) and are the only way to enter raid instances. Unlike raid instances, heroic instances are a higher difficulty level of normal instances. They usually contain higher level mobs with better loot. Most normal instance do not have a heroic mode.

When players enter instances (InstanceMap::Add) InstanceBinds are created. There are four types:
a) Solo PlayerInstanceBinds are created when the player enters an instance alone.
b ) GroupInstanceBinds are created when any member of the group enters an instance.
c) Permanent PlayerInstanceBinds are created for Raid/Heroic instances when a boss is killed in the instance for the group members that are currently in the instance and aren't bound permanently already. This is referred to as being "saved" to the instance in game terms and these "saves" are shown in the character window. Players become permanently bound to an instance (if they aren't already) when they are inside when a boss is killed or when they enter an instance that the group leader is permanently bound to. At these moments, players are assigned raid IDs.
d) Permanent GroupInstanceBinds exist iff the leader has a permanent PlayerInstanceBind for the same instance. This `perm` property is only cached to avoid doing a query when the group leader is offline and we need to know if players should be saved to the instance when they enter.
Note: In fact it would be enough to use only PlayerInstanceBinds for storing all this information since groupinstancebinds copy information from the group leader's bind. That however would use more memory and would need additional queries for cases when the information is required but the group leader is offline.

A player or a group can be bound to exactly one instance of a map at a time, at a certain difficulty. You can for example be permanently bound to an instance in heroic difficulty and still be able to enter the normal difficulty of the same and even reset it multiple times.
If a player logs into a dungeon map but without being bound to any instance of it (can happen if he had been unbound from the instance while inside and logged out for example), a new instance will be created for him and regardless of his original position, he will be positioned near the entrance of the instance.
In a group there can be players that are permanently bound to different instances and even players that are not permanently bound to one. Due to this, when the group's leader is changed the group binds are updated to reflect the new leader's permanent binds. The leader's binds determine which instance players entering a dungeon will be put into and whether they will be "saved" to it. However when a member other than the leader enters a new instance, the group will be bound to it.
When a solo player invites someone to form a group (not joining another group!), his bound instances are not reset but the player binds are converted to group binds. More precisely his permanent binds will be bound to both him and the group but normal player binds are cut and replaced with group binds. The converted normal instances are reset if they can be after he leaves the group however.

The MapInstanced class is currently a container for InstanceMaps. It will probably be removed later.
Currently InstanceMaps are created (and/or looked up) by MapInstanced::GetInstance via a call to GetMap. The player's permanet player bind is taken into consideration first, then the player's group bind and finally the solo bind. If none of these exist a new instance is created.
InstanceMaps are unloaded a configurable amount of time after the last player exited the instance. There are a few special cases:
1. The "Reset all instances" can only succeed if there is noone in the instances so in that case the maps are also unloaded immediately.
2. When a group is disbanded the members get a one minute homebind timer and the map will be unloaded after two minutes. The same thing is done one minute before the global reset time if players are in the raid instance at the time.
3. When the difficulty was changed by the group leader the map is unloaded immediately after everyone has left the instance.

Resetting an instance is a less intuitive term for deleting it from the database (instance, creature_respawn) and unbinding players from it (group_instance, character_instance). The name comes from the fact that from the player's perspective if he tries to enter an instance that has been reset, he will he will see that all mobs have reappeared, have come back to life, have been reset. In server terms this means that he gets put in a newly created instance.
Note: If you were inside when the instance was reset you may still kill mobs so the entries for the instance in the creature_respawn table will only be cleared after the map is unloaded.
Instances CAN be reset if there are no players permanently bound to it. Instances WILL be reset under several circumstances:
1. The use of the "Reset Instances" option in game. This can be done either by someone playing solo or by a group leader. Only normal instances will be reset in this manner. The player receives a message for each instance that the reset was successfull or has failed for some reason. This option is disabled for players inside instances. Group leaders may use this option but the reset will fail for instances that have members in them. Those also get a message telling them to leave the instance to allow the leader to reset. The instance is not unbound from the player/group if the map cannot be reset.
2. The use of the "Change difficulty" option in game. (disabled inside instances) You get almost the same messages as "ResetInstances" for success/fail.
a) In the solo player case
Permanent instance binds remain untouched. Everything else is unbound and reset for the previous difficulty.
b ) In the group case
When the leader sets the difficulty, this is set for all members of the group even if they are in the instance at the time. If those instances can be reset, they are reset immediately. Players inside get a warning message but don't get a homebind timer. They can continue playing in the instance but if they leave and come back or relog, they get put in a new instance.
3. Raid/Heroic instances have a reset time, a global moment in time when all such instances of the same map are reset. These global resets occur once every few days at a given hour.
The reset times are shown in character window for all "saved" instances along with a raid ID , identifying the instance. This information is sent to the client by SendRaidInfo every time a player opens the character window. When the reset time is nearing, several warnings are sent out to all players in the instances until finally they get a homebind timer.
4. When joining a group, all solo instances are reset, including heroic instances that you haven't been bound to!
5. When a group is disbanded (everyone has left) all instances it was bound to will be reset if they can. Generally when a member leaves the group inside an instance he gets a homebind timer.
6. Normal instances are reset after a long enough time that all the mobs inside respawn by themselves. This reset time is updated in memory every time a mob is killed and the instance is not reset while players are inside. For performance reasons the resettime in the DB is only updated when the instance gets unloaded and even then only because the resettimes are loaded from the DB every time a player logs in. The value is recalculated at server startup based on the values in the creature_respawn table.

InstanceSaves are templates for creating InstanceMaps and they have nothing to do with the ingame term of being "saved" to the instance. The bind classes have pointers to them and they are stored in the InstanceSaveManager. To save memory not all InstanceSaves are loaded. InstanceSaves contain lists of the players and groups that point to them. InstanceSaves are kept in memory as long as there exists a PlayerInstanceBind or a GroupInstanceBind for a particular instance.
Since PlayerInstanceBinds only exist for online players, InstanceSaves may be unloaded when players log out.

Database changes:
- instance_template: The reset_delay now denotes the number of days between each global reset for that map. If it's 0 the value is taken from the DBC. The resulting value is multiplied by the Rate.InstanceResetTime config.
- instance: The new column `difficulty` is 1 for heroic instances and 0 for normal ones. The `resettime` is now 0 for raid/heroic instances. Note: while the server is running this value is not updated every time the reset time changes for normal instances, but the value is recalculated at server startup.
- instance_reset: This new table stores the global reset times for each map. It is updated at every server startup.
- character_instance: This reflects the character-instance binds as described above. The perm property is 1 for the equivalent of being "saved" to the instance in game terms.
- character: The new field dungeon_difficulty preserves the value of the chosen difficulty in the interface between relogs.
- group_instance: Same structure as character_instance and with values as described above.
- group: The new field `difficulty` preserves the value of the chosen difficulty in the interface between relogs. This may not match value for each group members if they were not online when the difficulty was changed.
- creature_respawn: No changes but used for calculating normal instance reset times which by default is max creature respawn + 2 hours.
- areatrigger_template: New fields in this table are related to additional requirements when entering the instance. These are `required_item2` for an additional item, `heroic_key` and `heroic_key2` for two keys that can open the instance in heroic mode (you need atleast one of them), `required_quest_done` for a required quest and `required_failed_text` for instance specific requirement messages.
- creature and gameobject: Each map has a spawnMode that is currently used for the difficulty setting (0 for normal, 1 heroic, 2 epic). The new field `spawnMask` in these tables allows spawning a creature in one or more spawn modes. Each bit in the mask corresponds to a different spawnMode. So the decimal value of 3 (binary 11) will spawn the same creature in both normal and heroic mode while the value of 2 (10) will only spawn that creature in heroic mode.

TODO (things that are features of the instance system but I will not implement yet):
- You can only zone into 5 different instances per hour.
- If you die and release spirit in an instance, you are teleported to the spirit healer in ghost form (unless there is no path from the spirit healer to your body on foot in which case you are resurrected at the healer, eg. Tempest keep). It is considered that you have left the instance and others can enter. The instance may become full before you reach the portal so you will not be allowed to enter in ghost form either. You will not get any warning message but you are resurrected immediately at the portal.
- When a new instance is created for the player at login, he should be ported to the entrance of the instance.