First, I’m envisioning “system” according to the Lumpley principle of which I’ll give two formulations:
1) By Ron Edwards : System (including but not limited to ‘the rules’) is defined as the means by which the group agrees to imagined events during play.
2) By Vincent Baker (lumpley) : However you and your friends, moment to moment, establish and agree to what’s happening in your game, that’s your game’s system.
As you can see, in roleplaying games your system is everything you’re using to produce fictional content to which everybody agrees. Of course, rules and mechanics are used to produce such agreement like “I rolled a natural 20, I hit the guy with my sword !” That’s the obvious part. On the other hand, I’ve noticed that we often tend to oppose “system” to “setting” when we discuss games. Here, I’m saying that we should in fact oppose “setting” to “mechanics+procedures/techniques” while both opposites are comprised under the umbrella of system.
The idea is simple : Setting lets you know what can and cannot happen in the fiction by putting limits to what can be the case in this fictional world. For example, you’d rarely say that you’re getting on a plane in a game set in a medieval-fantasy setting. Setting here is creating expectations and assumptions on what can happen by telling us whether or not you can “get on a plane” in this fiction (in our example, setting is telling us that we can’t).
This also imply that with games that have a detailed and extensive setting, system mastery (which helps you play better) cannot be achieved without learning a great deal about the setting. Without information about how things are done in this fictional world, you as a player are unaware of options otherwise available to your character. This effectively limits your choice (once again, as a player) of actions during play.
Moreover, I’ve noticed that some players are more inclined to tolerate inconsistencies in the application of rules than inconsistencies in the portrayal of the setting (provided that they have a good knowledge of the setting). Fiction has to make sense while rules can be eyeballed (and sometimes even ignored).
All of this also means that games with detailed settings but simple mechanics can still be “system-heavy” if great portions of the setting are used in gameplay. I played a couple of sessions of Vampire: The Masquerade recently with players who are well-versed into World of Darkness lore. We, as a group, then had to sort through that body of lore to pick up the pieces that would be in use in the game and choose which parts wouldn’t. There’s often talk about “rules-lawyers” but in this case, we had “setting-lawyers”. So, time was spent rectifying or simply evacuating expectations instead of just playing and coming up with stuff by ourselves. This isn’t bad per se because it is really handy when playing a game where published material is wholly integrated in the campaign, but when you want to do your own thing, this becomes a burden.
On top of that, when you’re playing a game that aims to be faithful to the official setting, this means that participants need to learn about that setting. Considering that not every player wishes to read about all that stuff, the task of teaching the setting is often the responsibility of the GM. How should it be done ? Big crash courses before the game begins so that players can make better informed choices when creating their characters ? Long and windy explanations mid-game so that players get a better grasp of what’s going on ?