On terms of Super Mario Flash 2, a layer is the cape employed in the level editor, being a group of sprites (tilings, enemies, items, etc). In SMF2, there are two layers that can be used in the level editor. Is frequent that people use layer 1 for the main sprites, and layer 2 for design, but some users change this, being layer 2 to be used more.
Some users (including professional level creators) abuse layer 2 to create spectacular designs, but in the process, sometimes leave glitches that may break the game in important ways. Read Glitch List for more info.
The Layer 3 in the game does not exist, but alludes to an optical illusion used by several users to add more ambiance from the perspective of anyone and makes the level quite unique, the layer 3 optical illusion can be done with the combination between layer 1 and layer 2.