One important aspect of glitch finding is knowing where to look.
One very good place to look is other glitches that people found. For example, it was known that exchanging equipment in FF3j while in battle can cause your old equipment to be saved in the wrong section of the inventory. A very minor problem, or so it seemed; until pirohiko found out it can be used to change your characters' classes to make them walking gods, teleport around the world map and summon airships out of thin air.
Other than that, having some knowledge of how games are programmed helps. A lot of game have imperfect collision detection for purposes of speed. This makes sense: perfect detection wouldn't be possible on the old hardware, and imperfect detection works quite well 99.999% of cases. Jump
just right, and you'll go right through the wall. It's not just collision detection, but quite a lot of things that can glitch up in case an extraordinarily rare set of circumstances is true. With the benefit of save states and frame-perfect mobility, you can easily find that one rare occurrence and exploit it. (Perhaps the most triumphant example of this is the DelayStageClear glitch in
Shinryuu and Finalfighter's Mega Man TAS, which is so absurdly rare that it required a specially programmed bot weeks of trying out random permutations before it could be triggered.)
Another common source of glitches is the phenomenon known as
race conditions. Simplified: the developers
expect you to do A before B, and if you do, everything works as intended. If you manage to do B before A (using your superhuman playing skills that TAS tools provide) you can end up severely confusing the software.
If you have some starting point, all it takes is a lot of experimentation. Some glitches require a good deal of programming knowledge (or at least the ability to read and search the game's live memory).
