I've been helping Potato with the TAS, mostly researching hat teleports, and I've found a lot about how they work. Bit of a long post.
The first thing to understand is the follow path. It's a list of the 256 most recent x,y positions the front bro has been. Actually, on a frame of normal movement both bros advance 8 indices on the follow path. The front bro overwrites 8 entries and goes to the 8th one, and the back bro is just moved 8 entries ahead. They're 64 entries, or 8 frames apart. So the whole list is looped every 32 frames. When Luigi is underground and their speed is halved, they each advance 4 entries per frame.
The follow path is "reset" often: this means that the first 64 entries are overwritten with 64 equally spaced positions between the bros, with the front bro at entry 64 and the back bro just behind entry 1 (where entry 0 would be if it existed). Resets happen when entering a new area, switching, and finishing a bros move such as high jump, spin jump, mole, or mini.
The magic happens when Mario uses a glitched hammer on Luigi and then high jumps on him and gets off, causing Luigi to move at half speed while Mario retains normal speed. The high jump resets the follow path, so Luigi is at 64 and Mario is at 0. After a frame of walking, Luigi advances 4 to 68 and Mario advances 8 to 8. Each frame the distance between them shrinks by 4, so after 16 frames of walking they both end up at entry 128. The next frame of walking, Mario is moved to entry 136, and this actually happens before Luigi overwrites the 4 entries ahead of him. This can cause a teleport because the position at entry 136 may be far away.
In this glitched movement state, Luigi actually has normal speed in the air. So if Luigi jumps, moves for one frame, and then moves for 16 frames after landing, the bros will coincide at entry 136 instead, and then Mario will warp to entry 144. So we can warp to any entry on the follow path (that's a multiple of 8) by moving for different amounts of time in the air. Of course, warping to entries 1-128 is not as interesting because they will definitely be overwritten after high jumping, whereas entries 129-256 can be "stored" from previous movement.
This explains where the game tries to send Mario. But in practice Mario rarely goes to the exact position on the follow path. The biggest breakthroughs have been figuring out what causes this.
First, the game doesn't want to move Mario more than 0x8000 in each direction. So, for both x and y, it only looks at the lower two bytes of the follow path position, and chooses the upper two bytes so that Mario ends up as close as possible to his current position. In other words, it adds a multiple of 0x10000 to the x and y position on the follow path so that the position Mario is teleported to is within a square of side length 0x10000 centered at Mario. Imagine the lattice of points separated by 0x10000 in both dimensions, starting at the follow path position. One of these points will be inside the square, and this is the point Mario is teleported to.
In many cases, the game will also avoid teleporting Mario through walls. If there is wall parallel to the y-axis between Mario and the destination x coordinate (after being adjusted as above), he will be sent to the wall's x coordinate instead. Similarly for the y coordinate. However this doesn't always happen - all of the useful hat teleports are for going through walls, of course. It seems like most walls will block Mario in this way, but there are plenty that don't. I don't understand why walls act differently, but it's easy to test this for any wall. I've noticed that when travelling north and west it tends to be easier to go through walls than when going south and east, but this isn't a universal rule.
There's the matter of where Luigi is sent as well. After the teleport, Mario faces his previous position, and Luigi is sent the usual distance in front of Mario (in the direction he's facing). This is important to keep in mind in some cases. Also teleporting resets the follow path, except in some weird cases that I haven't investigated much. If the follow path isn't reset, Mario goes 8 more entries ahead and teleports again on the next frame of walking. This might be pretty powerful if we can figure out when it happens.
To cancel the glitched movement state, you need to high jump and actually jump. However there seems to be a weird bug where hat Luigi gets stuck at a lower z-level than the ground after landing (from a high jump or a normal jump) if the ground is above z=0. This is a problem because Luigi can't emerge unless he is exactly at ground level. This means you have to end the glitched movement somewhere with ground at z=0. If there is a way around this, it would be great.
Now here are some strategies for setting up a teleport once you have figured out what your destination and position before warping will be, and verified that the teleport doesn't get blocked by a wall. First you need to find somewhere to set a follow path entry to the right value. Remember only the lower 2 bytes matter in the end, so you don't need the exact position. Usually it's best to find a position in the previous room that matches the lower 2 bytes of the destination. If you get the position into entry 136, all you have to do is walk 17 frames for the warp to happen. If the position is in a later entry, you will need to jump before the warp, which costs ~20 frames. Warping to position entry 256 requires only 15 frames of air movement, so any entry besides 136 will take the same extra time to setup.
The tricky thing is avoiding overwriting the follow path while quickly moving to the position you need to warp from. The gist is, you need to reset the follow path somehow right before walking far enough to overwrite the key entry. Hammering Luigi and high jumping on him are part of the setup, so these can be used as resets. However during the time between the hammer and the high jump needed to switch action commands, if you walk the whole time, you'll overwrite the whole follow path. Ideally, you can set the follow path position during this time, then enter the loading zone, walk a bit and warp. But this only works if you can do the warp pretty close to the area entrance. If you need to go further before warping, switching just before you overwrite the key entry is a good choice. High jumps can be used if you need to gain height while maintaining the follow path. Spin jumps can be great because you travel at normal speed during them without overwriting the follow path, but they're inconvenient to access, since Mario has glitched commands. The earlier the key entry is on the follow path, the more often you'll need to reset the follow path, which is slow. So I imagine it'll often be optimal to use entries late in the follow path, even though they require the extra time to jump before warping.
I might have missed some important details or got them wrong, please let me know if you have any questions, insight, or ideas.
Your first priority is to get in the right position.
You can use →↓, ← and ←↓, → inputs.
Then your next priority is to approach the wall you want to clip into (top or bottom).
Are you trying to use the trick in Bowser's Castle room 1? I may have tried that and failed, but I can't remember.
