I managed to shave 24 frames off of my last result:
http://dehacked.2y.net/microstorage.php/info/1414/swordless-wip007.fcm
Omitting the Boomerang wouldn't save much, if any, additional time - the Goriyas appear in the way, and I would have to walk around them if I didn't blow them up. And if I'm going to blow them up, I might as well nab the Boomerang. There might be a couple of Keese rooms where this could be useful. Also, it might help if it is necessary to pile up multiple enemies to destroy them all at once with a Bomb, in case the spawns aren't cooperative.
Next, I'm going to use Up+A, then go left and gamble four times, then go up and buy the Blue Candle, and then attempt to obtain the Ladder from Level 4. Hopefully, I can make it through without Wooden Arrows (which would be most quickly picked up on the way to Level 6). The biggest challenge is going to be clearing the Ladder room - three Like Likes and two Zols, if I remember correctly. Like Likes take three Bombs apiece, which is going to require some *very* careful manipulation to get them all in place at once.