Using a different button? Nope, it's hard coded into OpeniBoot. Someone could compile an OpeniBoot that uses a different button but the problem there is you won't be able to install it if you can't go into recovery mode (broken Home button).
Although this is rare, I once went into DFU mode automatically when I just let the battery ran out when running iDroid. You can try leaving it on the iBoot selection and letting the power drain (takes ~3 hrs). Then after 3 hours, plug in your USB and see if iBoot, Recovery or DFU (blank) screen is displayed.