There is a tiny chance he might say no or simply fail to respond as you would wish. Sometimes you think it is the right moment to ask him out and then walk in to find him not in the mood. You should have a plan B just in case it does not turn out well. Plan it right, so you do not look awkward leaving or standing. You can make an excuse of having to complete some workplace assignments and reports or anything relevant. Make sure it sounds cool.