Stray would imply that his hand, subconsciously, moves to the phone. So maybe he was in a bad/awkward situation and then his hand found the phone all by itself.

If you 'stray,' you go off course (you go where you didn't intend to go). So it sounds as if his hand went slowly towards the telephone, almost without him thinking about it. Perhaps he couldn't decide whether to make the call or not. Perhaps he didn't want someone to see he was about to make a call, so he kept busily talking whilst his hand crept towards that phone. I have a friend who likes wine. Likes it a bit too much. When she's talking her hand keeps straying towards the wine bottle, and she pours herself another glass. I'm sure she doesn't even know she's doing it!

I'd say it means "to reach for the telephone slowly."

The "stray" in this sentence means to move casually to the telephone.