White sees that QxR+ doesn’t mate because Black’s knight protects his rook. But since this capture would remove Black’s rook from the back rank and also distract Black’s knight away from the defense of e8, White wonders if the sequence might enable him to then mate with Re8. Actually it doesn't; for after White plays 2. Re8+, Black can interpose his bishop on f8: again the knight’s move unmasked a route for his bishop to use to support the back rank’s defense. Since the bishop would be guarded by its king, there is nothing White would be able to do to get past it. This position illustrates the importance of considering how all of your opponent’s pieces (and all of your own) bear on a sector before you start an attacking sequence there.
Surprisingly enough, by the way, QxR+ nevertheless is White’s best move. Notice that his queen and rook are both under attack. After 1. QxR+, NxQ; 2. Re8+, Bf8; 3. RxN, White has a rook against Black’s minor pieces and retains a fighting chance. Any alternative move by White here leaves him worse off than that. (For example if White plays 1. Qd1, allowing BxR and recapturing QxB, he is left with a queen against two knights and a rook: grim prospects.)