White has limited resources, but one of them is a knight on e5 that can give check with Nxf3 or Nxg4. Those checks are of no use now, but they're important to see because they mean the knight is poised to deliver a fork if one of Black’s pieces can be lured onto another square that the knight is able to reach from f3 or g4. Another way to put the point is that any other squares the knight can reach from f3 or g4 may now be very safe for White to occupy with one of his other pieces (he would be happy to see Black perform a capture there and walk into a fork). Since White only has one other piece, a rook, the natural thing is to experiment with places it can go that take advantage of this forking potential. There is one such square: f6. If Black replies to 1. Rf6 with QxR, he gets forked with Nxg4 and loses his queen.
Well, so what if Black doesn’t play QxR? The first way to explore that question is by asking what checks you next would have—and seeing Rh6, which turns out to be mate. (Notice how constrained Black’s king is.) So actually Black does have to play 1. …QxR to avoid the mate threat, and this results in the aforementioned knight fork.
Once you understand White's mating idea here, it might occur to you to start instead with 1. Rxg5. This, too, puts White one move away from mate on the h-file (with Rh5). When you look for ways Black could fend off the mate, you see Qh6—and realize that this would arrange Black's king and queen to be forked with Nxg4+. Indeed, this achieves the knight fork without giving up a rook at the beginning, so at first blush it looks better than the brazen Rf6. But there is a catch. Are you sure Black would walk into the fork by replying to Rxg5 with Qh6? Rxg5 is a formidable threat, but it isn't a check, so Black has some freedom in replying to it. Consider not only Qe8 but the possibility that Black can go on the offensive with Qb6+. White is pretty much forced to play his king to e1 (Kf1 results in Qg1#). And then Black holds the initiative with another check: Qe3+—and notice that this is a queen fork of the rook now on g5, which White is about to lose. And then White has to worry about the Black f-pawn promoting, and will have to sacrifice his knight to stop it. (Starting with Rf6 avoids the mess just described because it puts the rook on the sixth rank, allowing it to reply to 1. ...Qb6 with 2. RxQ.)
So the short of it is that starting with Rf6 wins the game for White; Rxg5, which looks safer, loses it. To reiterate the crucial train of thought, you might overlook the resource Black has in 1. ...Qb6 by focusing too much on how he can defend against mate, since from that perspective Qh6 does seem to be his only move. The key, again, is to ask not just about defense but about what counterthreats Black might try, and especially what checks he can give (Qb6)—and then what further checks (Qe3 or Qg1). That's the biggest difference between working with checks and working with mate threats: they both force your opponent's replies to some extent, but the mate threat gives him the option of seizing the initiative if he can find a way to do it; so you must ask if he can.
Going back to the original idea, of course you might have seen it not only by thinking about your knight check but also by starting with the stuckness of Black’s king; it has no safe escape from the h-file. This makes you think of putting a rook onto the h-file, but it’s hard to get it there because Black has a queen positioned to defend against any such effort. Yet for the sake of argument you go ahead and imagine 1. Rf6, QxR—and then see that this puts Black’s king and queen both on dark squares, enabling them to be forked with Nxg4+.