Tu quoque arguments are posited on finding a contradiction or tension in the other side's position with regard to the matter at issue, and then holding on that basis that the other side is wrong or at least not qualified to speak to the issue. I've argued elsewhere ("Tu quoque arguments and the significance of Hypocrisy" and "The truth about hypocrisy," with Robert Talisse) that sometimes these arguments are acceptable — e.g., if someone keeps contradicting himself, that's evidence he doesn't know what he's talking about. Other times, the inconsistency of the other side is simply irrelevant to the issue (the classic example: even if your father smokes, he's right that you shouldn't smoke, and the fact that he is a smoker is at best irrelevant to the issue, and perhaps actually improves his case, as he, himself, is a testament to how addictive it is).
The tu quoque comes in a variety of forms. The most significant differentiation to make is between the inconsistencies of speech and speech and speech and act. The first is about a person who can't keep his story straight. The second is about hypocrites. Often the hypocrisy is actual — the person really says "do X" and they turn around and do not-X. But sometimes, the inconsistency of the other side isn't something that's an actual inconsistency, just one that's likely. One that would happen…. That is, sometimes the other side may not now be inconsistent, but if things were a little different, the other side would be singing a different tune. So you say, "You say that now, under these circumstances, but were the shoe on the other foot…" Colin called this phenomenon subjunctive tu quoque.
I've been on the lookout for it and for a few varieties, and I've found an interesting one in Sam Harris's The End of Faith (Norton, 2004). Harris makes the case that we (in the West) shouldn't be too hard on ourselves for all the just war norms that we bend when we fight against Muslims. His reasoning is perfectly subjunctive tu quoque. First, in defending the way Israel deals with Palestinian aggression:
Ask yourself, what are the chances that the Palestinians would show the same restraint in killing Jews were a powerless minority living under their occupation and disposed to acts of suicidal terrorism? (2004, 135)
Harris uses the same form of reasoning when mitigating blame for disproportionate use of force in Iraq:
If the situation had been reversed, what are the chances that the Iraqi Revolutionary Guard, attempting to execute a regime change on the Potomac, would have taken the same degree of care to minimize civilian casualties? What are the chances that Iraqi forces would have been deterred by our use of human shields? (2004, 146)
The reasoning is appealing, but it doesn't support the conclusion that it's OK to be more cavalier in war with Muslims. Jus in bello isn't affected by how the other side would be treating you, if they had the upper hand. If it's unjust to wage war indiscriminately, it's unjust; and the fact that the other side has a clear inclination toward injustice may be a good reason to be at war with them, but it is not a reason to break the rules of war.
This said, I do want to retrieve what's appealing about the reasoning. It does seem wrong for someone to insist on the rules of war when it's also clear that they, themselves, would not feel bound by them were they the dominant power. It seems, first, dishonest. And second, it seems like the use of moral argument is strictly strategic, instead of moral. The most that would follow from the Harris arguments would be that there is a member of the discussion who is not an honest arbiter.
One final thing is that these subjunctive moves carry a weird burden of proof, that it seems, is difficult to satisfy. It's one thing to show that someone's a hypocrite — all you need to show is that he said "Do X" and then show that he did not-X. But how do you show that the person, after having said "Do X" would nevertheless would, if circumstances were different, would do not X?