I've seen a lot of speculative explanations for this passage but it can really only be understood in the context of the whole Gospel, which Elaine Pagels explains brilliantly.
In Beyond Belief, at pages 61-63, Pagels says that although the evangelist grudgingly accepted Peter as leader, he frequently has an anonymous “disciple whom Jesus loved” surpass Peter, as if the author wished to undermine Peter. Pagels says that the author of John was trying subtly to put Peter down, perhaps because he had become too venerated, even worshipped. For example:
- This disciple reclined next to Jesus at the last supper. Peter
dared not ask Jesus who would betray him, but this disciple did.
- Even after Peter betrayed Jesus and fled, this disciple remained with his mother at the cross. Jesus entrusted him with the care of his mother. This pericope is unique to John's Gospel.
- This disciple saw the Roman soldier break the legs of the other crucified men, but spear Jesus.
- He and Peter ran to the grave, but this disciple arrived first, so he was the first who “saw and believed”.
- When the risen Jesus appeared by Sea of Tiberius, this disciple was the first to recognise him, saying to Peter, “It is the Lord”.
- Although Jesus gives Peter the role “Feed my sheep”, the style suggests Jesus was not sure Peter loved him.
- Then, at this last appearance Jesus says, or seems to say, that this disciple shall not die.
I would add to Pagels points that only John's Gospel portrays Peter as not the first disciple to meet and follow Jesus.
In Matthew’s Gospel, Jesus most certainly chose Peter to lead his church, but John’s Gospel makes it clear that Peter was out of favour. In John 21:20-22, Peter sees the beloved disciple appearing to take the role he expected for himself and asks what the beloved disciple will do, seeming to expect Jesus to say that the disciple will lead the Christian church, but Jesus asks “What concern is it of yours?"
John 21:20-22: Then Peter, turning about, seeth the disciple whom Jesus loved following; which also leaned on his breast at supper, and said, Lord, which is he that betrayeth thee? Peter seeing him saith to Jesus, Lord, and what shall this man do? Jesus saith unto him, If I will that he tarry till I come, what is that to thee? Follow thou me.
Verse 23, which jars somewhat with the preceding verses, is an authorial aside written to explain that Jesus had not meant the beloved disciple would live forever, at the same time repeating and emphasising Jesus' rebuke of Peter:
John 21:23: Then went this saying abroad among the brethren, that that disciple should not die: yet Jesus said not unto him, He shall not die; but, If I will that he tarry till I come, what is that to thee?
Verse 22 certainly seems to say that the beloved disciple (generally assumed to be John) would not die, but the next verse sets out to negative this without denying that John might live until Jesus' second coming. If, as Elaine Pagels says, the real theme of the passage was an implied rebuke of the apostle Peter, then we ought not look for any sense that the beloved disciple (whether John or another unnamed disciple) would live to see Jesus return to earth, or even just that John would not become a martyr.