Thomas Aquinas gives four reasons:
When "he was still living among them ... they were not prepared, for carnal love is contrary to the Holy Spirit, since the Spirit is spiritual love." He connects this with 2 Corinthians 5: "And he died for all so that those who live should no longer live for themselves but for him who died for them and was raised. So then from now on we acknowledge no one from an outward human point of view. ... So then, if anyone is in Christ, he is a new creation; what is old has passed away—look, what is new has come!"
It was not necessary to send the Spirit while he was living because he was their helper. They didn't need the Spirit until he was gone.
"Christ as human does not have the power to give the Holy Spirit, but he does as God. ... So that it would not seem that it was a mere human being who was giving the Holy Spirit, Christ did not give the Spirit before his ascension." (He cites paragraph 19 of On the Trinity book 1 for this idea, but I don't see it there.)
"The Spirit was not given at that time to preserve unity in the Church. We saw that 'John did no sign' (10:41), and this was so in order not to divert the people from Christ, and to make the superiority of Christ over John more evident. But the disciples were to be filled with the Holy Spirit so that they could do even greater works than Christ had done: 'And greater works than these will he do' (14:12). If the Holy Spirit had been given to them before the passion, the people might have become confused as to who really was the Christ, and they would be divided: 'You have ascended to the heights, and have given gifts to men' [Ps 68:18]."
There are a couple other concepts that are necessary to understand here. When Jesus speaks of "going away" in John 16, he chiefly has his death in mind, rather than his ascension, though there does seem to be some deliberate conflation. So the main idea here is not that Jesus has to ascend after his resurrection, but that he needs to die, in order to send the Spirit. As John Gill says:
If Christ had not gone away or died, there would have been nothing for
the Spirit to have done; no blood to sprinkle; no righteousness to
reveal and bring near; no salvation to apply; or any of the things of
Christ, and blessings of grace, to have taken and shown; all which are
owing to the death of Christ, and which show the expediency of it: the
expediency of Christ's death for the mission of the Spirit to his
disciples, is very conspicuous; for hereby they were comforted and
supported under a variety of troubles; were led into all truth, and so
furnished for their ministerial work; and were made abundantly
successful in it, that being attended with the demonstration of the
Spirit and of power.
There are also a few other verses worthy of consideration. Aquinas references John 14:12. I also believe Romans 10 is illustrative. I will discuss these verses below, though John 7:39 (referenced by Gill and Aquinas both) and the entire Farewell Discourse are worthy of study, as well as a word-study of "glorification" in John, and of John 12:32: "And I, when I am lifted up from the earth, will draw all people to myself."
John 14:
I tell you the solemn truth, the person who believes in me will
perform the miraculous deeds that I am doing, and will perform greater
deeds than these, because I am going to the Father.
Some take this to mean Christ's disciples would do things greater in kind than Christ, but I'm inclined to agree with Gill that it has to do with greater number:
The apostles, in a long series of time, and course of years, went
about preaching the Gospel, not in Judea only, but in all the world;
"God also bearing them witness with signs and wonders, and divers
miracles and gifts of the Holy Ghost", Hebrews 2:4, wherever they
went: though perhaps by these greater works may be meant the many
instances of conversion, which the apostles were instrumental in, and
which were more in number than those which were under our Lord's
personal ministry: besides, the conversion of a sinner is a greater
work than any of the miracles of raising the dead, &c. for this
includes in it all miracles: here we may see a sinner, dead in
trespasses and sins, quickened; one born blind made to see; one who
was deaf to the threatenings of the law, and to the charming voice of
the Gospel, made to hear, so as to live; and one that had the
spreading leprosy of sin all over him, cleansed from it by the blood
of the Lamb yea, though a miracle in nature is an instance and proof
of divine power, yet the conversion of a sinner, which is a miracle in
grace, is not only an instance of the power of God, and of the
greatness of it, but of the exceeding greatness of it.
Furthermore, a lot of things we take for granted now only exist because of the church. In the western world at least, hospitals became widespread as a direct result of the Christianization of Rome. The first orphanages were built by Christians. There are whole books (such as The Victory of Reason) showing that everywhere it went, the Gospel brought with it charity and social reform. This largely owes itself, as Athanasius spills much ink on in On the Incarnation, to the awe-inspiring spectacle of Christians not fearing death whatsoever, and boldly carrying the gospel to the ends of the earth. To wrap things up, let's look at Romans 10:
For Moses writes about the righteousness that is by the law: “The one
who does these things will live by them.” But the righteousness that
is by faith says: “Do not say in your heart, ‘Who will ascend into
heaven?’” (that is, to bring Christ down) or “Who will descend into
the abyss?” (that is, to bring Christ up from the dead). But what does
it say? “The word is near you, in your mouth and in your heart” (that
is, the word of faith that we preach), because if you confess with
your mouth that Jesus is Lord and believe in your heart that God
raised him from the dead, you will be saved.
We don't need to travel for miles to hear Jesus preach or have him heal us. With the Holy Spirit, all believers have God within them. God reveals himself through his church and not through one man alone. If Jesus still walked among us, who would trust his messengers? People would say, "You're not holy until you've actually met and talked to Jesus himself." But we can meet him through the word he left us and the (imperfect) fellow believers, in whom he has made himself at home. (John 14:23)
No, the Holy Spirit was definitely not absent before Jesus went away. Bob Deffinbaugh did a study on the Holy Spirit in the Old Testament. It is well worth reading in full, but I will summarize aspects of it here.
The Holy Spirit was a giver of life (Genesis 1:2; Job 33:4; Psalm 104:30), a teacher and guide (Nehemiah 9:20; Job 32:8; Psalm 143:10), and the manifestation of God's presence (Psalm 51:11; Psalm 139:7; Haggai 2:4-5). The Spirit filled and empowered men in the Old Testament and was responsible for the gift of prophecy.
The Old Testament also prophesied that the Spirit would be active in the life of the Messiah. In the gospels, we see the fulfillment of these prophecies: Jesus was begotten by the power of the Spirit and did miracles by the Spirit's power. The sending of the Spirit at Pentecost is viewed by Peter as a fulfillment of prophecy as well.
So the question naturally arises, in what way did the working of the Holy Spirit differ between the time before Jesus left and after? Or at least, why did Jesus say he was sending the Spirit?
There's an article on the same website answering that:
In the Old Testament, the Holy Spirit was given selectively and
temporarily to indwell certainly individuals for special ministries.
It was not universal nor was it permanent. ... Christ explained the
difference in the ministry of the Holy Spirit in John 14:17 when He
told the disciples, “but you know Him, for He dwells with you and will
be in you.” The change in prepositions (“with” and “in”) and tense
(present versus future) suggests a difference in the ministry of the
Spirit in Old Testament times (Pentecost had not yet occurred and the
church had not yet begun when the Lord spoke these words) and New
Testament times when the Spirit came to indwell all believers
permanently. If you note, in John 14:16, Christ said, in relation to
the Spirit, “that He may be with you forever.”
So the gift of the Holy Spirit went from "selective and temporary" to "universal and permanent." In the Old Testament, only a few were given the Spirit and it wasn't always permanent. In the New, the promise is of permanence, and that all of God's people will be filled with the Spirit.