We today have the image that Jesus began small and his group of followers steadily grew larger and larger. However, the opposite may be the case: namely, that he began with a large group, but people gradually left as Jesus did not live up to their expectations. Of course, many were expecting the "messiah," the political revolutionary who would unite the people and kick the Romans out of Palestine. When Jesus preached "Repent, for the Kingdom is close at hand," one can well imagine that many of the Jews thought he was talking revolution against the Romans.
JH 6:15: "Jesus knew that they [= the people of Galilee] were about to come and seize him in order to make him king by force; so he went off again to the hills by himself."
Comment: Jesus refused to be the military "messiah." The gospels imply that after the event of the feeding of the multitudes, the Galilean crowd wanted to make Jesus "king." When Jesus refuses, people drift away. John's gospel (after sermons about Jesus being the "bread of life") continues in verse 66 "Because of this, many of Jesus' followers turned back and would not go with him any more." He is left with just the twelve apostles.