What is destroying the people is that we are still following God from sensual motives. Our love for God is narrowed to what we can get from Him and not how we can relate to Him.
Jesus is telling the people that God, not Moses, gave bread, but they have seen Him multiply bread and do all kinds of signs, and they still refuse to believe.
