Born in Munich, Gruber initially trained as an actor before making his opera debut in his native city in 1900. From 1901 to 1902, he worked at the Thalia-Theater in Saarbrücken. By that time, he had already begun studies as a singer, partly with his father. In 1902, he made his debut in the Stadttheater Regensburg and already in 1903, he was offered a position at the Staatstheater am Gärtnerplatz in Munich, where he achieved success primarily as an operetta singer.