Benedikt Howedes headed the German side in front four minutes before half-time at the Karaiskakis Stadium and, after Djamel Abdoun equalised, Klaas Jan Huntelaar grabbed what proved to be the winner.
The Holland striker then missed a penalty but his side nevertheless made a winning start.
In a quiet first 15 minutes, Paulo Machado had the only effort of note but Kyriakos Papadopoulos blocked.
Huntelaar threatened for Schalke before the two teams traded free-kicks, Jose Holebas shooting wide for the hosts before Schalke's Tranquillo Barnetta struck one into the wall.
Rafik Djebbour headed Machado's corner wide before Pablo Contreras blocked Christian Fuchs' shot at the other end.
Captain Howedes was booked for a foul on Francois Modesto but Schalke pushed forward once more, Jermaine Jones shooting over.
Papadopoulos saw a headed goal disallowed but defensive colleague Howedes broke the deadlock three minutes later.
Jefferson Farfan's shot was deflected wide and the Peru forward took the corner himself, picking out Germany international Howedes to head home from close range.
Fuchs drew a sharp save from Balazs Megyeri early in the second half, Barnetta shooting over on the follow-up, but the home side then seized the momentum.
Abdoun was denied by keeper Lars Unnerstall and Joel Matip cleared Djebbour's follow-up, but the equaliser was not long in coming.
Djebbour laid the ball square to Abdoun, who slotted home in the 58th minute.
But Schalke immediately retook the lead, Huntelaar latching on to Lewis Holtby's pass and firing low past Megyeri.
He should have made it 3-1 on the hour after Kostas Manolas brought down Barnetta in the box, but smashed his spot-kick against the left-hand upright.
Ariel Ibagaza curled a shot over for Olympiacos and they went close twice in the 81st minute, Machado's header blocked by Fuchs before substitute Leandro Greco headed Machado's corner wide.
Jones was forced off by injury late on, replaced by Marco Hoger, but his side held on as Djebbour shot over from six yards deep in injury time.
Barcelona will wear this new home soccer jersey next season. They also have a new away jersey. Get them both.
Manchester City will try and win back the Premier League title with this new soccer jersey from Nike!
The adidas Nitrocharge is the most talked about new cleat in years. It was released on May 15 and will take your pitch by storm.
The hoop soccer jersey is classic and cool. The USA hooped jersey is perfect for every fan. Get in the Gear.
Jamaica is in need of a win against the U.S. Get the jersey of the Reggae Boyz.
The Nike Hypervenom is the latest cleat silo from Nike. It will change your game and is worn by the game's top players.
Germany is one of the top teams ever and they will play the US on Sunday.
England will debut their new away jersey against Brazil at the Maracana Stadium on Sunday. Get in the Gear
The USA will wear their blue away jersey against Jamaica in their World Cup qualifier on Friday night.
Support the USA national team with an official 2013 Centennial soccer jersey from Nike. Its Official.