Updated June 12th, 2019.
The Mac Photos app in macOS 10.14 (Mojave) changes the keyboard shortcut for toggling between a zoomed-in view and the thumbnails. It used to be Spacebar— tap spacebar to pop open a selected thumbnail, tap it again to go back to the thumbnails. Spacebar doesn’t do it anymore. Now you use the Return key.
In practice you’re likely to double-click a thumbnail to open it. That’s the easiest.
You can “return” to the thumbnails three ways:
1. Double-click again
2. Click the “back” button at top left
3. Hit the Return key
If you’re used to using the Spacebar it will take some re-training before the Return key feels natural. I think it’s a good change though: when you’re in Edit mode, one tap of Return clicks the “Done” button, and the next tap of Return “returns” you to the thumbnails. Sort of makes sense.
Bonus: Command-Return (⌘-↩) takes you from Edit mode back to Thumbnail mode without requiring the second tap of Return. Similarly, if you’re in Thumbnail mode, and you have a picture selected, Command-Return takes you straight to Edit mode, saving a click.