Apparently it does matter - quite a lot in fact!
From a post on the Zwift Blog
Zwift Weight/Watt/Equipment Speed Tests
"Equipment does matter–especially wheels! The Zipp 808’s (the level 10 unlock) are 19 seconds faster over a full Richmond lap than the “stock” 32mm carbon wheels. The fastest bike available to all riders (other than the TT bike) is the Cervelo S5, which is 11 seconds faster over a full Richmond lap than the stock Zwift Carbon.
The difference between the stock setup (Zwift Carbon and 32mm carbon wheels) and the fastest race-legal setup (Cervelo S5 and Zipp 808’s) is 33 seconds per Richmond lap, or approximately 70 seconds per hour."
There's a lot of other observations about rider height and weight on that page that are quite interesting - such as the fact that since you can't upgrade the wheels, the much-coveted Tron bike isn't really that fast. I guess I'll be sticking to the Zipp 808's and the S-Works frame for races or any other time where speed matters.