You do see that solution advocated a lot by folks like Richard Bjetlich, who has the Tao of Network Security Monitoring book among others. Where it breaks down in practicality that logs have an advantage is in volume and resources. It takes more expertise to pick apart packets than it does logs (and it already requires expertise to pick apart logs), it takes a lot more storage to keep packets, and logs tend to distill information more quickly.
There is a powerful case for them to be used in conjunction with each other, though - if you could go back to the packet trace from something suspicious in the log data, or going the other way alert from the packet data into the SIEM to be warned about potential bad behavior that logs can't see but packets can, they become Better Together. (Hopefully downstream, the stuff Rob talks about with deep packet analysis can extend in this direction - Beta for SolarWinds "Deep Traffic Analysis" now available).