Can I install sod over top soil which covers soil with sand and gravel in it?

We have a lot of old pea-sized gravel and some sand in our yard and we'd rather mix it into our current soil instead of removing it before adding topsoil and sod. Any issues?