Are caves a hazard or will they just be there to explore?

That depends on the cave, but they are definitely an exploration thing and there will probably be nice goodies in there to be found.