I think it would be really cool if rock climbing were added to the game as a standard mechanic.
The way I would implement it is by adding a special kind of rock to the sides of steep mountains or cliffsides, reminiscent to mushrooms on trees. Those rocks would function as any other kind of ladder or could be broken off into small stones. Some of them might be cracked or unstable, so once the player passes over them, they would fall apart beneath them. These rock climbing stones could even be crafted by players to be placed anywhere.
I think this would solve the issue of outright adding climbing for every kind of block, while adding some extra mobility to early game exploration, especially cliffy areas. Perhaps certain kinds of gloves could even increase the player's climbing speed.