There are three obvious ways to get up there.
1. Brute forcing a ladder. With a group of people against one on the ledge, items can be dropped faster then one player can pick them up. I'm not sure what the minimum number needed is.
2. Straps with 10 clubs in them. I haven't tested it, but it might allow one person to build a platform fast enough.
3. Building a ladder so high that the player on the ledge can't reach it, then dropping down.
None of these are ideal. Straps mean less room for potions, and scavenging items for a ladder might take too long. These techniques would be even harder if the player is on the roof of a building.
So I found a forth tactic which takes only two players, and even provides enough height to get onto buildings. One player builds a ladder onto the other player's head, then they both shuffle along. Since the ladder is further away, it can't be picked up by the player on the roof.