Most sUAS will measure altitude (AGL) based on the take-off location using an internal barometric sensor. And yes, technically the rule is written that the 400 ft. AGL is measured relative to where the drone is flying, not where the remote pilot-in-command is standing.
So if you fly out over a cliff, as an example, technically the drone would have to stay no more than 400 ft. AGL over the terrain, if you’re tracing a line straight down from the drone down to the ground.
That said, we’ve heard pilots justify that if they are just wanting to fly around the area on top of a mountain or cliff, you could consider staying within the 400 feet limit of the ‘structure’ (land formation). If your drone operations software (like the DJI app) measures distance from the take-off point, it should be easy to maintain the 400 feet limit.
If this is something that you’re truly considering, be careful of wind shear and turbulence that you may encounter as you make your way down into a valley. There could be some ‘not so obvious’ conditions that could cause problems.