How do arbitrary-precision libraries like GMP store extremely large floating-point numbers represented in memory?
I would imagine that if for instance you wanted to compute Pi or Euler's constant to say, 2,000,000 digits that you would allocate a massive array of bytes for the digits to the right of the decimal place. Each byte would store 2 decimal place values and the array would be a member of a data structure with the number of digits and number of bytes used to store the value.
Is this how it works?
Current computers have 32 or 64-bit registers, so doing calculations on bytes is very inefficient. Also, computers work in binary, so using a base that is a power of 2 is more efficient. They'll use base 232 or 264 like Mysticial said. Each computer word will store a digit of the number and they work digit-by-digit.
In some cases you don't need much calculations but most of the time you're inputting and outputting decimal characters instead. This case using a base that is a power or 10 is more efficient. You can use base 109 in 32-bit computers and 1019 in 64-bit ones because that's the largest power of 10 you can store in a 32 or 64-bit value