The advance is paid back from royalties earned from the sale of the book. Once the advance has been paid back, the book is said to have "earned out", and all subsequent royalties go to the author.
If the book fails to earn out (some never do), the author is not required to pay back the advance.
The amount of the advance, whether it is paid all at once or in installments, and the individual royalty rates for various editions of the book are all part of the author's contract with the publisher.
Also see: royalty.