it is not possible to use the GPL for library X and use some other license, or indeed no source code license at all, for the body of your program. The GPL itself prohibits this.Huh? Yeah it is. You can release part of your program under the GPL and part of it under BSD or other more permissive free software license if you want.
There's a whole list of licenses whose terms are (by the FSF) supposed to be GPL-compatible; meaning that you can release software under those licenses linked with GPL software. (BSD, X11, LGPL, et al)
There's nothing that says you can only use GPL software with GPL software; the only problem is when software with a more restrictive license than the GPL is linked with GPL software and distributed. In that case, there is no way that the GPL can protect your rights as a user, so it is forbidden.