. = free, A = used by process A, B = used by process B, and so forth

starting condition
RAM [.................] SWAP [..................]
program A gets executed and occupies some RAM:
RAM [AAAAAAAAAAAA.....] SWAP [..................]
program A starts going idle (for example a GIMP-process that is currently not used):
RAM [AAAAAAAAAAAA.....] SWAP [..................] (note that it doesnt get swapped, yet)
program B gets executed and occupies some more RAM, program A gets partly swapped:
RAM [AAAABBBBBBBBBBBBB] SWAP [AAAAAAAA..........]
user quits program B now.
RAM [AAAA.............] SWAP [AAAAAAAA..........]

And thats where you end up having free RAM but still something in your SWAP.
If you start using program A again (or better: if process A starts demanding its data again) the memory will get read back into RAM (and deleted out of the swap).