I think back to basics here. Find out what is using the swap space and not just try and guess. Here is a very simple script:
for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r
Or this question on Stackoverflow has a few scripts to try.
Or another page with various methods to investigate swap space.
Cheers.