You tried Xfce4 and now you want rid of it but how? rpm -qa|grep xfce just lists a crap load of xfce4 dependencies, rpm -e didn’t work, neither yum remove xfce – the fix?

This will remove all the dependencies, job done.

Do:

yum remove libxfcegui4-4.4.2-1.el5.centos