Asterisk is not removed, as long as all its dependencies: the remove button removes only the package which configures the system. When the package has been removed, the services are stopped and port are closed.
To remove everything you need to use yum undo.