Deleting a GEDCOM removes it from the UI, but not from the INDEX_DIR - the .ged, _conf, and _priv files are left lying around.
This is by design. Delete only deletes from the database. After deleting, you can then re-add, if you choose. It is how you would rename a gedcom file.
I agree it might be unexpected. Not sure whether it is best to update the help/text or change the functionality.
Maybe the right answer is to ask the user if he wants to delete the GEDCOM, too, or leave it?