Wildlife friendly yard - how friendly is too friendly?

It was a very nice morning.  TPP walked out to take a glance at the kitchen garden. In that short distance, a rabbit, a chipmunk, a squirrel, and several birds crossed his path.  Moments earlier the view from the bathroom window (a most excellent view) featured a very large, well-fed ground hog (whistlepig, woodchuck, Marmota monax).  Unfortunately, a well-fed ground hog is not a good thing for gardens, so this latter wildlife denizen may get relocated to a friendlier location.  The kitty girls are in love with the idea of chipmunks as playthings, but this will not happen as one of the reasons our gardens are wildlife friendly is that the cats are house cats (one is occasionally walked about on a leash).  All three (four) of our local swallowtail butterflies (black, tiger, giant, and probably spicebush too)  were hovering around Mrs. Phactor's perennial bed.  However, members of the rue family (notably Citrus), are the host trees for the larvae of the giant swallowtail (bird-dropping camouflaged), and they are generally in short supply here in north central USA. This is our largest butterfly.  The spicebush swallowtail has become more common because it's food plant does grow in our gardens.  It looks a great deal like the black swallowtail, but with less yellow and more blue on its lower wings especially.  Also happy to see some monarchs flitting about, although milkweeds have not been particularly successful in our gardens for reasons unknown.