I have created an installation profile for most of my sites, which I use when creating new platforms on my Barracuda-Octopus-Aegir setup. When creating new platforms I only utilize the make-files, which pretty much leaves the installation profile untested - until a second developer joins the project.

Therefore I looked into a way of getting my changes tested. All my sites are on github.com, and thus I could utilize Travis CI-integration. Turned out that it was not to difficult. I simply enabled Travis CI for the project and added the following .travis.yml to the root of my Drupal 7-installation profile.

To see the results, see the build status on Travis, and you can even implement a little image on you website telling whether your installation profile builds correctly.

E.g. the build status for the installation profile for vih.dk:

More to test

There is probably more that you could test automatically. What are you including in your tests?