The beauty of Jenkins is the simplicity at setting up a job with the web frontend - but once you get over a certain level of complexity this is actually one of the bigger drawbacks.
Sure, we've got some templates, but how far can you really stretch it?
In my situation, I need to:
- Trawl SVN/other version control for all packages available - several hundred
- Only if the package has tests, add an entry to the CI suite
- Adapt to packages which require E_ALL & ~E_STRICT to run happily under that
- Packages which require dependencies, but can't be installed, still need a mechanism to install said dependencies
- And some which need to be invoked with the legacy AllTests.php
- Detect when a package has migrated to github
- ... and update an existing build/job with a new tool when required
I had tackled part 1 with pear's "packages-all" SVN link, which pointed to the trunk branches of all relevant code, and written some scripts for cruisecontrol to find all directories with a /tests/, but I find myself in need of something more.
So, my code is on github for now, and you can see the current CI system where those scripts have installed new jobs.
I'm quite sure that pyrus and a local installation will deal with the dependencies; as they are all described with PEAR's package.xml format. Also; detecting when a package has shifted to github should be fairly easy to tackle, as there is much work underway to deal with migration.
The one area I need to explore is manipulating jenkins jobs via xpath, to understand what parts of a job are already present and what need updating - basically number seven in the above list.
I'm curious who's done this sort of thing before, regardless of language, and if there are any libraries which make it easier to do this sort of thing.