Maven, eclipse and continuous integration

2008-12-19T10:21:00.003+01:00

One of the key factors in mixed eclipse/PDE/maven projects and the most difficult to set up is a working build system with maven. Setting up a large maven project with a working eclipse/PDE build and non-eclipse projects isn't easy and unfortunately there isn't much help on the internet either, because the whole thing is very complex.

We had several redesigns of the build structure and it took a long time to get to where we are now. The good news is: it works great. Here's our set up:

Key facts:

contains eclipse plugins built with the eclipse/PDE builder from maven (maven eclipse plugin)
contains plain projects packaged as OSGI bundles to be used in non-OSGI environments such as the server and OSGI environments such as the eclipse platform
easy to integrate in CI build systems
produces an export of the product in a .zip file
executes tests
mvn site not integrated yet
note that plain projects used in the eclipse platform should always be 'bundled', i.e. in jar with an OSGI manifest in order to prevent classloader problems.
standard libraries such as spring, hibernate, apache commons stuff etc. should be 'bundle wrapped' and copied into the eclipse platform in our SVN structure. You don't need to bundle wrap them yourself, most libraries are already bundle wrapped by spring source: http://www.springsource.com/repository

Subversion structure


/pom.xml
/build
---/eclipse-build (contains the eclipse environment that actually DOES the build)
---/eclipse-platform (contains the eclipse environment used for launching the application, note this also contains all required OSGI-bundles such as spring etc.)
---/maven (contains hierarchical maven pom.xml's to be used as parent poms for the project) 
/myproject
---/module (eclipse plugins and OSGI-bundles or plain projects)
---/product (the product defining eclipse plugin)
---/pom.xml

More details:
/myproject/module contains most of the projects. Each project must contain a META-INF/MANIFEST.MF for OSGI. Optionally they can be eclipse projects (i.e. with a plugin.xml, .project and .classpath files etc.). At first we used BND to generate the manifest files automatically but now we maintain them manually as there are not too many changes being done. Do not check in .project and .classpath as they're generated automatically by mvn eclipse:eclipse.
Same goes for /myproject/product, except that there are only one or two projects: the primary eclipse plugin with the product definition (and optionally any test projects that can be run during the test phase of maven).

Now to the maven pom hierarchy:
/build/maven/ contains ALL parent poms used. It doesn't have a parent.
/build/maven/pom.xml is a simple pom that just lists all sub directories in the module section.
/build/maven/settings/pom.xml is a pom with standard dependencies listed and the according version number. This is so that you can omit the version number in your projects when adding dependencies because of the inheritance of maven poms. This is a feature of maven2. This pom does not have any parents.
/build/maven/javasettings/pom.xml has the settings pom from above as a parent and contains settings for java builds in general, such as the maven-compiler-plugin using a configuration to build java 6 projects.
/build/maven/bundlebuild/pom.xml has the settings pom from above as a parent and is responsible for settings for plain java projects in order to maintain an OSGI conforming manifest file (in our case it just contains a build -> plugins entry with the maven-jar-plugin artifact telling it to archive the manifest file for the test-jar goal.
/build/maven/eclipseproject/pom.xml contains everything necessary for building eclipse projects (uses the maven-eclipse-plugin). Uses the javasettings pom as a parent
/build/maven/pluginbuild/pom.xml is reponsible for eclipse plugins and uses the bundlebuild pom as a parent.
/build/maven/productbuild/pom.xml is a pom for PDE products and uses pluginbuild pom as parent. Contains profiles that can be selected for specifying the target OS and uses the pde-maven-plugin for actually building the product.

The hierarchy is probably a bit too complex but allows child poms to select the proper parent pom. Each parent pom in the /build/maven directory has its own dependencies and settings.

Now to the myproject directory:
You can name this directory whatever you like. This is the main directory containing the production code and test code.
/myproject/pom.xml is a simple pom with module entries pointing to the directories module and product. The parent pom is just the top pom in your trunk.
/muproject/module/pom.xml is a simple pom with module entries containing all your modules used in your project. It just uses the parent pom from one directory level up.
Same goes for /myproject/product/pom.xml, except it has a module entry for each of your product defining plugins. Normally only one but having multiple product plugins should be common enough, e.g. if you have products using a subset of all your modules/plugins.

Now any project under /myproject/module/ should have a pom with the bundlebuild or pluginbuild poms as parents, depending on whether you have plain projects or eclipse plugin projects. You just need to specify the dependencies (omit the version number if using dependencies already defined in the settings pom) and specify and additional settings you might need. Because of the pom inheritance you have everything build related already in there.

Finally you need a 'super' pom, the one in your top level directory. This one should just have 2 module entries: build/maven and myproject. When executing mvn install all the magic starts. Use mvn eclipse:eclipse to generate the eclipse specific files so that you can edit the projects within the eclipse IDE.

If you're too lazy I might put the skeleton files of the above system online but that might take a few days (need to remove any company specific stuff) ;-)

/tom

Applying the Agile Manifesto in practice

2008-12-04T14:10:00.001+01:00

Just today I came across referring to the agile manifesto. We have it framed on the wall just beside the door to our office. It usually doesn't get much attention.

We started a new sprint yesterday and a fellow team member was giving out about another (non-scrum) team regularly changing the database schema which we share. It doesn't happen often but when it happens it usually breaks our continuous integration build. My scrum team is a small sub project as part of a bigger project. We're basically developing a set of components for a specific customer. Most of the components work automatically in the background and process data relevant for the customer. In our sub project we're developing a multi tiered GUI client based on the Rich Client Platform and we're basically the only agile project in the entire project. The only interface to the other projects is a massive database.

Anyway, back to my ranting colleague. I was discussing with him and the rest of the team if it's worth putting this on the impediment list but the tenor was, 'well, uhm actually it's not that bad but in the last sprint we had this happening two times and it doesn't really fit into the agile process. If they want to change something they should tell us and we can plan it into the next sprint'.

This didn't sound very much like scrum to me. 'They' and 'us' is always an indication that something is wrong with communication. And I can't imagine the other teams being happy about waiting up to two weeks before we can integrate the system. So I remembered the good old agile manifesto and there was the solution, based on two of the four principles:

Individuals and interaction over processes and tools - talk to the guys, we're all working on the same thing. The customer doesn't care that we have multiple projects. Even though we have a agile process based on scrum, talking to the individuals is much more important than following the process by the book. It's much easier to just talk to the other folks and ask them to inform us whenever they plan to change the shared database schema so that one of us can change our persistence layer.

Responding to change over following a plan - change in the form of changing database schemes is apparently necessary in the entire project so the best thing is to 'embrace change'. Ok if it really is stopping somebody in the team from working it should be put on the impediment list and resolved soon, but this isn't really the case. It takes a minute or two to implement the changes necessary to make our software compatible to the database changes.

I thing this is a very good example where going back to the agile manifesto was a good idea. No need to discuss a lengthy process of applying database changes nor any tiring email discussions. Being agile and pragmatic works :-)

coffee.spawn

Maven, eclipse and continuous integration

Applying the Agile Manifesto in practice