Friday, January 1, 2010

Which build tool to use...

This blog posting is mostly spot on.

I guess I need to try out Rake, but the idea of using Ruby to solve a Java problem, seems like a bad idea too. Ant, with all its warts, is still my preference after all these years.

My favorite quote from that posting:

"Maven builds are an infinite cycle of despair that will slowly drag you into the deepest, darkest pits of hell (where Maven itself was forged)."

I couldn't agree more with everything the author said. I've been there. If you are using Maven to do builds, you should have your head checked (or even better, commit privileges revoked).

18 comments:

Christian Schneider said...

While I can´t say that maven is perfect I think it helps big companies in standardizing their builds. The maven way is really most times what you need. Of course you could do some things better by hand but this leads to projects that are not very compatible.

Besides standardization I really like the dependency management. It allows you to combine different projects without much effort. I still remember the days before maven when it was vitually impossible to combine several projects like cxf, spring and hibernate into a running application. Something always didn´t work like it should and you had no idea how to fix it. Nowadays the big and important projects can be combined without much problems. Of course maven tends to draw more dependencies than you absolutely need but this is mainly because many open source developers rather use another lib than write 10 lines of code. What really sucks is if projects are moved to new group ids or are simply configured crappy (like many sun jars). They really make your life hard. But even sun is getting better at it ;-)

Jon Scott Stevens said...

Your tolerance for crap must be greater than mine. =)

Personally, I have no problems with dependency management using the system I use.

Alexis said...

You should have a look at Buildr!
You get Rake, Ant, the few good parts of Maven, support for multiple languages and much more!

http://buildr.apache.org

Adam Monsen said...

Posting here since Kent doesn't allow comments.

I agree with most of the warts the original poster sees with Maven: configuration is painful, builds are slow, etc.

But Maven got a ton of stuff right: tiny core, rich plugin infrastructure. Writing a plugin is dirt simple. Super fast development when conventions are followed (ie: rake). Dependency management kicks Ant's butt (no more committing .jar files to scm).

I'm split. I have issues with both Ant and Maven. Maybe buildr or something else would be better.

Jon Scott Stevens said...

I've said this before, but I'll say it again. Committing jar files to svn is a good thing. I'm not sure who started that belief that it isn't, but they should be kicked in the nuts.

The problem with maven repos is that they simply aren't safe. It seems that pretty much anyone can upload jar files there. I've had several of my projects get jar files uploaded to the repos by who knows who. Other than md5/sha signatures against the files that I distribute (do you check all your jar files?), who knows what those jar files contain. I've been tempted for years to illustrate this insanity by putting a corrupt jar up there.

Second, if someone decides to remove a jar from the repo or Maven decides to change their repo format (which they did between m1 and m2), your build is broken. That is horrible. 10 years from now, you really want to be able to check your project out from svn *at any revision* and just know it will build.

If you have a ton of projects in svn and you don't want to have to check in 50 jar files into each project, you can have a central project (at work we call it KinkAlexandria) which is just a repo of jar files. There is a simple jar.properties which is a mapping of name of the jar file to a version. ie: foo.jar=foo-1.1.jar. Each project just ant imports that properties file. This system works great. Then each of our projects just references those jar files. I've been doing this for 3+ years now and we haven't had a single problem with this solution. I know exactly what versions of jar files go into my ear files and where. There is no magic and there is no confusion.

Christian Schneider said...

There is really one thing that could go wrong in a maven repo. Someone could overwrite an existing release version of a jar. Using the right server software and access policies this can be avoided and I am quite sure that the central repo has both.

Storing the jars in a subversion repo is not necessary in my opinion as each jar should only be written once.

I was curious how you solved the dependency problem but I did not really understand what you are doing with the property files. Wouldn´t it suffice to store the files in a directory hierarchy like maven does? (e.g. /groupId/artifactId/version/artifactId-version.jar).

I also have another question. How do you deploy your own lib jars? Do you also store them in your repo or do you always build from source?

Btw. another build tool some people really like is ivy. I have not yet used it myself though.

Greetings

Christian

Jon Scott Stevens said...

If I get some time, I'll write up a nice long blog posting explaining how I do builds at Kink. Suffice it to say, it works great for us and we use nothing more than a relatively simple ant build file and svn.

Adam Monsen said...

Committing jar files to svn is a good thing.

I disagree; it's not that black-and-white. Every solution has its drawbacks and benefits. I prefer means of dependency management other than committing jars. Source control is for source. jars are just an intermediate representation of Java code, a convenience. Do you also commit C libraries?

Anyway, sounds like you've got a homegrown solution that does what you want. Right on.

Maven provides a solution, too, if you host your own repository you control exactly what's in there. I host a small static repository with a few artifacts that are not in other repositories, but Nexus (a caching repository server) looks pretty straightforward too. Using something like Nexus, you need only manage specific jars... most of the time the ones you pull down just work. And you can probably tell Nexus to hold on to disallow deletions or whatever.

Your jar.properties file is your implementation of metadata about libraries. That's good. Maven has that too, and "mvn site" uses the metadata to generate dependency graphs and a report on licenses used by your project(s).

As for Maven repository safety, that sounds like an issue with mvnrepository.com.

Again, I agree that Maven has warts. But dependency management ain't one of them.

Jon Scott Stevens said...

Nexus is no different than checking in files into svn. It is just another repo. The adage that svn is only for source is just silly. Where did you come up with that rule from? Do you only check in your original photoshop files and not your jpegs? Yes, if I had a need to check in static libraries, I would prefer to do that then make up inane rules about what goes into a repo and what doesn't.

As for reports, I just don't see the need. How often do you really look at those reports? My dependency graphs are really simple and if they are really that complex, then you are doing it wrong. You have failed the KISS test.

Jon Scott Stevens said...

Oh about mvnrepository.com. It has nothing to do with mvnrepository.com. Let me be a bit more clear.

I'm one of the core developers of subethasmtp. None of us core developers uploaded the jar file into the maven central repository, nor did we authorize it. That means that anyone can upload files into the central repo. That is inherently unsafe.

http://mirrors.ibiblio.org/pub/mirrors/maven2/org/subethamail/

phil swenson said...

"I guess I need to try out Rake, but the idea of using Ruby to solve a Java problem, seems like a bad idea too. Ant, with all its warts, is still my preference after all these years."

well XML isn't java, so Ant is using XML to solve a Java problem. I still don't understand how people are so tolerant of writing scripts in XML.

Gradle, Buildr, or Rake + Antwrap gem. Use a real language! You get magical things like conditionals, loops, variables, debugging. You can sort of do these things in ant, but god it it awkward.

Jon Scott Stevens said...

Yea, I agree... XML is a pretty bad way to describe a build file. I'm not advocating for ant, I'm just saying that it works and is a better solution than Maven. =)

I will spend some time soon on some of the other tools and see how they work.

mumbly said...

Also check out gradle.

Maven is an abomination. It is horribly costly, also, as I keep having to replace monitors that I punch in anger.

The fact that it is nigh on impossible to do any reasonably complex builds without including the ant plugin (or something similar) or custom coding makes it obvious you should just skip the middleman and use something reasonable.

ant is far from ideal, but you can make it do what you want to do and once you've built an infrastructure with it, you can define your builds in a very simple manner that is much more understandable than the equivalent pom.xml. With maven, it feels like you basically have to trick it in to doing what you want.

And yes, there is nothing wrong with checking in jars. It works supremely well and you don't end up with things magically "upgrading" on you when you weren't expecting it.

And please don't tell me I don't understand maven or haven't learned it well enough to love it. Maven junkies have been using that argument for years and have not realized that if smart people (not me) have been trying and failing to work with maven in non-trivial projects without wanting to put out their eyes, their may be something fundamentally flawed with their system.

Volker Seibt said...

Most of this discussion seems to be obsolete, because since february 2010 your project subethasmtp uses maven, as the projects svn shows.

Maven build (of release 462) runs nearly instantly (if you remove the jar-signing or know the keystores password).

Jar-signing can be moved to a maven profile to be used on release or other demand.

You should use snapshot-versions in trunk to prevent from building different jars with the same version.

The maven release plugin is very helpful for releases, not only for versioning but also for svn-tagging and forcing everything to be checked in.

And have a look on m2e eclipse plugin even if it adds some eclipse specific stuff to the poms. It's very good even for larger projects, Then you won't need the checked in libraries any more.

Keep it up!

Jon Stevens said...

Two+ years later and Maven is still a pile of shit. All of your comments require some arcane set of knowledge about Maven that I hope to never learn.

subethasmtp still has a build.xml in there. Someone else contributed the pom.xml. Just because one of my projects happens to have a pom.xml in it doesn't mean any sort of endorsement towards maven.

Volker Seibt said...

"Two+ years later and Maven is still a pile of shit. All of your comments require some arcane set of knowledge about Maven that I hope to never learn."

I realy can't understand to argue with such hatred against a tool(!), which in Ken Spiller's case he obviously can't handle, and in your case even having a little knowledge about it, as you just have commited.

And there is nothing arcane, just ask the guys from your project:
"Some IDEs/IDE plugins, such as m2eclipse, will automatically download the source and JavaDocs jars and configure the IDE to use them (very handy for debugging and mouse-pointer-hover pop-up display information)."
(http://code.google.com/p/subethasmtp/wiki/ObtainingSubEthaSMTP)

I have to admit you have to learn a little to get benefit from maven, but it's the same for ant or any other build tool, and hey, afraid of learning?

I am one of those guys Christian Schneider meant in a previous comment, who have to integrate (not only) OpenSource libraries into sometimes big projects. So I'am a possible user of your library, too. A
standardized way like maven dependencies helps a lot with this and to work with such libraries (see abover quotation).

"Personally, I have no problems with dependency management using the system I use."

I have! In case of subethasmtp (if no maven support had been added by some friendly developers from your team):
1. I would have to integrate the libraries from the lib-folder to my dependency management.
2. I would have to do some additional research to get the sourcecode (for learning and debugging) and the versions of mail.jar and activation.jar (for integration).

Maven repositories are the de facto standard doing professional dependency management, or what do you think why any tool for this (like ivy, gradle, buildr etc.) uses it? Because it does not work? Thanks to the maven guys building up this infrastructure!

In advantage, if I'm using more than one library or framework in my project, there may be conflicting dependencies. maven does a good job solving this conflicts automaticaly but let's me also decide which library to use.

(continue reading next comment, I had to split...)

Volker Seibt said...

"I've said this before, but I'll say it again. Committing jar files to svn is a good thing. I'm not sure who started that belief that it isn't, but they should be kicked in the nuts."
...
"Nexus is no different than checking in files into svn. It is just another repo. The adage that svn is only for source is just silly. Where did you come up with that rule from?"

I don't want to decide who has to be kicked in the nuts, but I think with this you are wrong, especially for OpenSource projects.
Reason: If somebody for example wants to build your project on his own, if he checks out he will also get the whole stuff from the lib folder, seeing he already has most of the libraries in his own dependency repository (maven or not), because everbody has some kind of such a repository. If you look at e. g. junit-4.5.jar: it's always the same jar and will never change, whatever trustable source you use, so it's a realy silly idea to check it into SCM.

Another characteristic of svn and most other SCM is to forgett nothing. So if you switch to a newer version of a library or trying out a different library, if you have commited it, everything will be kept in svn forever. No possibility to clean up like in a Nexus repository, if you know for sure something is not used any longer or hasn't been realy used at all. This doesn't matter for svn if you have a lot of space on your harddrives you can waste. But if you will use a distributed SCM, you will start to move megabytes of old unused jars (, next to the jars in use, which you could obtain better from a more trustable source). And be aware, more and more people uses svn by the use of local git-clones to be able to work offline (or using other cool features of git).

For small projects this may be barely acceptable, but for bigger projects, e. g. frameworks like spring, hibernate, smooks etc. you will have at least 20 other dependencies for using and more than a hundred for developing (or even bugfixing). For this kind of projects "your way" will be realy inacceptable.

"I'm one of the core developers of subethasmtp. None of us core developers uploaded the jar file into the maven central repository, nor did we authorize it. That means that anyone can upload files into the central repo. That is inherently unsafe."

That's wrong. There are some people who can, but not "anyone". And again, ask your team. Since february 2010 the pom of your project contains deployment descriptors to deploy the libraries to a centralized repository.

And with your license, no further "authorization" is needed.

"subethasmtp still has a build.xml in there. Someone else contributed the pom.xml. Just because one of my projects happens to have a pom.xml in it doesn't mean any sort of endorsement towards maven."

At the end, if you can't bring yourself to have a closer look at maven, you should have a look at gradle (for the reasons meantioned above). I did not use it myself so far, but it seems to have "ant targets" as "first class citizens" and a maven-like IDE-integration.

Christian Schneider said...

I think the question is not really about maven. It is about handling external binary dependencies.

Basically external dependencies do not belong to your project. They typically have their own scm somewhere and you only use the binary they produce. So I think it makes sense to not include them in your project and instead reference them and let the reference point to an external repository. The biggest of these repositories is maven central. So I think it makes sense to use it.

The fact that these dependencies are binaries is also important. You can not really diff two binaries so the biggest advantage of a scm does not apply to them. This is the reason why maven does not use a scm as a backend.

The last reason why I would not add external dependencies to your scm is that you add a jar without global unique identity. So you are never sure which version it really is and where it came from. If you use a maven uri as reference you know it is unique (at least when talking about one repo). So when you have two projects that reference the same dependency you know they really mean the same. If they both have a jar in their scm that is labelled spring-core-3.0.0.jar or even spring-core.jar you can not be sure.

So my advice is to at least use maven central even if you do not use maven and to also export your project to maven central so others can use it.