Tuesday, December 14, 2010

jmxtrans : queries jmx and writes out the data in whatever format you want

For a long time now, many large Java projects have diligently added support for JMX to their systems. Even the big players like Cloudera's Hadoop have great blog posts about how they have all sorts of great metrics available through JMX. In addition, Tomcat, JBoss and ActiveMQ all have metrics exposed through JMX.

Sadly, there is not a single project out there that makes it easy to take the data out of JMX and convert it into something that can be graphed. Some suggest using jconsole to look at things. jconsole is great if you want to look at a server at a time. Others suggest graphing or monitoring with one of the many tools available. JMX4Perl, Hyperic, Cacti, OpenNMS, Nagios, etc. These tools all have their advantages and disadvantages. However, what they all have in common is that they are ridiculously difficult to configure and use in a production environment.

Enter my new project: jmxtrans. Yes, stupid name, but it really serves a purpose. jmxtrans allows a sysadmin to define a simple json configuration file and then execute that config to produce a list of results that can be transformed into whatever format you want. Out of the box it supports cacti/rrd databases and graphite. I'll add more as time goes on. It can also scale to support massive clusters of servers with a fully multithreaded engine. It also has an api which makes it easy to add to your existing application and it can be used from any one of the Java'ish languages (Scala, Groovy, JRuby, PHPJavaBridge, JPython).

This image...

Was generated with this simple json configuration...

  "servers" : [ {
    "port" : "1099",
    "host" : "w2",
    "queries" : [ {
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.GraphiteWriter",
        "port" : 2003,
        "host" : ""
      } ],
      "obj" : "java.lang:type=Memory",
      "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ]
    } ]
  } ]

This project has been about 3 solid days of work and I'm really proud of it. Graphical monitoring of your JVM is very important in order to understand what is going on in your application. I'm looking forward to seeing how this project grows.

Wednesday, October 27, 2010

How to fix missing source for latest Java for Mac OS X 6 22

This will make clicking through to the JDK source work in Eclipse again after updating to the latest Java for Mac OS X.
  1. Go to http://connect.apple.com and download Java for Mac OS X 10.6 Update 3 Developer Package
  2. Install it.
  3. Open a Terminal.app window
  4. sudo -s
  5. cd /System/Library/Frameworks/JavaVM.framework/Home
  6. ln -s /Library/Java/JavaVirtualMachines/1.6.0_22-b04-307.jdk/Contents/Home/src.jar .
  7. ln -s /Library/Java/JavaVirtualMachines/1.6.0_22-b04-307.jdk/Contents/Home/docs.jar .

Monday, October 18, 2010

New 'Java Application Server'

After a few days trying out various technologies to see what I liked and what would work, I finally got what I consider a cool new basis for a Java application server built.
  • Based on top of Tomcat 6
  • Deployed as a single simple expanded war file with minimal xml configuration
  • Integrates:
    • Spring 3.0 - all annotation based
      • Method/Database level Transactions (@Transactional)
      • JPA Entity Persistence (Hibernate based) (@Entity)
      • EntityManager / Container managed Datasource via JNDI (@PersistenceContext)
      • Lifecycle (@PostConstruct/@PreDestroy)
      • Injection (@Inject/@Provides/@Bean)
      • JMX beans (@ManagedResource)
    • Guice -  still not sure about this route, may just go with Spring for it
      • Guice-Servlet
    • Hessian for remote method calls
The key feature of all of this is that it cold starts up (with a connection to the database) on my laptop in under 4 seconds. Comparably, JBoss 6m5 default config without even connecting to a database or deploying any war files starts up in 22 seconds.

Next up, I'm probably going to integrate RESTeasy for JAX-RS support. I don't quite need to serve up html pages, but I could always just use jsp's or HTMLeasy.

The idea is that this will make a really good platform for developing enterprise Java applications without having to be tied to the slowness of JBoss startup times (and the way that a new release always breaks something), commercial products like tcServer or the dreadful thought of using Oracle's Glassfish. Sorry, not a Larry fan. I also tried Resin 4.x, but after getting no response regarding the JMX support on the mailing list and the general bugginess of it, I gave up. I'm also not sure that their CDI implementation is all that solid because of the way they extend your bean classes at startup.

Monday, October 11, 2010

Apache Board Should Consider Dumping Java Development

I have been contributing to open source since 1993. I have been a server side Java developer since 1997. I was one of the primary people who started doing Java development within the Apache Software Foundation. I became one of the first members of the ASF based on contributions in Java. I was a party to the discussions which convinced Sun to open source Tomcat and Ant. I helped create the Jakarta project. I started or contributed to dozens of projects over many years. My code is used on servers all around the world. I continue to work with Java on a daily basis and have created many other open source projects on my own. I am a huge fan of Java for enterprise software, because I've personally seen it work exceptionally well.

Based on what I read today about IBM giving up the legal fight over the issues which have surrounded the JCP since day one, I feel that now is the time for the ASF to stand up and stop supporting Java development. The process that now surrounds Java does not reflect the open nature of the ASF.

In the past, the thought has been that as long as we are part of the (JCP) process, we can continue to exert our pressure on corporate forces to engage them and encourage them to embrace openness in what they do. Working from the inside gave us an advantage. In a lot of ways we were really successful and that help Java grow immensely as a language. I feel that door has now been closed and we've been left outside, only to get Sun burned.

This isn't about forcing projects to leave because of choice of language or being angry or being the loud voice in the room. It is simply about standing up for what you believe in and knowing when to throw in the towel. The ASF tried for many years to take a hard stance about what they believed in and failed. It is time to realize that and give up. It is hypocritical for the ASF to support a language and set of corporations which doesn't want or need its involvement anymore.

That said, I truly believe Java is not dead. Not for a long shot. Especially since there really isn't anything better to replace it for large scale enterprise applications. With IBM backing Oracle and Java7 and the legal roadblocks now removed, I think there is actually a bright future for it to grow. OpenJDK is technically open. Maybe not in the way the ASF would like, but the truth is that there is still billions of dollars being invested in this technology. I actually look forward to seeing what is in store next.

I know that some people within the ASF will worry that the foundation will become irrelevant without Java. It makes my heart flutter a tiny bit to think that I had a hand in creating something so huge. However, the irrelevance happened long ago for ASF's Java projects. The process for developing code within the ASF has become so inundated with bureaucracy that the fun has long since dried up. It is far easier to create your own project, get free hosting and check in whatever you want than it is to work within the ASF's walled off garden. As far as I can tell, the only projects that really thrive are those with corporate backing. At which point, the ASF is relegated to being a method to garner notoriety and free development resources.

Other communities have developed far better tools for supporting collaborative software than what the ASF can ever hope to provide. I suggest that current ASF Java projects be moved onto github or googlecode project hosting. These services provide a better environment in the long run anyway. Links should be made to the new project pages and the ASF should focus on development elsewhere.

In the end, I really feel that the ASF should put its weight behind languages that aspire to reflect the sense of openness that the ASF embodies. We tried for many years with Java, but the reality is that with IBM gone, there is nobody left to back us. The fight is over, move on.

Wednesday, September 15, 2010

reliable multicast memcached

PHP has a major issue with clustering sessions. Namely, reliability and scalability in a clustered environment. Network caching like this has been solved in Java land for ages now. ehcache+jgroups does a beautiful job of doing this. However, because PHP doesn't have a VM running all the time, you have to resort to outside solutions.

Thus, the only options I found are:

a) Store the sessions in a database which is really slow and a potential single point of failure.
b) Put them into memcached. This requires duplicating session data over the network to multiple memcached servers in case one of the servers goes down (repcached only allows replication between two servers).

The optimal solution is to use a reliable multicast publish/subscribe system where PHP would write out the data onto a multicast network where memcached is listening. There would be a memcached instance on the same server as PHP as well as several 'remote' memcached servers available on different hardware. If one memcached server goes down, PHP should just failover to another one in its configured list. All of the memcached servers would have the same exact data in their caches, but with multicast, only one message needs to go out on the network.

The issue with this is that multicast can be very hard to implement. Thankfully, the people over at zeromq have made this easy. Just a few lines of python and a big portion of the problem has been solved. The subscriber code below listens on the multicast network and stuffs data into memcached. Nice and simple. zeromq takes care of the reliability of multicast and makes it trivial to send messages from a publisher.

import time
import zmq
import memcache

def main ():
 ctx = zmq.Context(1)
 s = ctx.socket(zmq.SUB)
 mc = memcache.Client([''], debug=0)

  while True:
   msg = s.recv()
   print msg
   split = msg.split("|")
   mc.set(split[0], split[1])
 except KeyboardInterrupt:
if __name__ == "__main__":
    main ()

Obviously, there is more work to be done here, such as what happens when a memcached server is restarted (how do you populate its cache?), and I'll approach that soon. But, for a hacked together proof of concept, things are going pretty well so far. I've really missed hacking up prototypes like this.

Saturday, July 3, 2010

New project: Sweetened

I've created a new googlecode project called Sweetened.

If you use Ant to do your builds and Eclipse to do your development, you want this project.

Dear Maven and Ivy, you suck.

Saturday, May 29, 2010

how to solve the daylight savings time problem (cron)

As it stands now, daylight savings time (DST) is a royal pain in the ass. Spring forward, fall back. This reeks havoc on computer systems that expect time to always move in a forward direction. If your computer is on UTC/GMT time, then this isn't an issue because DST isn't even a concept... time keeps on ticking. However, if you are like me, living in PST/PDT, two times every year, my clock goes bonkers.

There are two sides to this equation. Spring and Fall. This year, for spring forward, the time between March 14th, 2010 @ 2:00:00am till 2:59:59am just does not exist. You go from 1:59:59 to suddenly being 3:00:00. Now, if you have a scheduled job (via a cron type system) that runs every day at 2:00am, then on March 14th, your job just won't run. It will pick up again on March 15th. Of course, this is a big problem for systems where a job needs to run every 24 hours. Generally, the solution to this is to fix the computers time in UTC/GMT so that DST rules just don't apply. However, at my work, we need to run our jobs in our client local timezone.

As I mentioned before, there is two sides to this equation. Now we have to consider the fall back problem. In that case, November, 7 2010 @ 1:00am happens two times. You go till 1:59:59 and then suddenly the clock jumps back to 1:00am. So, if you have a job that runs every day at 1:30am, then on November 7th, it will run two times. If it is a long running job that takes up a lot of system resources, that is not a good thing.

One obvious solution to this whole problem is to not run any cron jobs between 11pm and 4am. This covers most of the DST changes across the entire planet. However, if this ever changes and it changes a lot, then suddenly you might be running into this issue in some weird timezone you have never heard of before. It is also important to consider that those are prime times for execution of background jobs because generally the systems are more at rest in the off hours. I'd hate to give up that many hours in a day.

After much discussion with my coworkers, we finally came up with a fairly brilliant solution to both sides of this problem which I have not seen written up elsewhere. For both solutions, the trick is to look at the UTC offset. For PST->PDT spring forward, the offset grows from -8 to -7. If your cron is scheduled to run every day at 2am, then just move the cronjob and the next scheduled time to an hour later (4am is when things will run) when you notice the change in offset. Your job will run an hour late, but that is better than not at all.

For fall, things are a tiny bit more complicated because you probably want to run the second instance of 1am instead of the first. The reason is that you will have that extra hour worth of data to process. So, when the first 1am rolls around, always check to see if the next scheduled cron execution is a different (smaller, -7 to -8) UTC offset. If so, then don't do anything. This way, the execution will run correctly the next time.

I hope that clarifies things. Now you are as much of an expert in DST as I am.

Tuesday, March 30, 2010

Eclipse and Ant 1.8.0

How come I can't find anything on the intertubes about getting the latest Ant 1.8.0 to run natively under Eclipse? I know I can hack it in, but I'd like something more drop-in.

Monday, March 15, 2010

Review: Fulcrum Racing Zero 2-way fit wheels with Hutchinson Fusion 3 tires

Review: Fulcrum Racing Zero 2-way fit wheels with Hutchinson Fusion 3 tires

I pretty much expect everyone to be riding these by the end of the season. If not, you are really missing out. I cannot speak highly enough of these wheels. The tires corner like they are made out of sticky glue, they climb like they have been inflated with helium, they handle bumps like they are made of gel. I know it sounds like a lot in this economy, but for less than $1400 total including tires, I think this is the best upgrade you can possibly do to your bike. Tubeless is awesome!

For those of you who don't know, tubeless is the next generation technology for road wheels. It has been around for a while in the MTB scene, but is now moving over to the road crowd. It is the perfect upgrade path because you can still use your old clinchers/tubes on them. If you get a flat, pull out the valve stem and put a tube in it. If it is a small enough hole, you can also carry a bottle of tire sealant with you (or just put it in the tire in advance and you probably won't even know you have had a flat!). You can also attempt to super glue the tire too using the Hutchinson Rep'Air kit. Just pinch the tire at the hole, put the glue in the hole, let go of the pinch and let it dry. Thing is that you really won't get flats because you also run the tires at a lower pressure. I rode at 90psi for the last two days, one 20 mile hilly road race and a 50 mile hilly team ride. Oh, one more thing about lower pressure, without a tube, there is no possibility to get a pinch flat.

Putting the tires on is super simple. It is just like a clincher, but you don't have to deal with a stupid $5 tube. Think of the environment and the money you will save! The tires aren't even that hard to get over the rim, I did it with one plastic tire iron. After they are on, you just start to inflate them. Piece of cake.

The wheels themselves are bomber. The spokes are nice and thick, yet aerodynamic. The wheels can be trued and come with the tools to do it. They are also a cool red color and definitely stand out. The rims have had all the weight carefully removed from them, but the strength has been left in all the right places. They are also perfectly balanced with a bit of extra weight opposite the valve hole. The hubs are made out of carbon fiber and appear to be really well sealed against water/grime.

The Fusion 3 tires are brand new and the Fusion 2 have been discontinued. I was caught in the middle of the switchover so it took me a bit longer to get my F3's. This should be pretty much cleared up by now and dealers should have the F3's in stock. I'm not really sure what the difference is in the tires, but the F3's are all black and the F2's had grey stripes on the sides. I personally like the all black better.

I've looked at every single wheel manufacturer's website that I could find online (I'm OCD like that). These are the lightest tubeless wheels you can get today for the price and features (these also have the latest rage, ceramic bearings, whatever that is worth). They are lighter (1460g) than most of the carbon rim wheels I've found and you don't even get tubeless with those. Also consider that Fulcrum is a subsidiary company to Campagnolo, so you are getting a set of fine italian engineering wheels for like half the price.

Ship to a business address. I think it is slightly cheaper that way. When you make your order, send them an email with a copy of your drivers license and credit card that you used to make the purchase. Otherwise, it will delay your order cause they won't ship it until they have that information due to issues they have had with fraud. Much less expensive than competitivecyclist @ $1660!

http://worldclasscycles.com/clincher_tires_only.htm (fusion 3 tubeless) @ $50.50 /ea
For your first order, this place will only ship to the billing address. If you want less weight, go with the atom's, but you risk more flats cause they are a thinner tire. I'm more than happy with the f3's. You can pay $80 @ cc if you want... NOT!

I don't get any sort of deal with these companies, but please mention this blog post in the comments section of these websites so they know who sent you.

Wednesday, March 10, 2010

Code Bubbles: Java IDE Porn


This is going to revolutionize Java development much in the same way Eclipse and other modern IDE's did when they first came out. They based their IDE on top of Eclipse so you get all the benefits there as well. Yes, Eclipse has some warts, but for the most part it does the job well. This is another reason why strongly typed languages are great! There is no way to develop something like this otherwise.

Friday, February 12, 2010

AdPlanner: Stalking your competition

With all of the recent buzz over Buzz and privacy issues, I'm curious what you all think of a new service from Google called AdPlanner. Instead of people being worried about their privacy, now we get to have companies themselves worried about their competitive advantage.

A picture is worth a thousand words:



For all I know, the information in these images is generally regarded as confidential within companies. Now it is public knowledge for all to see.

Suppose you want to rank higher in ad placements than your competition, now you know exactly which keywords to bid more on. You get a nice list of sites that people also visit (and can get their keywords). You can also compare the amount of traffic your site receives against someone else. If their traffic is down and yours is up, you now know that they will most likely be trying new tactics to take traffic away from you. The daily unique visitors trend is also interesting in that around December, there is a clear traffic spike on these two major retailers for the holiday season.

Just wait, this gets better... if you look at the data for specific countries, you also get user profile data!


This is information that I know some sites pay a lot of money to consultants for. Thanks google for giving it away for free now!

Wednesday, January 27, 2010

Edacious: implementation of the withings scale api in java

The Withings Scale is a really cool product. It is a weight scale that has built in wifi so that when you weigh yourself, the information is automagically uploaded to their website where you can track your loss (or gain!) over time. It also tracks BMI, Fat Mass and Lean Mass. Sure, you could do this yourself with a vastly less expensive device, but the convenience of having something you just step on once in the morning for about 10 seconds is really cool. The scale is very well designed and looks nice in any room.

The best part of this device is that they provide geeks like me with a way to access the data you upload to their website. I'm working on a secret little project to make these numbers fun to play with. I needed a Java implementation of their API to access the data, so I wrote one and made it open source. Click the link above to buy the scale from amazon and keep your eyes open for the application I'm working on.

Monday, January 4, 2010

sardine - a very partial webdav client for java

For ages now, I've had the need for a webdav client for java that just implemented the features that I needed and wasn't an overcomplicated undocumented mess (cough jackrabbit cough).

Primarily, I just needed the ability to get directory listings, put files on the server and get an InputStream to a remote file. I also wanted it to use a recent version of Apache HttpComponents.

Since nothing like this exists, I created one... it is called Sardine. Hopefully someone else will find this smelly fish useful. Enjoy.

Friday, January 1, 2010

Which build tool to use...

This blog posting is mostly spot on.

I guess I need to try out Rake, but the idea of using Ruby to solve a Java problem, seems like a bad idea too. Ant, with all its warts, is still my preference after all these years.

My favorite quote from that posting:

"Maven builds are an infinite cycle of despair that will slowly drag you into the deepest, darkest pits of hell (where Maven itself was forged)."

I couldn't agree more with everything the author said. I've been there. If you are using Maven to do builds, you should have your head checked (or even better, commit privileges revoked).