My blog has been moved to http://www.javacraft.org/, on Blogger. I was waiting for the domain to become active to announce it.
This is the last post for my blog on WordPress.com. See you at Blogger’s. 🙂
Poch Apache Attitude Rant !
I’ve been banging my head on a few configuration problems lately setting up a proper development environment. During which I encountered and worked around some annoying bugs. Some of which in Maven.
I thought that in the brotherhood of open software development, it was regarded desirable to contribute. That’s how it stays alive. I also thought that reporting bugs was regarded as a valid way of contribution. I’m regularly doing this because I am way too busy to delve into the code of each tool I’m using. Certainly if there’s a workaround. But always do I report the problem in some way. To make at least the owner aware of the problem. And indirectly the users too as Google is indexing everything anyway.
I thought – naively it seems – it would be appreciated. Not so at Apache. Things seems to be different there. And unfortunately it’s not the first time I’m getting the this kind of replies from them.
https://jira.codehaus.org/browse/SUREFIRE-833 => Where’s the implementation for the change you propose ?
https://jira.codehaus.org/browse/SUREFIRE-832 => Log a different ticket for that last sentence. I’m too lazy to do it myself.
Hey guys – this is the open source community at work ! Not some brainless commercial corporate operation. If you like rules that much, go work for them. Or become one of their managers if you have such a disdain for what people do for your project.
Book Review: Programming Concurrency on the JVM
It’s been a long time since somebody wrote an in-depth book about programming concurrency for Java. To my knowledge it dates back to 2006; with the ground breaking “Concurrency in practice” by none less than Brian Goetz, Dough Lea, Joshua Blosh and a few other gentlemen. Even though not the object of this review, that is a book every Java developer has to read someday. Otherwise you can’t honestly claim you understand concurrency in Java. And read it again if you didn’t feel the blood drain from your face. When thinking about the consequences of what you read on your code at work. 😉
That being said, I’ve just read this book hoping it would be the next concurrency manual for Java. It turned out to be interesting lecture. But not as fundamental as the aforementioned book.
What this book isn’t
- An extensive explanation of the post-2006 concurrency API in Java. More precisely the package java.util.concurrent. It does scratch its surface though.
- A Java-only book. In this book Java is just one of the many JVM languages available today. Scala, Groovy, Clojure and JRuby are also covered. With Scala and Java as the main languages.
- A reference book about concurrency. The book explains the principles and does contain many code samples. After reading it all you’ll still need to do more reading- and experimenting before applying it all to real applications.
- An entry-level manual to concurrency. I would advise at least some Java programming experience before starting with this book.
What this book is
- An introduction-level description to the available tools in the JVM for concurrency and the JVM memory model. It is the first book I read that describes the memory barrier in understandable layman’s terms. Also JDK7’s Fork/Join is covered. So there’s a good point for Java amateurs.
- An extensive guide to practical STM using Clojure, Akka and Multiverse. All for Java, Scala, Groovy, JRuby and Clojure.
- An extensive guide to practical Actors using Akka. For the same JVM languages.
- A practical guide to mixing both.
A positive note this book sets is the fact that Java is not alone any more on the JVM. Other languages have arisen to a level they might just be useful for professional use. And because they all compile to JVM byte code, they are potentially able to share their unique capabilities with each other. The evident proof of this is that the main subject of this book, Akka, has been written in Scala. And yet, it can be used in your own favorite JVM language. Even if that isn’t Scala. Even if you don’t know Scala that well. Even if you’re only familiar with Java for that matter.
Maybe this book is at least beneficial for that: Even if some would argue Java slowed down to a snail’s pace, we can still write “rabbit” code in another language and integrate with Java. So personally, I don’t much mind the Java controversy any more. I won’t hide my personal preference for Scala and functional programming. And I’ll welcome the first occasion to demonstrate their usefulness for our profession. Also to non-Scala developers. Although I won’t argue Scala’s toolkit is far from being as robust and complete as Java’s.
Designing for concurrency
The book starts wit sections about concurrency. What it is, why you should matter and a few practical hints depending on what you want to run concurrently. Is it CPU intensive ? Then you better not have more threads than processors. Is it I/O intensive ? Then you can have more threads because some will be waiting for I/O to complete. Etc…
Personally, I think it is a good, common-sense, introduction to concurrency. And about what to take into account when using it. I haven’t yet seen much concurrency books doing this.
There are 3 solutions …
After the basic principles are clear, the author designates the evildoer: mutable state. These are the variables that can be changed by different threads. Whether this is concurrently or not. Because of the modern computer architecture, those threads might actually perceive a different value for the very same variable ! The problem is that different caches can cache that variable. And if you do nothing to “flush” those caches, different threads will see different values for the same variable. And that’s the first solution: synchronizing shared mutable state. And that means using the JVM’s tools. The author doesn’t hide it’s an art on its own. Because there are no fail-safes in the JVM. It will most certainly silently fail. And therefor it’s a heck of a problem to debug.
A less-know solution for the equation is to make the variables immutable. Apply shared immutable state. This avoids the dirty cache problem. Practically this means making them final. Unfortunately these are no build-in checks in the JVM to make sure you only share immutable objects. You might unknowingly share mutable state. Even when you really though you made everything final. So this is not so easy as it looks. Unless you rely on STM transaction managers which won’t allow you to change anything; unless you explicitly started a transaction first. It’s very comparable to transactions for databases. All changes occur or they don’t. Except that this all happens into memory.
(Personally I would advise to make everything final that can be. It avoids useless synchronization problems. And it’s an easy way to state your intention: This should not be modified. Applied to method arguments is avoids overwriting their values.)
The third way is the way of the Actor. Each thread has its own, isolated, state no other thread can access. Save for immutable message objects. Usually sent as the payload of asynchronous messages. That’s the solution of the isolated mutable state. If nobody else can access it, then there’s no need for synchronization. Right ? 🙂
Tackling Shared Mutable State
A practical use case is presented on how to use concurrency for computing the total size of directory. This is obviously a recursive algorithm that can benefit from concurrency. By having different threads scanning different sub-directories. But it also gives rise to the problem of shared mutable state when those threads are reporting their results. All these results need to be aggregated into a single result. Which represents the shared mutable state.
The tools available in the JVM are shown in different versions of the use case: Thread pools (because creating threads is expensive), Fork/Join, Locks, Latches, …
What I learned from this section:
- Concurrent collections are better than synchronized collections.
- The fork/join in JDK7 does work stealing. Resulting in better performance.
- Use the Exchanger class to share data between threads. I didn’t know about that class.
Software Transactional Memory
The author starts with an in-depth introduction to STM. And proves in practical ways this no mere theory any more.
For us, JVM developers, there are three ways:
- The Clojure language; which has the implicit restriction that everything is immutable. Unless it’s done during a transaction. Even though this is a different language, we can borrow this behavior in other JVM languages. Including Java.
- The Multiverse library. Again, because it’s all the same JVM family, it’s available to other languages too.
- The Akka library. With the same advantages.
The author finishes with a warning though: It’s not a magic bullet. It’s okay if you’re not frequently writing. Too many writes will negatively impact performance. Exponentially with the number of concurrent transactions. So beware ! 🙂
What I like with this approach is the build-in safety of not being able to actually do it wrong. And the ease of use of the libraries is demonstrated. And that the transaction manager will retry failed transactions. Something we can only dream of for our ORM libraries. Maybe some day Hibernate will ?
Actors
The second novel tool in the toolbox are actors. Comparable to the STM section, the author starts with an in-depth introduction of what it is. Followed by practical use cases of Akka in different languages. Also in Java. Which turned out to be a breeze really. Just fire messages away using non-blocking methods. And occasionally wait for a response. If that’s ever needed.
What’s particularly impressive is the typed actor. It’s really an actor in disguise. It’s used as a regular object while under the covers it is using asynchronous message exchange. (I guess some byte code weaving takes place there.) And if one machine is not enough, you have remote actors. The different Akka instances communicate with each other over the network with immutable, Serializable, messages.
Just like for STM, the author ends with a warning: It’s very efficient for “fire and forget” method invocation. Occasionally waiting for a reply is no problem either. But often waiting for replies is asking for trouble. And that’s not what it is meant for anyway. The Actor model is really about asynchronous data exchange.
Transactors
For those willing to push the envelope even more you’ve got transactors. Which are actors running in the context of a transaction. This becomes interesting when more than one actor is involved. Because in the end they need all to run using the same transaction. And the transaction must be rolled back if any of them fails.
Luckily Akka has a solution for this too. And as already demonstrated, it’s no big deal doing this.
Personal Conclusion
After reading this book, I think some people might consider its content too “experimental” or “novel”. To be immediately applicable in the Java/JVM enterprise world that is. I can not tell for me personally because I have no practical experience with STM or Actors. But now that I know the principles and the practical approaches to implement them, I’ll recognize the first occasion to give it at least a try. And if it performs as expected, I would certainly implement it.
I think, unfortunately, it will still take some time before either solution becomes mainstream. Simply because there’s no way to prove it does its job. Classic ORM tools rely on robust and proven transaction managers. Whose effectiveness are not being questioned any more. Even if all my tests prove STM works, how am I ever going to prove it will never fail in a production environment ?
That, my dear Watson, is the question…
If I were to give a score from 1 to 5, I would give it 3.5. Not 3, because it’s a good and informative book. But not a 4 either because this is the kind of book meant to make you aware of the existence of certain facts. If you want to apply them, you’ll have more reading and experimenting to do.
Book References
Programming Concurrency on the JVM: Mastering Synchronization, STM, and Actors
by Venkat Subramaniam
280 pages, 2011-09-02
ISBN:978-1934356760
Book’s homepage
http://pragprog.com/book/vspcon/programming-concurrency-on-the-jvm
JRebel Rocks !
I’m giving JRebel a spin. Man… this thing copes with any change you can image ! Before JRebel, I could occasionally hot-swap code in debug mode. But more often than not I had to stop, redeploy and restart. Now I’m restarting Tomcat no more. At all.
What’s original also is that this is the first software I see encouraging its user. 🙂
Devoxx 2011
This year I went to Devoxx with the firm intention to learn about the programming model of Android and JavaFX. And above all check what the HTML5 hype is all about. Especially since it becomes clear Java developers can’t just hide anymore from JavaScript and CSS. Denying your involvement into JS & CSS today would amount to telling you’re not interested in the ways your Java services are used. And you’ll simply be unable to solve bugs which flow brings you into the front-end code. Which today is heavily relying on JS. You can’t just refuse to use it because you don’t like it.
Besides this, I’m checking for the feasibility for these technologies for my employer’s products.
Personally, I don’t like JS. It’s just too chaotic – in my opinion – for reliably building complex application. And the fact it runs in the front-end means anybody can interfere in its execution. Hence messing with the data sent to the server. Which you therefore need to check again in the back-end code. Not to mention JS code is dynamic by its very own nature: Functions can be added anytime to objects. And its execution browser-dependent. Sometimes even the OS and hardware interfere there too. So really… 🙂 But there’s no denying it: it’s here to stay.
I avoided the talks about the current JDK & Oracle controversy. No need to see Mark Reinhold’s slides a third time about Jigsaw. Or hearing Brian Goetz telling how incredible SAM’s are. They all already said this at JavaOne 2011. For me personally, the dust still has to settle about the way things goes with Oracle now. Not the least because the Oracle speaker’s opinions are often very Oracle-minded. While in Sun’s days nobody was afraid to speak his/her own mind of their employer.
Anyway, I was not disappointed by Devoxx for my expectations. Also note that I’m expressing my personal opinions only.
Android
- Current release is 4.0.
- The API is a subset of the JDK’s API. With it’s own component model added – just like it is for Swing.
- The source code doesn’t compile to regular byte code. It compiles to executables for the Dalvik VM. Which is register-based instead of stack-based for the regular JVM. Allowing it to gain better performance. Although the application needs to be tuned towards the limited amount of resources the mobile device has.
- All major IDE provide support for developing Android application.
- There are emulators for local development.
- … much other cool stuff !!!
My notes can be downloaded here. Excuse the handwriting, I’m trying to write with a stylus to eliminate all paper notes. 🙂
Unfortunately I have to discard Android for coding at work. There is no reason to write an Android-only app. And even if it was, Android is nowhere near to the heavy graphic- and data volumes we need.
JavaFX
Just like Android, JavaFX is one of the main subjects of Devoxx this year. Being familiar with the previous releases I was wondering what they had done to make it more potable for Java. As this was the main objective of JavaFX 2.0: Port it from JavaFX script to Java. And maybe they have solved a number of deployment restrictions in the process. They didn’t…
- Current release is 2.0.
- It’s really a port of the JavaFX 1.x script API to Java. Luckily they kept the scene-graph model. Which allows really cool graphic stuff.
- The OS-dependent video-codec availability is still there.
- It’s hardware accelerated. On ATI, NVidia and Intel hardware that is.
- What’s new:
- WebView: Webkit HTML rendering. But what’s the point ? A browser does the same and doesn’t require additional software to be installed.
- ScalaFX & GroovyFX: The same API but written for Scala and Groovy. Which is much more concise than the Java API.
The JavaFX guru’s would probably refute, but for me personally I think it’s too little too late. By removing the JavaFX script language, this is now just the new Swing. (Although a very improved one !) No doubt this is a quantum leap forward for the looks of Java desktop applications. But …
- It won’t make Java desktop development any more popular. Seeing the capabilities of HTML5 and the landslide development pace it took. I think Java as of now really is a back-end language.
- Not to mention the restrictions for distribution and deployment of JavaFX applications. Which don’t apply at all for regular web applications.
- What about mobile devices ? Sun’s original plan was to port it to mobile and TV.
It’s a pity. Because I was very fond of the original JavaFX. But the current one has nothing to differentiate itself from the rest. It has impressive graphical capabilities. But who’s going to need them ? And to what extend are they different from HTML5’s WebGL ? It’s also hardware accelerated. I personally don’t get all the media attention Oracle puts into JavaFX. But I hope I’m wrong about it’s future. Still got a soft spot for it. 😉
My notes can be downloaded here.
HTML5
No doubt HTML5 was thé star of Devoxx 2011. Many talks were scheduled for the various aspects it covers. I was particularly attentive to them because it’s very probable I’ll be confronted with it very soon for my job.
It appears HTML5 is even moving faster than I originally thought. It’s not a hype; it’s a landslide ! Catching-up speed with every new browser release of the ongoing second browser war. I might be wrong, but from what I saw Google is the lead. What became also clear to me also is that JS libraries are now available in high numbers. Just like they are for Java. I interpret this as another sign that JS, CSS and HTML5 are taking over the desktop and browser. It’s just too popular to simply go away. Last week I knew JS was becoming ever more popular. But I had no idea it was to this extend.
Soon webGL is expected to become mainstream. Which should open the door for JS application to modern graphic capabilities. A domain that was reserved to desktop applications only.
HTML5 features among others:
- Easy 2D graphics.
- Multi-media support (without the codec restructions of JavaFX)
- Support for offline mode & local storage.
- Event listeners for various system & browser events (ex. whether the page is hidden).
- Integration with world-wide service providers according to their type (ex. sending email, processing pictures, …)
- Run code in background.
- Database + transactions support.
- Tabs.
- … much other cool stuff.
Again, I’m not happy with JS becoming so incredibly present in web development. But denying it would be plain stupid. If that’s the way it goes, that’s the way it is. Accomplished Java web developers have to know about JS/CSS/HTML5 very soon now.
My notes can be downloaded here. A book recommendation I had: Introducing HTML5, second edition.
Spring
I’ve heard about this one before and this was the occasion to learn about it. The basic idea is to write Spring Repositories (aka DAO) using only Java interfaces. No concrete class is being written !
The interfaces declare methods whose signature tells everything about the intention of it. JPA Repositories works according to a convention. If you adhere to that convention, it will generate your DAO implementations for you. In case you have complicated queries- and frequently reoccurring constraints, you can implement those as predicates. And reuse them in the methods.
Also support for auditing (who’s done what and when) and paging (query records batch by batch).
How cool is that !? 🙂
My notes can be downloaded here.
JVM / JDK
These were the sessions I attended. Apart from the last, they were disappointing. The first was an overview of generational garbage collection. And even that is not possible in a one hour time-span. The second appeared to be a – abbreviated – overview of Project Coin. Which no doubt everybody can find tenths of more complete sources in seconds. The last one, however, was worth it.
The file system API is located in java.nio.file. And basically the class Path replaces the good ‘ol File. Many operations you need Commons IO for are now native. File and stream copy- and move, to name a few.
What’s really new is the fact that finally the file attributes are also available. Permissions, owner, … And they’re updatable ! For performance reasons these can also be collected in bulk. And indeed, I often noticed File.isDir() on top of stack traces when profiling applications. I hope this is finished now.
What’s really cool is the notification of changes for directories and files. No need to poll any more.
And my favorite: Zip and jar files can now be used as file systems ! 🙂
An interesting blog entry about what’s store for the JDK.
I’ve got notes here, here and here. My hand writing is getting better, isn’t it ? 😛
Java Enterprise Edition
The Glassfish presentation was short, but informative, introduction into the versioning of applications. Basically it comes down to give a version number at deployment time. The previous version is disabled and makes places for the new version. Should the new version be broken is some way, you can revert back to the old one. There are various options to switch back and forth without losing a single session or request.
Doing this for a cluster is still manual though. Cluster-wide deployment would really be helpful though. 🙂
Currently just one version can be active at one time. There are plans though to have more than one. For instance to have the regular users run the old version and the final testers the new version. After which the old version is disabled and all traffic rerouted to the new version. But the active versions won’t be isolated from each other. They really act as concurrent applications to the same backend.
The JMS presentation was one of my biggest expectations. Finally one of the flagship JEE modules gets a workover. I have no doubt a lot of progress was made, but the presentation turned out to be a big disappointment. Lots and lots of explanations about what’s wrong with JMS 1.1, the challenges to define a new api, etc… I think the lesson is that the API will be much simpler (which was about time) when sending messages. And will rely on CDI for injecting the various related objects. It will be released with JEE 7, Q3 of 2012. I’ll check on it then – IF I need JMS that is !
That’s what you get for letting managers present technical subjects. Even if they are the spec leads. Pity…
It tuned out that the Infinispan presentation was one of the best I saw this year. I’ve heard about Infinispan before, but never had the occasion to check on it. Until now.
Infinispan is both a caching- and datagrid solution. Depends on what you use it for. You can use it as a 2nd-level cache for Hibernate. For which is seems to be able to do everything besides cooking your meal. 🙂 It can be embedded or distributed. In both cases it can be transactional. Monitored by JMX if needed. Provides cache statistics. Overflow to a persistent store. Event listeners supported. Preloading of caches. Encryption. Authentication. Etc…
In case of distributed caching, it relies on JGroups for communication. So basically, if there’s a wire between the machines, it will work.
What’s neat is that the cache synchronization in a distributed case can happen according to various strategies:
- Replication: Any change is propagated to the other nodes. Exponentially expensive with the number of nodes.
- Distributed: The propagation is done according to a hash function which designates to which nodes to propagate. Less expensive than replication.
- Invalidation: The cache entries are not propagated. Only their keys. Which are used by the other nodes to remove that particular entry from their own caches. The least expensive.
In case of a datagrid solution you add- and query data to/from the cache. And if the need comes to it the contents of the caches is persisted into cache stores. Which can be files, cloud, a JDBC database, Berkeley DB (OODB) and any store of your choice if you implement the SPI.
I definitely give Infinispan a try. Usually I work with EHCache; which is simple and quick. But doesn’t support transactions.
I’ve got notes here, here and here.
JVM Languages
Being a Scala enthusiast myself, I couldn’t miss the meeting of Europe’s Scala amateurs. It seems we all have the same problem: We would like to introduce it at work, but we can’t convince anybody to start with it. 🙂 England’s the best country for Scala jobs though.
Matt Raible gave a very entertaining talk about his quest for convincing the organisation of Devoxx to bring him to Belgium. To drink beer that is. So the theme of his presentation is applying Scala technology as a means to that end. Next to JavaPosse, he’s the most entertaining and captive speaker I’ve ever seen. 🙂
More precisely stacking a whole lot of successive products: Play, Coffeescript, PhoneGap, Jade, HTML5, JS2Coffee, Less and Anorm. Just to name a few. Because he’s not only entertaining; he’s also bloody fast ! Usually there was a fast-paced introduction to a product which ended with “… and I blogged about it”.
It’s so fast, that I’m still trying to cope with the implications with what I saw:
- There is no need to develop in JS. Coffeescript has a more robust- and concise notation.
- JS2Coffee is an online tool to convert JS into Coffeescript. Seems to work okay.
- Play is an innovative Scala web framework which allows quick development cycles. Everything you need to know is shown in the browser. If it works, you get the expected result. Otherwise you’ll get an explanation of to why it didn’t.
- No need to spell out HTML when using the Jade template engine.
- Anorm is the “Hibernate” of Scala.
- Learning Lift can wait.
At the end Matt shows a video he shot during the development process. Grinning his head off. No doubt because he can finally hear the soundtrack on full theater strength. I really advise to check his video on Youtube. His slides are on Slideshare. Other speakers left their slides there too btw.
Earlier this year Jetbrains announced the imminent release of their own JVM language named Kotlin. And now, there it is ! I attended because of the reputation of Jetbrains to make software that matters. Their language has many things good. But really, it’s too late. However good it is, are we going to learn just another language ? If they had released two years earlier, I think I might have ditched Scala for it. It’s concise and has implicit robust design support. Maybe it will get more coverage later. But I’ll be very surprised they ever reach Scala’s current support level.
Sorry Jetbrains… I am not going to learn a second just-for-fun language.
Cracking Clojure is one I attended out of curiosity. I’ve read two or three things about it. Knowing it’s LISP’s spiritual offspring only made me more curious about it.
I held firm courageously for the first 15 minutes. When the examples became longer and longer I drowned in the flood of parentheses. I have no doubt Clojure has a niche application market. But I’m still searching for it. It’s certainly not business. It might be maths or technology. But even there, I’m not quite sure. What’s intriguing is the fact it considers code as data. And inverse. Functions are lists of statements. And statements are lists of arguments with a name. Etc… A Clojure application seems to be a tree-structure which nodes are sequences of … things. Although I won’t stake my reputation on it. 😛
Clojure is a curious beast which usefulness still eludes me. And especially because it does, I’m decided to find it out ! 🙂
I’ve got notes here, here and here. The slides on Slideshare.
Jackpot
This one was attended because the title appealed to me. Not for the first time did I which for a tool a can do application-wide source code refactoring. Without doing it manually file by file.
Jackpot is an old Sun project revived in Netbeans 7.1 in the form of source code hints. The IDE will advise of a certain (potential) problem. That’s one. And the cure(s) to solve it. That’s two. What modern IDE do today.
The difference is that Jackpot is custom-configurable. It has an extensive search & replace DSL to select parts of Java source code satisfying certain criteria (the problem). And when found, the possible code it can be replaced with (the cure). And what’s really making this a live-saver is that the IDE can be instructed to apply this in bulk; using the new Inspect and Transform functionality of Netbeans 7.1.
Yeap. Netbeans got even better !
I’ve got notes here.
UI Design
Joe Nuxoll presented his UI design wisdom in a more than crammed full room. Being involved in visual prototyping myself and a BIG fan of the JavaPosse, I just could miss the occasion.
It appears design is nothing rational. It’s all about taste; and therefore directly about the mood of the people who decide what’s looking good. Often communicating with them is the key.
The remainder of the talk was about “golden eggs” – concrete advise of do’s and dont’s. Even if instinctively you know them make sense, it’s good to see them confirmed. I’ve written them down into the notes.
I’ve got notes here. The writing’s terrible – I forgot my stylus in the car. 🙂
Two book recommendations:
- A book of James Garratt. Since there’s only one book, I assume it’s this one.
- Don’t make me think, by Steve Krug.
JSR-107 Java Caching
A talk presented by one of Terracotta‘s guy’s. Judging by his accent I would guess he traveled quite a distance to be here. From down below ? 🙂
JSR-107 is not the first attempt to standardize caching. But it’s probably the first one which will actually deliver. There’s a specification that’s currently up to its 0.4 version. The key concepts are almost identical to those of EHCache – one of Terracotta’s products:
CacheManager -> Cache -> Entries -> Keys & Values.
Unfortunately this talk was a bit too EHCache-minded in my opinion. Anyway, I’ve heard nothing new.
Except for one gem: The annotations allow – finally – for memoizable methods. Meaning by simply annotating a method in some way you can tell the method should be executed just once. The next times the returned result will come out of the cache. I’m usually pulling this off with a custom annotation, AOP and EHCache.
I’ve got notes here.
My Devoxx 2011 action list
- Read the HTML5 recommended book.
- Give Play a spin. It is quite different from servlet’s & co and it just might be the leverage to introduce Scala at work.
- Port code at work to Infinispan.
- Give Spring JPA repositories a try at work. If this works, it will greatly help adoption of JPA.
- Read the design recommended book.
- Try to find a concrete, unforced and naturally fitting use case of Clojure.
Conclusion
- As ever Devoxx was great. And was finished way too fast ! 🙂
- The wristband is an understandable attempt to easily identify the participants. But really… it’s sticky and irritating. I’m still feeling it. I wouldn’t be surprised if some people complain about rashes. 🙂
- The WIFI was less good than last year, but still well enough to make Devoxx more enjoyable even.
- Let’s not talk about the coffee, shall we ? But the HTML5 coffee mug is great ! 🙂
- I’ll be there next year !!!!
PS. Does anybody in here knows a nice, old-looking, WordPress theme ? I’ve tried loads…
Silly me … :-)
An application with compilation errors can run in Netbeans. As long as the faulty classes are not used I presume.
In that effect Netbeans asks you to run anyway.
Don't check the checkbox !!!
Maybe this dialog gets annoying when this happens frequently. You might get tempted to check “Always run without asking”. Be very careful with that because there’s nowhere else to uncheck it. I might be wrong, but I have the impression this sometimes also messes up other classes too. I frequently had to clean the Maven project before running a test all over again.
If by any change you happen to have checked the checkbox of the dialog, you’ll have to delete a file first. And for good measure quit Netbeans before that. If you remove the file
.netbeans/7.0/config/Preferences/org/netbeans/modules/java/source/BuildArtifactMapperImpl.properties
Why didn’t I thought about that earlier ? SILLY ME ! 🙂
Distributed EHCache Setup
I’m in the middle of troubleshooting a weird problem in a clustered web application. Hibernate does – sometimes – not persist changes to the database. EHCache is used as the second level cache; configured with automatic multicast discovery over RMI.
As I don’t have the same configuration available for testing I’m doing the next best thing: Setting up a number of VM’s (with Virtualbox) locally. If I can reproduce the error, I’m hopeful I’ll be able to find the cause.
Everything is set up now, but EHCache was particularly difficult to configure. I’m putting it here so I never have to search for it again. And hopefully somebody will benefit from it too.
Automatic peer discovery
The peers, or nodes of the cluster, need to know about each other’s existence before sharing anything with each other. This is done by using a multicast address every peer should listen to. There is a regular poll to know who’s there and who’s not responding any more. Those responding for the first time are added to each peer’s known list of peers. Those not replying any more are removed.
I chose for automatic peer discovery because it offers interesting advantages:
- Easy configuration: The configuration is identical for every peer/node in the cluster. As opposed to manual discovery, each peer has a unique configuration. It has to list every other peer.
- Scalable: Any number of peers can be added anytime. If adding more servers solves your performance problems, then this is the way to go. There’ll be more network traffic however.
Example configuration
<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="peerDiscovery=automatic
,multicastGroupAddress=224.20.20.20
,multicastGroupPort=4444"/>
No surprises here. Adding this fragment into the ehcache.xml files tell the applications to discover the others by means of the multicast address 224.20.20.20 on port 4444.
Enable multicast
Where it got surprisingly difficult, is to enable the actual multicast at OS level. It appears for Ubuntu this is not fully enabled by default. A number of settings need to be changed:
(1) Add, or uncomment, these lines in /etc/sysctl.conf
-
net.ipv4.icmp_echo_ignore_broadcasts=false -
net.ipv4.ip_forward=1
(2) Lookup the network device used for the multicast traffic by typing ifconfig. You’ll get a list of devices with their names left. Typically eth[0-9]. Remember the name of the device you’ll want to use.
(3) Add this line to /etc/network/interfaces
-
up route add -net 224.0.0.0/4 mask 240.0.0.0 dev eth[0-9]
(4) Reboot. There’s probably another way to do this without rebooting. But personally I want to see the result when the server restarts.
(5) Check the network settings
- Type route -v. There should be a 224.0.0.0 route related to eth[0-9].
- Type cat /proc/sys/net/ipv4/ip_forward. This must return 1.
- Type cat /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts. This should return 0.
Cache synchronization protocol
The next item is to tell EHCache what to tell the other peers when one of its own caches is being modified. Surprisingly this is not done by multicast. But by sequentially notifying the peers by unicast. With “sequentially”, I mean one peer after the other. Not all at once by multi-threading. So there’s a real danger of keeping peers out of the loop when an exception occurs during the synchronization.
The synchronization with the other peers can be done in a number of ways for a number of events (add, remove and updating cache entries). There’s the choice of (not) notifying peers for a certain event. And there’s the choice of (not) sending over the complete cache entry over the wire.
What’s really cool is when entities are sent over the wire, peers can have a cache hits on something they didn’t get themselves from a database. The downside, however, is that it costs quite a lot of effort to serialize, send and unserialize the cache entries.
So, personally, I’ve opted for synchronizing only the remove and update events. And let the peers invalidate those entries in their own caches. That’s a lot faster because only two events are used and only the identifiers of the cache entries are sent over the wire. The downside is that the peers will have to access the database to update their caches. And from a monitoring perspective, when checking the cache state, none of the peers have the same amount of entries in the caches. Making it difficult to check whether the synchronization actually works.
Setting up the actual synchronization requires an additional element in ehcache.xml
<cacheManagerPeerListenerFactory class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory" properties="port=30000,socketTimeoutMillis=10000" propertySeparator=","/>
This element means the peers will all listen on port 30000 for cache synchronization from other peers. That’s for getting synchronization data from other peers.
For the actual sending of synchronization data, each <cache ….> element must be provided with a listener. This listener will listen to the required events and send the synchronization data to the other peers.
<cacheEventListenerFactory
class="net.sf.ehcache.distribution.RMICacheReplicatorFactory"
properties="replicateAsynchronously=true,replicatePuts=false
,replicateUpdates=true,replicateUpdatesViaCopy=false
,replicateRemovals=true"
propertySeparator=","/>
This configuration means that
- Synchronization will happen asynchronously (replicateAsynchronously=true). Meaning the application won’t wait for all peers to acknowledge for receiving the data. Whether you do it (a)synchronously entirely depends on the nature of the application. If it can be done in background, then do it. Otherwise make it synchronous. But that’s going to cost performance too.
- Only cache entry identifiers are shared (replicateUpdatesViaCopy=false). This just means that other peers will invalidate the corresponding cache entries.
- Adding an entry is not synchronized (replicatePuts=false). Since the principle of the game is to invalidate the other peer’s caches, it makes no sense in telling the others a new entry has been added.
- Modifying and removing an entry is synchronized (replicateUpdates=true and replicateRemovals=true)
Set RMI host address
The cache synchronization relies on the RMI protocol. And although nodes of a cluster are typically located on the very same network, I’ve had problems let the nodes connect to each other over RMI. I’ve solved it by adding a property to the startup command
-Djava.rmi.server.hostname=<ip current host>
Usually this is due to an exotic network hardware where the original ip address gets lost. Just add it for good sake of measure. At least you won’t be losing time on this one.
Enable JMX
JMX is fantastic for monitoring the state of servers. Of what’s actually going on inside remote JVM’s. By default, however, a JVM won’t allow a remote application to connect with JMX. It must get a number of properties at startup first. These are typically
- –Dcom.sun.management.jmxremote (enable JMX for remote connections)
- -Dcom.sun.management.jmxremote. port=<port> (port for the remote connections)
- -Dcom.sun.management.jmxremote.ssl=false (do not encrypt the connections)
- -Dcom.sun.management.jmxremote.authenticate=false (don’t ask for user/password)
As JMX relies on the RMI protocol, it is subject to its limitations. One I typically encounter is that the connections can’t be made due to the current networking magic of routing and firewalls. Typically I can connect as soon as I’m adding a fifth property to the command line
- -Djava.rmi.server.hostname=<ip current host>
This property tells the remove connections to use the mentioned ip address for their connections. Apparently this sometimes gets lost when there’s too much networking magic going on between you and the server.
Checking it actually works
Now it’s time to start the application on all the nodes of the server. As the distribute caching seems quite error prone, it’s always a good sign when everything starts without an exception.
There are various ways to check whether it all – finally – works
Ping the multicast address
Pinging the multicast address is an easy way to check whether everything works at OS-level. Run the command ping 224.20.20.20 (as configured previously). There must be a reply from all the server in the cluster. If that’s not the case, check the network configuration of those not replying.
Now, do the same on the command line of every server in the cluster. There should be a reply from the other servers. You need to be absolutely sure everyone can see the others.
Check who’s listening the unicast port
As mentioned earlier, each of the nodes get their synchronization data via unicast. This means they all have to listen to the configured port (30000 in the example). This can be done by running the command telnet <node host name> <port>. If somebody is answering, that’s okay then.
Check with VisualVM
VisualVM is an incredibly handy tool shipped with the JDK. It’s a JMX client on steroids. The successor of jConsole. EHCache registeres its cache manager with the local JMX server (already included in the JVM). This allows tools like VisualVM to check the state of the caches. Assuming $HOME/bin is in the path, type jvisualvm to start it.
What I am typically doing is to chose a cache and double click on its number of elements statistic. VisualVM will show you a running graph of this statistic. I’m doing this for the same cache on every node.
When you have a graph for each node, ask for the cache manager on a random node to clear all its caches. If you’ve done it well, the graphs for all the nodes will plunge to zero. Repeat this for each node.
What about TerraCotta ?
EHCache promotes the usage of TerraCotta for the cache synchronization. The setup requires almost no configuration of the application. If I’m not mistaken, adding a single element in ehcache.xml is all it takes. I’ve tried it. And it works as advertised.
In my case, however, there were fundamental considerations that made us not using it after all
- The included RMI synchronization as described here works for us too. Since TerraCotta is not particularly cheap, I couldn’t sell it to my customer anyway.
- TerraCotta is a server product. All the nodes connect with it to exchange their synchronization data. This requires the purchase of additional hardware. With all the required maintenance overhead being added too. Again, being more expensive.
- Being a server, this makes it a single point of failure for cache synchronization. Data integrity can quickly be jeopardized if this server breaks down. Because the nodes’ caches will more and more divert. The same problem applies for regular maintenance by the way. To cope with this you can add a fallback TerraCotta server. As mentioned earlier however, this doesn’t help to reduce costs and maintenance.
Personally, I think cache synchronization must be a distributed, dynamic, effort. Each node should bear its part of the burden. The RMI solution is way more elegant and robust compared to TerraCotta’s.
There is no single point of failure. You can remove and add nodes to the cluster anytime you want. There’s no harm done if one of the servers goes crashing. The fallback is build in into the architecture. Uniform administration and monitoring. And the hardware is limited to the cluster node only.
I can imagine however TerraCotta scales better when there is a huge amount of synchronization involved. Which wasn’t the case for my current project.
Was it any useful ?
Please, let me know if this article/text/… was any useful to you ! I’m always curious if I managed to help further a fellow software engineer. 🙂
Resources
JMX home page Everything you want to know about JMX is here.
Prioritized messaging with ActiveMQ/Spring
Recently I have been faced to a performance problem related multiple JMS consumers . Prioritizing the consumption was the applied solution. I thought the public might profit from this. Especially because making it work was much harder than implementing it. 🙂
Problem Context
The context is that there are different types of messages, say Foo and Bar. The consumption logic of Foo is quick and isolated. The consumption logic of Bar however is extensive and needs a number of Foo messages to be consumed prior to starting its own logic. This worked previously. When there was just one consumer : everything was sent in the right order. With multiple consumers, however, the order is not strict any more. To cope with this, the consumption logic of Bar does a repetitive sleep/polling to check whether all Foo’s it needs are consumed.
This sleep/polling is the source of a performance problem : The more Bar consumers are waiting, the less consumer threads are free for consuming Foo messages. Resulting in even more Bar-threads waiting. Etc… Usually resulting into one Foo consumer with all the others polling/sleeping.
Knowing the JMS specification, it would be a piece of cake to break this vicious circle by giving the Foo messages a higher priority than the Bar messages. As such, in theory, consumers would first all be busy with Foo messages. And once these were removed from the queue, they can start with Bar messages. And logically, none would go into sleep/poll mode. Because all the Foo messages would have been consumed by then.
Enable multiple consumers
For consumption we were using the JmsTemplate. That’s being replaced with the very versatile DefaultMessageListenerContainer. My main reason to do so was to be able to set at runtime the number of consumer threads via JMX. Also, in case of unforeseen problems in the production environment, I would be able to revert to a single-threaded consumer state w/o restarting/redeploying/…
Our JMS consumption configuration in Spring. What it was before :
<bean id="jmsFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="${settings.broker.url}"/>
</bean>
<bean id="jmsQueue" class="org.apache.activemq.command.ActiveMQQueue">
<constructor-arg index="0" value="jmsQueue"/>
</bean>
<bean name="jmsConsumerConnectionFactory" class="org.springframework.jms.connection.SingleConnectionFactory">
<property name="reconnectOnException" value="true"/>
<property name="targetConnectionFactory">
<ref local="jmsFactory"/>
</property>
<property name="clientId" ref="jmsClientIdFactoryBean"/>
</bean>
<bean id="jmsConsumerTemplate" class="org.springframework.jms.core.JmsTemplate">
<property name="connectionFactory" ref="jmsConsumerConnectionFactory"/>
</bean>
<bean id="jmsConsumer" class="org.foo.JmsConsumer">
<property name="template" ref="fastJmsConsumerTemplate"/>
<property name="destination" ref="fastJmsQueue"/>
</bean>
And what it is now:
<bean id="jmsFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL" value="${settings.broker.url}"/>
</bean>
<bean id="jmsQueue" class="org.apache.activemq.command.ActiveMQQueue">
<constructor-arg index="0" value="jmsQueue"/>
</bean>
<bean id="listenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
<property name="clientId" value="consumer"/>
<property name="concurrentConsumers" value="2"/>
<property name="maxConcurrentConsumers" value="5"/>
<property name="connectionFactory" ref="jmsFactory"/>
<property name="destination" ref="jmsQueue"/>
<property name="messageListener" ref="jmsConsumer"/>
<property name="receiveTimeout" value="60000"/>
<property name="idleConsumerLimit" value="1"/>
</bean>
<bean id="jmsConsumer" class="org.foo.JmsConsumer"/>
This setup allows a multi-threaded consumption of the queue in question. In our case, it’s even a multi-server environment. So each JVM will start at minimum the number of consumer threads (property concurrency) but won’t exceed the set maximum of consumers (property maxConcurrentConsumer). It will even ramp up/down these threads whenever the message volume requires it.
Client id uniqueness
As soon as more than one consumer kicks in, you’ll the enigmatic error message that host <your host name> is already connected to the queue. And that it will attempt to reconnect when refreshing the connection. And blablabla… It appeared the client id has to be unique for the whole setup. In the previous example I set this to “consumer”. Only the first JVM will make a successful connection. The others will be bounced because they “were already connected”. In fact they were bounced because another JVM with the same client id was already connected.
Because I don’t want to introduce server-dependent properties in the Spring configuration, I extended the listener container class to prepend the host name to the passed client id:
public class JmsListenerContainer extends DefaultMessageListenerContainer {
@Override
public void setClientId(String clientId) {
try {
final String name = InetAddress.getLocalHost().getHostName();
super.setClientId(name + "-" + clientId);
} catch (UnknownHostException e) {
super.setClientId("" + System.currentTimeMillis() + "-" + clientId);
}
}
}
And reflected into the Spring configuration:
<bean id="consumerContainer" class="org.foo.JmsListenerContainer"> <property name="clientId" value="consumer"/> <property name="concurrentConsumers" value="2"/> <property name="maxConcurrentConsumers" value="5"/> <property name="connectionFactory" ref="jmsFactory"/> <property name="destination" ref="jmsQueue"/> <property name="messageListener" ref="jmsConsumer"/> <property name="receiveTimeout" value="60000"/> <property name="idleConsumerLimit" value="1"/> </bean>
Upgrade to Spring 3.0.5
Once all JVM were able to connect to the queue I was observing erratic message consumption behavior. Sometimes the threads stopped consuming. And neither did the listener container ramp down the consumer threads.
This appeared to be a bug in Spring. Upgrading to the latest release at that time indeed solved it. The threads are behaving the expected way.
Set the property JMSPriority
JMS messages provide a property defining the message priority. Randing from 0 (low) to 9 (high). For ActiveMQ this is “4” by default. The priority can be set using the method javax.jms.Message.setJmsPriority(int). The Bar messages got 0 (low) and the Foo get 9 (highest).
When running the whole config however, I still see this vicious circle happening : more and more consumer threads are blocked because they’re waiting for their Foo messages to be consumed.
So, what’s going on here ?
Poking around in the Spring source code, it seems the priority in the message is ignored. You have to set it in the Jms template. And even then, you have to enable something called explicitQos. Otherwise the template’s priority is also ignored.
So, this makes the queue definition, producer-side, looking like this:
<bean id="jmsProducerTemplate" class="org.springframework.jms.core.JmsTemplate"> <property name="connectionFactory" ref="connectionFactory" /> <property name="explicitQosEnabled" value="true" /> </bean>
Make the queue prioritized
Once the priority problem out of the way, the dreaded vicious circle was still happening ! Even when debugging the message production process, I could see the priority being set. And even when browsing the queue’s content with VisualVM (great tool !) I could see the priority has been set correctly.
This time the possible culprit was the fact the queue has not been set to prioritized. Something that is available since ActiveMQ 5.4.0. We were running 5.2.0 and the queue was not set prioritized.
So, I’ve upgraded to ActiveMQ 5.4.0 and modified the server-side definitions as follows:
<amq:broker id="broker" useJmx="true" persistent="false" useShutdownHook="true">
<amq:destinationPolicy>
<amq:policyMap>
<amq:policyEntries>
<amq:policyEntry queue="fastJmsQueue" prioritizedMessages="true" maxBrowsePageSize="30000"/>
</amq:policyEntries>
</amq:policyMap>
</amq:destinationPolicy>
<amq:transportConnectors>
<amq:transportConnector uri="${settings.watchjobs.broker.url}" />
</amq:transportConnectors>
</amq:broker>
By the way, the property maxBrowsePageSize is set to a big value because I’m browsing the queue for its content. To make sure the Foo messages go first. By default this is set to 400. So when browsing the queue you’ll only get a sample – the 400 first message in the queue.
Mind you, there’s a caveat here : The elements of amq:broker must be in alphabetical order !!! The element amq:transportConnectors must be after amq:destinationPolicy. Otherwise you’ll get a funny xml validation error when the Spring container is initialized.
Running the setup again I could see in VisualVM the queue has finally set its property prioritizedMessages to true.
Upgrade to ActiveMQ 5.5.0
Running the test case again, however, still does show the vicious circle issue ! IntelliJ gave the hint of what was wrong: The prioritizedMessages property in the Spring configuration was shown in red. Apparently the xsd definition for ActiveMQ 5.4 does not specify this property. But it does once you upgrade to ActiveMQ 5.5.0. So, that I did.
And behold, finally, everything started to make sense ! Foo messages are consumed before Bar messages ! Even if the Foo messages were enqueued after the Bar messages. 🙂
Conclusions
- Use at least Spring 3.0.5 and ActiveMQ 5.5.0 releases.
- All JVM’s must have unique client id’s.
- The JMS priority is handled by the Jms template. Not by the message priority.
- In the first stages make extensive use of tools like VisualVM to
- Monitor the activity of the consumer threads. The expected amount ? Are they doing something ?
- Check the prioritized setting of the queue. It must be “true”.
- Check the contents of the queue. The priority must be set as expected.
I hope this has been any useful to somebody some time…
Local Tomcat memory profiling with VisualVM
I’m often using the excellent VisualVM tools to profile applications. It’s free (as in beer), delivered with the JDK and packed with enough tools to solve 99% of the performance problems. I have to give that credit to the authors they did an excellent job !
Today I tried to set up for a memory profiling of a Tomcat/Spring/Hibernate stack. The Tomcat instances didn’t show up in VisualVM’s "local" node. No problem. I’ll setup JMX to operate remotely and I’ll do it that way. Adding the properties to Tomcat’s …
-Dcom.sun.management.jmxremote.port=3333
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote=true
… and restarted. Now I can connect to the Tomcat instance using port 3333.
I’m most often doing the profiling using the Sampler plugin. It’s fast and accurate enough for most profiling cases. While the "real" Profiler tool does code weaving to achieve a higher precision. (I guess.) And it takes a lot of time to patch a Tomcat with its applications deployed on.
The next problem is that the Sampler does not want to allow memory profiling for "remotely" connected JVM’s. CPU profiling is allowed however. I could do memory profiling with the JVM’s in the "local" node of VisualVM. So I have to make Tomcat visible again to VisualVM. I know that once worked !
In the end the solution is very straightforward: Set the environment variable CATALINA_TMPDIR to "/tmp". At least to the same value as the one of VisualVM. Once this is done, Tomcat pops-up again in VisualVM’s "local" node and you can do memory profiling using the Sampler plugin. I have no rational explanation (neither irrational for that matter) for this problem. At least now it works.
It seems to be an old problem however.
To make absolutely sure I’m not ever confronted with this I have put "export CATALINA_TMPDIR=/tmp" into a script in /etc/profile.d. Even if I re-install Tomcat, it will always work.
Hope this helps somebody too.
Eradicate MySQL on Linux
I just blew up my local MySQL database on Ubuntu 10.10 because the max size for the innodb log files was too small. Unfortunately it was beyond recovery. It couldn’t be restarted with the services (ie. service start mysql). Neither was the manual setup successful; sudo mysqld_safe kept spawing new processes.
What the casual Linux user would think about is to remove- and install the MySQL package with sudo apt-get remove mysql-server followed by sudo-apt install mysql-server. But then it’ll strike that the install procedure is only asking once for the password of root. The usual procedure asks three times for the password (for a blank password ?). Unfortunately the reinstall doesn’t work that way. Directories previously owned by mysql were now by root, weird behavior, etc… I had this problem before and got to reinstall Linux to get rid of it.
Today, however, I stumbled on this article and found the way to reinstall MySQL without rebooting. I’m quickly writing it down so I know how to do it the next time.
- sudo apt-get remove mysql-server (removes the package and its dependencies)
- sudo dpkg -l | grep mysql (lists packages which names contain mysql)
- For each package found sudo apt-get purge <package name>
- sudo dpkg -l | grep mysql (empty list)
- sudo apt-get install mysql-server (asks three times the password for root)
Everything seems to work as before.
Java Artisan's Blog ... 