Bratislava World Usability Day 2016 and government IT

I wrote about sustainability and design takeaways from Bratislava World Usability Day 2016 in my previous post. World Usability Day (WUD) 2016 was organized on November 10th, 2016 in many places around the world. Theme for this year was Sustainability, but for us, working with and for the public sector, it was even more attractive thanks to the guest from UK and Estonia government agencies that implement or oversee the government services – services for real people, citizens. Services that should serve – just like the state itself should. And that is very touchy topic here in Slovakia.

Videos from Bratislava event can be found here, the page is in Slovak, but videos are easy to find and are in English.

Estonia: pathfinder or e-Narnia?

Risto Hinno came to us from Estonia, the state renown for it’s level of e-government. But if you imagined their world as a perfect place with flawless services you’d be wrong. Risto came to talk about their approach to the services and the problems they had to overcome and are overcoming.

Estonia and Slovakia are both countries from the Eastern Bloc, Slovakia is the successor of Czechoslovakia, while Estonia is one of the post-Soviet states. Both states are in NATO and EU and both use Euro, but there are also some important differences. I may not be historically accurate, but while in Slovakia we still have plenty of “old guard” people in their posts (like judges) and plenty of old-thinking politicians, many of them previously members of Communist party, now often using the sticker saying “social democrat”. In Estonia most of these were Russians and they simply were gone after Estonia became independent. And that allowed for deeper change, change that is much needed here in Slovakia but haven’t happened. Some ask: “Will it ever?”

But back to the services. As Risto put it, what we (citizens) want is value, but what we typically get is bureaucracy. The answer to this problem is to make everything smaller and simpler and really focus on the value.

Problems small and big

But just as with value-vs-bureaucracy problem there are opposite forces in play here. Even when the stakeholders agree on delivering maximum value for the money they often don’t agree on how to do it.

Very often the expectations are big and the budget follows them. Very often we don’t respect the systems our users already work with. And very often we deliver little value for a lot of money afterwards. Or worse, we often make the life of our users harder and they simply can’t understand what are the new system advantages we are talking about.

It is very important to understand that we need to deliver value in small chunks. Many times in my career I’ve heard: “…but we can’t deliver this useful system in small!” Really? How do you know you can deliver it on a bigger scale then? History shows us time after time that megalomaniac plans crumble. And, to make matters worse, they crumble often over many, many years.

Managers often expect that developers can plan their work while the developers have trouble to account for all the complexity in advance – often the accidental (that is “not essential”) complexity. And the accidental complexity always gets higher with bigger system, there is simply no remedy for that. Analyse as much as you want, you find out something unexpected the minute you start coding. Or when you meet with a customer. These are truths known for decades now, but still they seemingly make no sense to many managers and other key decision makers.

And so far we’ve only talked about mismatch in beliefs how to build complex systems. What does it matter whether you want to “build it” or “let it grow”, whether you are forced to “fixed time, fixed price” contract or you can do it really incrementally using whatever agile is currently chic – this all is not important at all when the true reason to spend the money is… well, to spend the money!

Yes, public money, aka nobody’s money – who cares? People care, of course, people who are in the chain somewhere. People who decide who should participate and have some piece from that big cake – competent or not, doesn’t matter. There are always subcontractor that will do it in the end. Money talks. And value is just standing aside. Just as users and their needs do.

It can be scaled down

Of course, it can, the question is whether we dare to be accountable and flexible to deliver clear value for the money. Value that is easy to see and evaluate whether it’s worth it or not. In Estonia they are also far from perfect, but they try hard to keep it small and simple (KISS principle). They limit their evaluation/analysis projects to 50k Euro and implementation projects to 500k.

I saw people laugh at this but 500k in these countries is a reasonable cost for 8-10 person team for a year. Yes, you have to mix juniors and seniors, which is pretty normal – and no, you can’t pay for 3 more levels of management above the team. Get out of their way and they will likely deliver more value than a similar team in a typical corporate environment that has to spend 20% of their time with reporting and other overhead (and that’s a low estimate).

If the cost calculation doesn’t work for you, take less people and make the project last half a year, not full. I’m not to be convinced that there is no way to deliver visible value within 500k Euro.

Risto Hinno also mentioned another very interesting thing. They decide how many services – or how much work if you will – they want implemented at a time. This way they prevent IT market in Estonia from heating up too much because that leads to very low quality. Companies start hiring everyone and anyone, a lot of code is written by external workers who often don’t care and everything is also done at way too high pace. These are all recipes for disaster. Things they seem to know in Estonia, but not here in Slovakia.

Problems with services

Risto talked also about typical problems they faced. The learned the hard way that services must have an owner. He also presented the maturity model of the services. Using my notes and not necessarily exactly his words the levels are:

1. ad hoc services,
2. simple list of services is managed,
3. services have their owners,
4. services are measured (including their value),
5. service management is a daily routine.

He talked about building measurement in the services. This part of the talk rang a lot of devops/continuous delivery bells. And he also talked about the future visions they have:

• Base future services on life events. This makes them more obvious to their consumers, that is citizens.
• Aggregated services – many simple services can collaborate to achieve more complex scenarios. Risto actually mentioned some crazy number of services they have, but also noted that many of them are really simple. Still – it’s easier to put together simple blocks than to cut and slice big blocks.
• Link between public and IT services.

So Estonia seems to have started well and they keep improving. I wish they keep on track because I loved the ideas presented – and many of them were familiar to me. I just needed to hear that it actually somewhere works. And now it’s time to get to the next level.

Designing the next generation of government services around user needs

That was the title of the presentation by Ciara Green who came to tell us how they do it in the United Kingdom. She works for GDS, Government Digital Service and she talked about the transformation of government services that, if we simplify it, started around 2010 with quite a short letter (11 pages) by Martha Lane Fox after she was asked to oversee a review of the state of the government services at the time. Sure, 11 pages seems long for a letter, but it was short in a world where you likely get hundred(s) pages of analysis that is not to the point in the end. The letter was.

After this government services all came under a single domain gov.uk and many other good things happen. UK is way ahead of Slovakia, historically, mentally of course (despite the Brexit and all the lies leading to it) – so it doesn’t come as a surprise that they decided to focus on value and they also used current agile methodologies.

They knew what happens if you deliver over many years and then surprise your customer or users – invariably not a good surprise. So they started to deliver fast and often, tested a lot, tested with real users including seniors, focused on UX. Just as Risto, Ciara too argued for making things simple. It is very easy to do things complex and longer and we should do the opposite. We should start with needs, real world needs, remind us these often. And we should do less    (reminds me the powerful “less but better” mantra).

Another interesting point was Good services are Verbs. Bad services are names. Of course there are also other components, various registers, but in the end the focus should be on services and on the activities (e.g. life situations) they cover. Sure, the verbs are a bit unusual sometimes. One very important service is called Verify and it verifies the identity of the user with various partners (banks, Royal Mail, and more) because in UK there is no central register of citizens. So they can do this without keeping personal data (I don’t know the details) and here in Slovakia we build various registers for years and they often add more problems than they solve.

Funny enough, when she talked about some services and the name was used, it still functioned as a noun in the sentence – quite naturally. So I believe the word class used for names may not be the most important thing but using verbs may remind us what we try to solve.

Back to Slovak reality

Ciara’s talk was pure sci-fi for us. She works for government agency where they develop services in an agile way. How crazy is that? Pretty much, if you say it in Slovakia. Slovak services are portioned between many companies, most of them with political background (not official, of course), and we spent around 800 million Euro for government IT that looks like a very bad joke. Each ministry takes care of its own sandbox and if there is some initiative to unify how IT is done it is executed extremely bad.

For example, there is some central portal for public services that acts as a hub and connects various parties in government IT. However, this “service” is good mostly for the provider of the service, not for its users. The protocols are crazy complicated, if you need to connect to it (you want to or is forced to, which is more likely) you need to conform to some strict plan for testing, etc. There is no way to do it agile, it only separates you from the service you want to talk to. It adds another barrier between you and the other party, not only technically but also organizationally.

It is said that one minister mentioned to a young woman working at the ministry, horrified how the state works, that she should not be naive, that sometimes things are as they are and we have to be realistic. He, reportedly, pointed at government IT and the bad companies who suck money out of it. Now this is all a matter of speculation, but the words could have been said. The tragedy is twofold.

First: The companies do what they are allowed to do. It is not that bad companies do whatever they want, they do it with connections to the officials of the government and various bureaus. As crazy as it sounds, stories that someone who worked for some company now works for the state and manages projects his previous employer delivers. Stories like this are uncovered on virtually a daily basis now.

Second: Even if it was true and the bad companies did whatever they wanted… then the state totally failed to do its basic job. It actually did fail in the first case too, but here it seems to be a very weak state, not the state our officials depict to us.

Final words

While the Slovak reality is pretty bleak, it was very refreshing to see that it can be done and how it’s done somewhere else. It’s nice to see that agile can work, even more – it can work in a state agency. And that state agencies can deliver true value, when they really focus on it. We have also learned that state can regulate how much he wants at once. This can – and should – be done in IT, but also in infrastructure projects like motorways (another anti-example from Slovakia). It gives you better quality for lower price and surprisingly it still gets done sooner in the end!

In any case, there is a long way for Slovakia and Slovaks to get to the point when we can focus on value and don’t have to fight with elemental lack of political culture (to which I include wasting/misusing/frauding public money as well).

Neither Risto nor Ciara brought any political topics in, but some of the Slovak political “folklore” obviously affected the questions that were asked afterwards. Corruption was mentioned not once. But these were areas where our speakers couldn’t help us (oh, how I envy them).

The presented topics were so interesting for us that UX parts were often left aside a bit – although focusing on value and user from the start is pretty useful recommendation. But as with anything simple, it is much harder to do it than to do something complicated and big.

Bratislava World Usability Day 2016 and the future of design

By a lucky coincidence I visited the World Usability Day (WUD) event here in Bratislava. It was November 10th, 2016 – as any other event of the same name around the world. Theme for this year was Sustainability, but for us, working with and for the public sector, it was even more attractive thanks to the guest from UK and Estonia government agencies that implement or oversee the government services – services for real people, citizens.

I will talk about government services in the followup post. This one will be more about design and how I feel about it. Mind you, I’m a software developer with some experiences with real users – I always prefer to hear from them although listening blindly to your users is also not a recipe for success. I’m not a designer. But I’m also a user of many things – and not only modern technology gadgets. Maybe I have some twisted programmer’s perspective but that doesn’t make me less a user.

Design of everyday things

Before going on, let me divert to a book I’m just reading – The Design of Everyday Things. I’ll probably never be a great designer but there are many basic aspects of design we can learn about and use them every day in software development. In the book I also found many funny examples of frustrating experiences – experiences we all have to go through sometimes.

I’m personally torn between the progress and stability. I understand the progress is inevitable – and in many cases it doesn’t affect the design. Technology performance and capacities get higher and it all gets smaller at the same time – this doesn’t mean we have to change how we interact with computers or computer-based devices like smartphones. On the other hand we can – and we even should because previous UIs were insufficient and current performance allows us to do so much better. Are we doing better?

Everybody now experiments with design but I doubt they test it properly. I wonder how Google – that definitely has facilities and resources – tested when they changed “add document” button to the bottom-right corner. Anyone I met who used computers and not tiny screens couldn’t find that button. Then you have products developed by a single developer – how should they experiment in design? How much should then learn before? How much of they learn can inhibit their creativity?

One of the ideas of the book is that the importance of the design will only grow. I have to agree. How is it possible that you need to set the current time on your oven to be able to bake a cake (not just one brand)? If we screw ovens after decades they worked already how can we design revolutionary devices? But maybe we can – perhaps the problem is not with designing new types of devices where we expect some searching. Perhaps we’re just too meddlesome and can’t resist redesigning what doesn’t need redesign anymore.

Role of Sustainability or Sustainability of Roles

Back to the WUD 2016 and the presentation that had the designated theme in its title, presented by Lukáš “Bob” Mrvan (with Avast). Videos from Bratislava event can be found here – and while the page is in Slovak, it is easy to find the videos there – and most of them are in English (all I mentioned are at least). Pity they are not made as a split screen between the slides and the presenter or that they don’t take the slide more often.

Sustainability definitely resonated throughout the presentation. This may seem annoying to some but not to me as I’m convinced our current lifestyle is unsustainable.

Another interesting idea was that too often we focus on technical part of the design and not on the whole experience. E.g. Bob was talking about their call centre – they needed to replace their insufficient application, but the most important change might have been designing their call scripts properly. Of course this wasn’t the first time I’d heard about this more holistic approach. So, just as the book says the importance of the design will grow, Bob claims the role of designer will change. And I agree.

But this all raises more questions, obviously. Maybe we need dedicated design experts on big projects, but what about small ones? How much of the design essentials must we take in to deliver useful software? How much an analyst and developer and tester should know about the design? And how to keep track of it when it develops like crazy nowadays? How to distinguish lasting advices from fashion trends?

Focus on people…

Part of the presentation discussed the speed of progress and its acceleration, talking about exponential Moore’s law vs our slow linear improvements in IQ. I take these only as visualization aids for the idea that the change is indeed inevitable. But when someone puts exponential curve on a linear scale and says “look at the pace of change since 2000” then I can move myself to 2000 and say “look at the crazy pace of change since 1985”. The rate is still the same it just affects more and more of our lives, that’s all.

Yes, society changes, design of things should get better and easier. The exponential curve doesn’t tell us anything different now than any time before. But right now it governs lives of virtually everyone (or soon it will). What to do with that is beyond the discussion about design, but the design is affected too.

…not just users, but workers as well

But there is one positive about these facts. Knowing that people evolve slower than technology we can focus on them – learn how we work, something about psychology (and psychopathology) of design, how we interact with things. This knowledge will last, it’s much better investment than learning something about the newest framework. Learning the technology is also necessary, of course, but we should find time for learning more important bigger ideas as well.

Bob mentioned it can be difficult to persuade our managers to give us time for learning and added a chart of performance of the top organizations vs average. The top organizations have also higher levels of employee satisfaction and learning culture is part of it. These are all known facts documented in many books, some of them decades old.

Some believe that in our line of work we should educate in our free time – and while I agree with this to a degree I refuse the idea that we should just be prepared anytime for anything at work. If organization doesn’t want us to practice at work at all, it can’t expect we will do it home, especially later in our lives with families. It’s also different to have a solo practice and a team practice.

To wrap it up

Bob’s presentation was much more cultural than technical. This seems to be the trend at the conferences nowadays. This is a good shift in overall although not all presentations are quality. This one was one of the better ones, definitely on the inspiring side of a spectrum. Bob also organized an exhibition about design, he is active in the community – so he’s got experiences of his own to present on the topic.

One of the questions about the design is – do we need revolutionary changes or will evolutionary suffice? Bob was more on the revolutionary side, it seemed to me. I understand the need for these in new areas. But revolutionary changes make me personally tired in many existing devices – especially the phones and web applications.

Productivity is directly tied to the design of things. If we need to relearn how to work with a phone every other year I don’t call that good progress. Like switching back and menu buttons? I have two phones with each of the buttons on the opposite sides!

Applications come and go and nothing is developed for reasonable time. Smart TVs are called a failure because people refused them, but producers refuse the idea that their Smart hubs (or whatever they call it) suck. They don’t improve the applications there. It’s been reported years ago that YouTube on Samsung smart TV does not use external keyboard – and it still doesn’t. If we don’t care about improving applications evolutionary as well, revolution will not bring anything good.

With this I’ll finish this post – mostly about design – and in the next one I’ll talk about government services. Those should also be about the design but are much more about politics, especially here in Slovakia.

AWS did it again, this time with Lightsail

I still remember how AWS (Amazon Web Services) re:Invent 2015 conference impressed me. AWS is extremely successful but they keep pushing innovations at us. I played with their smallest EC2 instances couple of times, considering whether or not I want my own server.

This year at re:Invent 2016 happening during this week Andy Jassy, CEO of Amazon Web Services, announced many new services in his keynote. One that resonated with me immediately was Lightsail. It is extremely affordable server with plenty to offer even in its weakest configuration. For $5 per month you’ll get 512MB of memory, 1 vCPU, 20GB SSD and 1TB data transfer. See also this blog post for more information.

With such a reasonable base price I decided to try it – by the way, the first month is free! My plans were nothing big, I actually just wanted to move my homepage there. But you have that flexibility of a Linux box ready for you anytime you fancy.

I spun my Lightsail instance in no time, choose AMI Linux just for a change (considering my Ubuntu preference) and with Google’s help I got nginx and PHP 7 up and running very quickly indeed. I used the in-browser SSH but because it’s not quite the console I like (but close, even Shift+PgUp/Down works as expected) I wanted to put my public key in ~/.ssh/authorized_keys. I didn’t know how to copy/paste it from my computer, but when you press Ctrl+Alt+Shift in the web SSH it will open a sidebar where you can paste anything into the clipboard and right-click will do the rest.

I really liked this experience, I like the price as well and in-browse SSH comes very handy when you are at a place where port 22 is blocked for whatever reason. (I’m sure it has something with compliance, but don’t want me to understand.) I’m definitely keeping my new pet server although I know that cattle is more common now. Especially in the cloud.

Eclipse 5 years later – common formatter quest

It was 5 years ago when I compared IntelliJ IDEA and Eclipse IDE in a series of blog posts (links to all installments at the bottom of that post). I needed to migrate from IDEA to Eclipse, tried it for couple of months and then found out that I actually can go on with IDEA. More than that, couple of years later many other developers used IDEA – some in its free Community edition, others invested into the Ultimate (comparison here).

Today I have different needs – I just want to “develop” formatter for our project that would conform to what we already use in IDEA. I don’t know about any automatic solution. So I’ll install Eclipse, tune its formatter until reformat of the sources produces no differences in my local version of the project and then I’ll just add that formatter into the project for convenience of our Eclipse developers.

Importing Maven project in Eclipse

I went to their download page, downloaded, started the executable and followed the wizard. There were no surprises here, Eclipse Mars.2 started. With File/Import… I imported our Maven project – sure that wizard is overwhelming with all the options, but I handled. Eclipse went on with installing some Maven plugin support. This is unknown for IDEA users – but it’s more a problem of Maven model that doesn’t offer everything for IDE integration, especially when it comes to plugins. It also means that plugins without additional helper for Eclipse are still not properly supported anyway. In any case, it means that Eclipse will invade your POM with some org.eclipse.m2e plugins. Is it bad? Maybe not, Gradle builds also tend to support IDEs explicitly.

Eclipse definitely needed to restart after this step (but you can say no).

SVN support

We use Subversion to manage our sources. I remembered that this was not built-in – and it still is not. Eclipse still has this universal platform feeling, I’m surprised it knows Java and Maven out of the box.

But let’s say Subversion is not that essential. I wasn’t sure how to add this – so I googled. Subversive is the plugin you may try. How do I install it? Help/Install New Software… does the trick. I don’t know why it does not offer some reasonable default in Work with input – this chooses the software repository which is not clear to me at all from that “work with”. I chose an URL ending with releases/mars, typed “subv…” in the next filter field and – waited without any spinner or other notification.

Eventually it indeed found some Subversive plugin…s – many of them actually. I chose Subversive SVN Team Provider, which was the right thing to do. Confirm, OK, license, you know the drill… restart.

But this still does not give you any SVN options, I don’t remember how IDEA detects SVN on the project and just offers it, but I definitely don’t remember any of torturing like this. Let’s try Subversive documentation – I have no problem reading. Or watching Getting started video linked therein. 🙂

And here we go – perspectives! I wonder how other IDEs can do it without reshuffling your whole screen. Whatever, Eclipse == perspectives, let’s live with it. But why should I add repository when the URL to it is already somewhere in the .svn directory in the root of the project? Switching to SVN Repository Exploring perspective, Eclipse asked for SVN connector. Oh, so much flexibility. Let’s use SVN Kit 1.8.11, ok, ok, license, ok. Restart again? No, not this time, let’s wait until it installs more and do it at once. This was wrong decision this time.

I followed the video to add the SVN repository, but it failed not having the connector. Were I not writing this as I go, I’d probably remember I have installed one. 🙂 But I wasn’t sure, maybe I cancelled it, so let’s try SVN Kit, sounds good. It failed with “See error log for details.” Ok, I give up, let’s try Native JavaHL 1.8.14 (also by Polarion). This one works. Restart needed. Oh, … this time I rather really restarted as my mistake dawned on me.

I checked the list of installed software, but SVN plugins don’t seem to be there – this is rather confusing. But if you go to Windows/Preferences, in the tree choose Team/SVN, then tab SVN Connector – there you can find the connectors you can use. Sure I had both of them there. My fault, happy ending.

So I added SVN repository, but as the Getting started video went on, I knew I’m in trouble. It doesn’t say how to associate existing SVN project on my disk with a repo. I found the answer (stackoverflow of course). Where should I right click so that Team menu shows enabled Share project…? In Package explorer of course. I added one project, hooray! Now it shows SVN information next to it. But I noticed there is Share projects…, I don’t want to do it one by one, right? Especially when Eclipse does not show the projects in the natural directory structure (this sucks). I selected it all my projects but Team menu didn’t offer any Share at all now!

Ok, this would throw me out of balance at 20, but now I know that what can go wrong goes wrong. That only project associated with SVN already – I had to deselect to let Eclipse understand what I want. Strictly speaking there is some logic in eliminating that menu item, but as a user I think otherwise. So now we are SVN ready after all!

I updated the project (not using other perspective), no information whether it did something or not – IDEA shows you updated files without getting into your way. Should have used synchronize, I know…

Oh, and it’s lunch time, perfect – I really need a break.

Quick Diff

This one is for free with IDEA, with Eclipse we have to turn it on. It’s that thing showing you changes in a sidebar (or gutter).

Windows/Preferences, filter for “quick” and there you see it under General/Editors/Text Editors. Well it says enabled, but you want to check Show differences in overview ruler too. In my case I want to change colours to IDEA-ish yellow/green/red. (Who came with these Sun/enterprise violetish colours for everything in Eclipse?) What to use as reference source? Well, with IDEA there is no sense for “version on disk” option. I chose SVN Working Copy Base in hope it does what I think it does (shows my actual outgoing changes that are to be committed).

Outgoing changes contain unmanaged files!

Ah yeah, I recall something like this. This is the most stupid aspect of SVN integration – it does not respect how SVN work. After seeing my outgoing changes in Team Synchronizing perspective (probably my least favourite and most confusing one for me) I was really afraid to click on Team/Commit… But as the three dots indicate, there is one more dialog before any damage is done – and only managed files are checked by default. So commit looks good, but disrespect of outgoing changes to the SVN underlying principles is terrible. Eclipse users will tell you to add files to ignore, but that is just workaround – you can then see in the repository all the files people needed to ignore for the sake of stupid team synchronization. Just don’t auto-add unmanaged files, show some respect!

Eclipse Code Style options

With quick diff ready I can finally start tuning my formatter. There are some good news and some bad news. Well, these are no news actually, nothing has changed since 2011. Code Style in IDEA is something you can set up for many languages in IDEA. It also includes imports. In Eclipse when you filter for “format” in Preferences you see Formatter under Java/Code Style and more options for XML/XML Files/Editor. These are completely separated parts and you cannot save them as one bunch. For Imports you have Java/Code Style/Organize Imports.

In my case it doesn’t make sense to use project specific settings. What I change now will become workspace specific, which is OK with me – but only because I don’t want to use Eclipse for any other projects with different settings (that would probably either kill me or I’d have to put them into separate workspaces).

And then we have Java/Code Style/Clean Up configuration (this is what Source/) and Java/Editor/Save Actions to configure and put into project(s) as well. Plenty of stuff you need to take care of separately.

Line wrapping and joining

One of the most important thing we do with our code in terms of readability is line wrapping – and one thing I expect from any formatter is an option that respects my line breaks. Eclipse offers “Never join lines” on Line Wrapping and Comment tab. It seems you have to combine them with “Wrap where necessary” option for most options on Line Wrapping tab, but it still does not allow you to split line arbitrarily – it joins the lines back on reformat, to be precise.

Sometimes I want to say “wrap it HERE” and keep it that way. In IDEA I can wrap before = in assignment or after – and it respects it. I don’t know about any specific line-break/wrapping option for this specific case. Eclipse respects the wrap after, but not the one before – it re-joins the lines in the latter case. Sure I don’t mind, I prefer the break after = as well. But obviously, Eclipse will not respect my line breaks that much as IDEA.

Just to be absolutely clear, I don’t mind when a standalone { is joined to the previous line when rules say so. There are good cases when control structures should be reformatted even when wrapped – but these are cases revolving mostly around parentheses, braces or brackets.

When I investigated around “Never join lines” behaviour I also noticed that people complain about Eclipse Mars formatter when compared to Luna one. Do they rewrite them all the time or do they just make them better? Do they have good tests for them? I don’t know. Sure, formatters are beasts and we all want different things from them.

Exporting Eclipse settings

Let’s say you select top right corner link Configure Project Specific Settings… in particular settings (e.g. Organize Imports). This opens dialog Project Specific Configuration – but do I know what is the scope of it when I select my top-level project? Actually – I don’t even see my top level project (parent POM in our case), only subset of my open projects based on I don’t know what criteria. That is ridiculous.

I ended up exporting settings using Export All… button – but you have to do it separately for whatever section you want. In our case it’s Clean Up, Formatter, Organize Imports and Save Actions. I simply don’t know how to do it better. I’ll add these XML exported configs into SVN, but everybody has to import them manually.

IDEA with its project (where project is really a project in common sense) and modules (which may be Maven “project”, but in fact just a part of the main project) makes much more sense. Also, in IDEA when you copy the code style into the project you feel sure we’re talking about all of the code style. If I add it to our SVN everybody can use it.

You can also export Code Style as XML – but a single one. Or you can export all of IDE settings and choose (to a degree) what portions you want to import. While this is also manual solution you need to do it once with a single exported config.

(This situation is still not that bad as with keybinds where after all these years you still can’t create your own new Scheme in a normal fashion inside the Eclipse IDE.)

Conclusion

Maybe the way of Go language, where formatting is part of the standard toolchain, is the best way – although if it means joining lines anywhere I definitely wouldn’t like it either.

I can bash IDEA formatter a bit too. For me it’s perfectly logical to prefer annotations for fields and methods on separate line, but NOT wrapping them when they are on the same line. Just keep the damn lines a bit different when I want it this way! Something like soft format with prefered way how to do the new code. This is currently not possible all the way. I can set IDEA formatter in such a way that it keeps annotations on separate lines and respects them at the same line as well – but all the new code has annotations by default on the same line.

This concept combining “how I prefer it” with “what I want to keep preserved even if it’s not the way I’d do it” is generally not the way formatters work now. I believe it would be a great way how they should work. This can partially be solved by formatting only changed lines, but that has its own drawbacks – especially when the indentation is not unified yet.

Templating localization with gender and case

In this article we will explore the problem how to localize messages that repeat a lot and only a small part of the message varies. The variable part will be typically some domain object and repeating messages are various confirmations, error and similar messages revolving around these domain objects.

Our code examples will be in Java using standard Java MessageFormat and version using ICU project, ICU4J to be precise. While ICU library is rather taxing on disk space (10 MB if you download it using Maven, as it contains all possible localization data) and also their patterns are more verbose than the one from java.text there is one big advantage. Besides the ICU4C (C version) also available on the ICU site there are other JavaScript implementations. Maybe not official, but they refer to ICU and this sounds promising if you need the same message resources both on Java backend and JavaScript frontend.

Before we go on I’ll point to various resources regarding best practices of localization. Virtually any bigger project where localization matters has something similar – and among other natural things (like prefer Unicode or never use hardcoded strings) there is one notoriously repeated warning: Don’t concatenate localized strings.

Reason for this is very obvious if/when you try any other language than your default because even similar languages have different structures for some things. Sometimes the number goes first and then the date, sometimes the other way – things like that. How does this relate to our problem?

Example sentences

Imagine an application where you can select multiple things in a table and delete them or delete using some filter. We need message that announces the result to the user. And we also can edit an object and we want to see that it was successfully saved. This is enough for our needs. Our domain objects are: Transaction and Client.

English language

Example sentences with bold showing our noun (domain object) that can change and italics for other affected words:

• Transaction was successfully saved.
• No transactions were deleted. (singular variation possible)
• One transaction was deleted.
• 2 transactions were deleted. (or more)

Problems to note:

• We need Transaction or transaction depending on the position in the sentence. Solution: Either use the whole phrase per domain object (noun). For insertion into some template format we need two different keys or a single key with some pre-processing (de)capitalizing the first letter. This either needs to be supported on the message format level or we can do it in the code after the whole sentence is completely formatted. We cannot choose/change casing for a single inserted word because for various languages it can be at different positions in the sentence.
• We need plural/singular for a noun – this possibly combined with the need for various casing of the first character. We also need to show the parameter (number), possibly with word versions for some cases (“No”). Finally, we need to use were/was appropriately. Solution: some choice format mechanism, mostly readily available.

In overall, no problem to add new domain object (nouns) and put them into the sentences somehow. But this ties us to English rather too much. Let’s see some different language to see the whole problem.

Slovak language

The same sentences in Slovak (bold for noun and italics for other affected words):

• Transakcia bola úspešne uložená. (The subject noun is in Slovak “nominative” case, and “bola … uložená” is affected by the feminine gender of “transakcia”. This sentence is needed only in singular.)
• Žiadna transakcia nebola zmazaná. (Word “transakcia” is in singular “nominative”, this time with lowercase. The rest of the sentence reflects case for “none deleted” with all three words affected by the feminine gender of “transakcia”. Just like in English, plural variation is possible. Unlike English, word “nebola” means “was not” – sentences with two negatives are normal in Slovak and these don’t cancel out.)
• Jedna transakcia bola zmazaná. (Singular nominative, feminine affecting the whole sentence, this time with positive “bola zmazaná”, that is “was deleted”.)
• 2 transakcie boli zmazané. (For cases 2-4: Plural nominative, feminine.)
• 5 transakcií bolo zmazaných. (For cases 5 and more: Plural genitive, feminine. Here the number plays the “nominative” role and the whole subject here is roughly like “five (of) transactions” and instead of “of” Slovak uses genitive case.)

Alternative example for “Transaction was successfully saved”, something like “Saving of transaction was successful”:

• Ukladanie transakcie bolo úspešné. (Here none of the words are affected by the noun gender, it wouldn’t even be affected by the number. However, the noun itself is not in “nominative” case anymore, instead it is in “genitive”. This cannot be generalized to some “object” case (in addition to normal “subject” case) as objects in Slovak can be in different cases, mostly “accusative”. This information also cannot be worked with in the code, not even if the code is related to a single message localization, because this information relates to a single language, here Slovak. We explore the possibilities with various approaches lower.)

Problems:

• Many words in the template itself are affected by the gender of the noun. Solution: Localize complete messages. Using templates we need to obtain the gender information somehow (part of the key? another key? what about languages where it does not matter at all?) and use it as a parameter for the formatting mechanism (some choice/select format).
• We may need to use various cases of the same noun, depending on the message or even a parameter of the message (like the count of transactions).

And what about the Client?

You can imagine the messages with “client” in English, just replace the single word. No inflection, no cases, just respect the number and letter casing depending on the position in the sentence.

Things are different in Slovak though. Let’s see the sentence about about Client being saved but first let’s repeat the one about transaction for comparison:

• Transakcia bola úspešne uložená. (Singular, feminine, noun in nominative.)
• Klient bol úspešne uložený. (Singular, but masculine, nominative.)

Rather innocent change in English is a nightmare in Slovak if you want to reuse the structure of the message somehow. So what are our options?

Approach 1: whole sentences

Pure Java solution

Imaginary localization file Simple.properties:

transaction.saved=Transaction was successfully saved.
transaction.deleted={0,choice,0#No transaction was|1#One transaction was\
  |1<{0} transactions were} deleted.

client.saved=Client was successfully saved.
client.deleted={0,choice,0#No client was|1#One client was\
  |1<{0} clients were} deleted.

Last word “deleted” might be included into each sentence too – and this is actually recommended when your choice already involves significant portion of the sentence anyway.

The same for Slovak in Simple_sk.properties:

transaction.saved=Transakcia bola úspešne uložená.
transaction.saved.alt=Ukladanie transakcie bolo úspešné.
transaction.deleted={0,choice,0#Žiadna transakcia nebola zmazaná|1#Jedna transakcia bola zmazaná\
  |1<{0} transakcie boli zmazané|4<{0} transakcií bolo zmazaných}.

client.saved=Klient bol úspešne uložený.
client.saved.alt=Ukladanie klienta bolo úspešné.
client.deleted={0,choice,0#Žiadny klient nebol zmazaný|1#Jeden klient bol zmazaný\
  |1<{0} klienti boli zmazaní|4<{0} klientov bolo zmazaných}.

And some demo program to print it out:

import java.text.MessageFormat;
import java.util.Arrays;
import java.util.Locale;
import java.util.MissingResourceException;
import java.util.ResourceBundle;

public class SimpleApproach {
    public static void main(String[] args) {
        showDemo("transaction", Locale.getDefault());
        showDemo("client", Locale.getDefault());
        showDemo("transaction", Locale.forLanguageTag("sk"));
        showDemo("client", Locale.forLanguageTag("sk"));
    }

    private static void showDemo(String domainObject, Locale locale) {
        System.out.println("\nLOCALE: " + locale);
        print(locale, domainObject + ".saved");
        print(locale, domainObject + ".saved.alt");
        for (Integer count : Arrays.asList(0, 1, 2, 4, 5, 99)) {
            print(locale, domainObject + ".deleted", count);
        }
    }

    private static void print(Locale locale, String messageKey, Object... args) {
        String message = format(locale, messageKey, args);
        System.out.println(messageKey + Arrays.toString(args) + ": " + message);
    }

    private static String format(Locale locale, String key, Object... args) {
        ResourceBundle bundle = ResourceBundle.getBundle("Simple", locale);
        try {
            String pattern = bundle.getString(key);
            return new MessageFormat(pattern, locale)
                .format(args);
        } catch (MissingResourceException e) {
            return "";
        }
    }
}

ICU4J solution

I’ll use ICU4J MessageFormat instead of the one from java.util. The usage is actually the same for both cases, only the import statement and loaded resource is different. ICU4J allows not only positional arguments, but also named ones. For this reason we also changed how the arguments are provided, because named parameters in a map are much cleaner. But first the resource files – SimpleIcu.properties:

transaction.saved=Transaction was successfully saved.
transaction.deleted={count,plural,=0 {No transaction was}\
  one {One transaction was}\
  other {{count} transactions were}} deleted.

client.saved=Client was successfully saved.
client.deleted={count,plural,=0 {No client was}\
  one {One client was}\
  other {{count} clients were}} deleted.

And for Slovak – SimpleIcu_sk.properties:

transaction.saved=Transakcia bola úspešne uložená.
transaction.saved.alt=Ukladanie transakcie bolo úspešné.
transaction.deleted={count,plural,=0 {Žiadna transakcia nebola zmazaná}\
  one {Jedna transakcia bola zmazaná}\
  few {{count} transakcie boli zmazané}\
  other {{count} transakcií bolo zmazaných}}.

client.saved=Klient bol úspešne uložený.
client.saved.alt=Ukladanie klienta bolo úspešné.
client.deleted={count,plural,=0 {Žiadny klient nebol zmazaný}\
  one {Jeden klient bol zmazaný}\
  few {{count} klienti boli zmazaní}\
  other {{count} klientov bolo zmazaných}}.

Program listing:

import java.util.Arrays;
import java.util.Collections;
import java.util.Locale;
import java.util.Map;
import java.util.MissingResourceException;
import java.util.ResourceBundle;

import com.ibm.icu.text.MessageFormat;

public class SimpleIcuApproach {
    public static void main(String[] args) {
        showDemo("transaction", Locale.getDefault());
        showDemo("client", Locale.getDefault());
        showDemo("transaction", Locale.forLanguageTag("sk"));
        showDemo("client", Locale.forLanguageTag("sk"));
    }

    private static void showDemo(String domainObject, Locale locale) {
        System.out.println("\nLOCALE: " + locale);
        print(locale, domainObject + ".saved", Collections.emptyMap());
        print(locale, domainObject + ".saved.alt", Collections.emptyMap());
        for (Integer count : Arrays.asList(0, 1, 2, 4, 5, 99)) {
            print(locale, domainObject + ".deleted", Collections.singletonMap("count", count));
        }
    }

    private static void print(Locale locale, String messageKey, Map args) {
        String message = format(locale, messageKey, args);
        System.out.println(messageKey + args + ": " + message);
    }

    private static String format(Locale locale, String key, Map args) {
        ResourceBundle bundle = ResourceBundle.getBundle("SimpleIcu", locale);
        try {
            String pattern = bundle.getString(key);
            return new MessageFormat(pattern, locale)
                .format(args);
        } catch (MissingResourceException e) {
            return "";
        }
    }
}

Message output is the same for both cases except for args part, but that is part of the label, not part of the final message. Please note that the demo listings don’t work efficiently with resource bundles as they call getBundle for every formatting. Do it better in production, or use some other abstraction above it, e.g. Spring.

Pros and cons

It’s easy! It’s easy to read the messages, or at least as easy as it gets. The trouble is that every time you add a new domain object you need to kinda replicate all the generic messages for it. Here we have two messages only, but imagine there’s ten of them or more. This is not exactly DRY, but in localization world it is quite safe way how to play it. The trouble is that if you decide to change the generic message you have to go over many places (consequence of duplication).

Can we do a bit better? Can we template the messages and combine them with some forms of those domain object names?

Approach 2: single template and noun nesting

We already foreshadowed all the problems our template solution must overcome. Concatenation is out of the question – but we’re playing it nicely with templates that allows structure of the sentence being completely different in various languages. But we still have to:

• Treat starting letter of the sentence somehow (when it is a sentence, but let’s say we always know).
• Use a specific inflection form in a template when the list of forms (cases and numbers) is not known to the code. It must not, of course, as the list may vary depending on a language. For nouns in English we just need singular/plural, but for Slovak you need nominative/genitive/accusative in both singular/plural forms – we have even more cases, but the unused ones are not important.
• Template message needs to know the gender of the noun (domain object) as it may affect a lot of words in it.

Reading the list of problems we need to retrieve some information about the object domain noun without our code really understanding it and pass it somehow to the template. So we don’t know much about the information we need about the noun in advance. Maybe we can narrow it to the case/number combination, but it actually doesn’t matter. We know there are multiple forms of the noun – but the code doing our templating doesn’t know the names of the forms. Only template and the noun knows – and only in a particular language.

We don’t even know what forms for a particular template are needed – the writer or the template knows. We can only settle on a set of needed forms for the nouns. In case we need a new form, for instance for some new template, we have to add the new form for all existing nouns. But this design is still orthogonal. The question is – how to offer all the available forms to the template?

ICU4J solution

I’ll start with ICU4J because it allows those named parameters we already used in the simple approach. We can just fill the map with all possible forms of the word and let the template do the job. So how can we get a map of all the forms? Will we have multiple keys in the bundle with some suffix? I don’t like that. Let’s do something more brutal – we put some serialized map into a single key. I’d go for JSON, but as I don’t want to import any JSON library, I’ll use some special characters I really don’t expect in the names of the domain objects.

English resource file TemplateIcu.properties is pretty boring:

domain.object.transaction=sg:transaction,pl:transactions
domain.object.client=sg:client,pl:clients

object.saved={sg} was successfully saved.
object.deleted={count,plural,=0 {No {sg} was}\
  one {One {sg} was}\
  other {{count} {pl} were}} deleted.

We have both domain object names in singular (sg) and plural (pl) forms. Templating for English is obviously a no-brainer, with only two templates for two objects we are already saving a lot of characters. How about Slovak property file TemplateIcu_sk.properties?

domain.object.transaction=gend:fem,nom:transakcia,gen:transakcie,\
  pl:transakcie,plgen:transakcií
domain.object.client=gend:mas,nom:klient,gen:klienta,\
  pl:klienti,plgen:klientov

object.saved={nom} {gend, select, mas {bol úspešne uložený}\
  fem {bola úspešne uložená}\
  other {!!!}}.
object.saved.alt=Ukladanie {gen} bolo úspešné.
object.deleted={gend, select,\
  mas {{count,plural,\
    =0 {Žiadny {nom} nebol zmazaný}\
    one {Jeden {nom} bol zmazaný}\
    few {{count} {pl} boli zmazaní}\
    other {{count} {plgen} bolo zmazaných}}} \
  fem {{count,plural,\
    =0 {Žiadna {nom} nebola zmazaná}\
    one {Jedna {nom} bola zmazaná}\
    few {{count} {pl} boli zmazané}\
    other {{count} {plgen} bolo zmazaných}}}\
  other {!!!}}.

Ok, at this moment it’s longer than the original file, but I’d say with the third domain object we would already save some characters – granted we don’t need more forms and genders. Eventually, for Slovak at least, we would add neuter gender too, but that’s about it.

Now the code that runs it:

import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
import java.util.MissingResourceException;
import java.util.ResourceBundle;

import com.ibm.icu.text.MessageFormat;

public class TemplateIcuApproach {
    public static void main(String[] args) {
        showDemo("transaction", Locale.getDefault());
        showDemo("client", Locale.getDefault());
        showDemo("transaction", Locale.forLanguageTag("sk"));
        showDemo("client", Locale.forLanguageTag("sk"));
    }

    private static void showDemo(String domainObject, Locale locale) {
        System.out.println("\nLOCALE: " + locale);
        print(locale, domainObject, "object.saved", Collections.emptyMap());
        print(locale, domainObject, "object.saved.alt", Collections.emptyMap());
        for (Integer count : Arrays.asList(0, 1, 2, 4, 5, 99)) {
            print(locale, domainObject, "object.deleted", Collections.singletonMap("count", count));
        }
    }

    private static void print(Locale locale, String domainObject, String messageKey, Map args) {
        ResourceBundle bundle = ResourceBundle.getBundle("TemplateIcu", locale);
        Map objectInfo = parseObjectInfo(bundle.getString("domain.object." + domainObject));
        // not generified, sorry; we know that objectInfo is mutable, so we do it this way
        objectInfo.putAll(args);
        String message = format(bundle, locale, messageKey, objectInfo);
        if (sentenceRequiresCapitalization(message, true)) {
            message = Character.toUpperCase(message.charAt(0)) + message.substring(1);
        }
        System.out.println(messageKey + args + ": " + message);
    }

    private static boolean sentenceRequiresCapitalization(String message, boolean isSentence) {
        return isSentence && message != null && !(message.isEmpty())
            && Character.isLowerCase(message.charAt(0));
    }

    // no sanity checking here, but there should be some
    private static Map parseObjectInfo(String objectInfoString) {
        Map map = new HashMap();
        for (String form : objectInfoString.split(" *, *")) {
            String[] sa = form.split(" *: *");
            map.put(sa[0], sa[1]);
        }
        return map;
    }

    private static String format(ResourceBundle bundle, Locale locale, String key, Map args) {
        try {
            String pattern = bundle.getString(key);
            return new MessageFormat(pattern, locale)
                .format(args);
        } catch (IllegalArgumentException e) {
            return e.getMessage();
        } catch (MissingResourceException e) {
            return "";
        }
    }
}

Remember again that we should not call getBundle as often as I do here. Take this just as a proof of concept. 🙂 Code also performs capitalization of the first letter when the message is sentence

• that’s the true argument in the call. This information could be embedded into the format too, but that could be awkward. Best solution would be some custom formatter for an argument, something like {nom, capitalizeFirst}, but I can’t find any mechanism to do that with this MessageFormat class. But that is rather minor issue.

Actually, there may be better alternative – just use two separate keys for nominative, that is “Nom” and “nom” (that is sg/Sg in English resources) with respective casing of the first letter. We don’t need this distinction for other forms, at least not here, not yet. Most of the cases will hardly appear at the start of the sentence.

We can easily create a tool that checks validity of the keys starting with “domain.object.” and indicate those that don’t have expected forms for a particular language. And we can also easily use this solution in JavaScript – which is very important for us.

Pure Java solution

It is not completely impossible to do this in Java, but you’d need to use indexes – and that sucks. If you needed to find all the occurrences of nominative it’s easy with ICU’s named parameters. Just search for {nom} which is pretty distinct pattern. With Java you’d have to decide fixed indexes for the forms and then … put the real arguments as indexes after these? What?! What if I discover I need to add another form? Will I renumber all arguments (only for that language, mind you) to make a room for it? And searching? Should I search for {0} as nominative? What about cases where we don’t use domain objects in the message and zeroth argument means something completely different?

Honestly, we will skip this solution altogether. It’s not viable.

Pros and cons

Before I wrote the proof of concept I was not sure if I tried this – but now I’m pretty sure we will try it! The key was to come up with the answer to the question how to feed the template message pattern with all possible forms? How to do it without code knowing about languages and concrete form names? With the map structure serialized in the domain object key this is all easy.

Sure, the generic messages are a bit more messy, but we don’t repeat ourselves anymore. It seems that combo gender-select/plural works for most cases. There are probably even more demanding messages, but I believe that the deletion message with count shows worthiness of this solution.

Multiple inserted words?

What if we want to insert multiple nouns? What if the sentence is something like “Transaction for client XY is above limit”? Here “transaction” and “client” are the domain objects and “XY” is a concrete name of a client (not a problem, we – hopefully – don’t want to inflect that). If we merge form maps for domain.object.transaction and domain.object.client one will override the other. That’s not good. What we need is to give these guys some role.

Before going on, I have to admit that this example is not a good one, because if only transactions can break limits and if we always want to show a client there, this would be message with a key like transaction.above.limit and all inserted words in all languages would be there verbatim, no insertion, only the name of the client would really be an argument. So before we get carried away but the opportunity to use it everywhere we need to think and prefer NOT to use it when not necessary. Just because we have our “multi-form” dictionary of domain objects doesn’t mean we want to use it. There is one legitimate case when you might to – when you expect that you change the word for “transaction” and/or “client”. But even if you think it may be actually useful for American vs British or similar – again, don’t. You want different bundles for both language variants with specified country.

We should not go on with wrong example, right? So how about the sentence “There are some transactions for a client XY – do you want to delete them as well?” From business point this still is silly, because we probably don’t want to delete anything like this, but at least we can see that this sentence can be used over and over for various combinations of domain objects.

I’ll not implement the whole solution here, let’s just shed a bit of light on it. Messages in English – this time we introduced singular with indefinite article (sgwa):

domain.object.transaction=sg:transaction,sgwa:a transaction,pl:transactions
domain.object.client=sg:client,sgwa:a client,pl:clients

object.delete.constraint.warning=There are some {slave.pl} for {master.sgwa} {name} - do you...?

The same in Slovak:

domain.object.transaction=gend:fem,nom:transakcia,gen:transakcie,\
  pl:transakcie,plgen:transakcií
domain.object.client=gend:mas,nom:klient,gen:klienta,\
  pl:klienti,plgen:klientov

object.delete.constraint.warning=Pre {master_gen} {name} existujú {slave_pl} - zmažeme...?

This time it was easy, no gender, but you can expect it can get more complicated. The point is we indicated the role of the domain object with the prefix, like master_. Now how we call message formatting to work like this? I’ll offer a snippet of code, but again, its API can be groomed, but you’ll probably finish this design with your needs in mind anyway:

...
print(loc, "object.delete.constraint.warning",
    Collections.singletonMap("name", "SuperCo."),
    new DomainObject("client", "master"),
    new DomainObject("transaction", "slave"));
...

private static void print(
    Locale locale, String messageKey, Map args, DomainObject... domainObjects)
{
    ResourceBundle bundle = ResourceBundle.getBundle("TemplateIcu", locale);

    // not generified, sorry
    Map finalArgs = new HashMap(args);
    for (DomainObject domainObject : domainObjects) {
        finalArgs.putAll(domainObject.parseObjectInfo(bundle));
    }

    String message = format(bundle, locale, messageKey, finalArgs);
    System.out.println(messageKey + args + ": " + message);
}

// format method like before, parseObjectInfo is embedded into following class:
public class DomainObject {
    private final String domainObject;
    private final String role;

    public DomainObject(String domainObject, String role) {
        this.domainObject = domainObject;
        this.role = role;
    }

    public DomainObject(String domainObject) {
        this.domainObject = domainObject;
        this.role = null;
    }

    // no sanity checking here, but there should be some
    Map parseObjectInfo(ResourceBundle bundle) {
        String objectInfoString = bundle.getString("domain.object." + domainObject);
        Map map = new HashMap();
        for (String form : objectInfoString.split(" *, *")) {
            String[] sa = form.split(" *: *");
            String key = role != null ? role + '_' + sa[0] : sa[0];
            map.put(key, sa[1]);
        }
        return map;
    }
}

That’s it! Not just 1 insertion anymore, but N – we know only two cardinalities 1 and N, right?

Your API for translation sucks!

There is no API, these are just demo programs. I’d aim for something much more fluent indeed. Resource bundle would be somehow available, for instance for current user’s locale. Then the code could go like this:

String message = new LocalizedMessage(resourceBundle, "delete.object")
  .withParam("count", deletedCount)
  .forDomainObject(editor.getDomainObjectName()) // optional role parameter possible
  .format();

This code can be in some unified object(s) deleted confirmation dialog that can be reused across many specific editors. Editor merely needs to provide the name of the domain object (generic one like “transaction”, not the name for the particular instance). I guess this API makes sense and it’s easy to get there.

Conclusion

Writing this post was part of the exploratory process here, I wasn’t sure what the conclusion be. I now see that the template solution is viable, but it builds on the MessageFormat that supports named parameters. As we already use ICU to align our backend (ICU4J) and frontend (we use yahoo/intl-messageformat) we can build on that.

I didn’t go into performance characteristics, but we’re talking about enterprisy software already. The whole message formatting is not cheap anyway. Unified template is definitely more complex to parse, at least the gender there is on top of original patterns. We can cache domain object maps, of course after we change the way we merge them with actual arguments as my solution up there adds arguments into this map. But otherwise I don’t believe it’s a big deal. I never compared ICU with java.text, but we need ICU features and don’t see any problems yet.

So, yes, templating of often used messages with some snippets changing in them is possible and makes sense. No it’s not concatenation. I believe we’re not doing anything internationally illegal here. 🙂 (GitHub project with sources)

How I unknowingly deviated from JPA

In a post from January 2015 I wrote about possibility to use plain foreign key values instead of @ManyToOne and @OneToOne mappings in order to avoid eager fetch. It built on the JPA 2.1 as it needed ON clause not available before and on EclipseLink which is a reference implementation of the specification.

To be fair, there are ways how to make to-one lazy, sure, but they are not portable and JPA does not assure that. They rely on bytecode magic and properly configured ORM. Otherwise lazy to-one mapping wouldn’t have spawned so many questions around the internet. And that’s why we decided to try it without them.

Total success

We applied this style on our project and we liked it. We didn’t have to worry about random fetch cascades – in complex domain models often triggering many dozens of fetches. Sure it can be “fixed” with second-level cache, but that’s another thing – we could stop worrying about cache too. Now we could think about caching things we wanted, not caching everything possibly reachable even if we don’t need it. Second-level cache should not exist for the sole reason of making this flawed eager fetch bearable.

When we needed a Breed for a Dog we could simply do:

Breed breed = em.find(Breed.class, dog.getBreedId());

Yes, it is noisier than dog.getBreed() but explicit solutions come with a price. We can still implement the method on an entity, but it must somehow access entityManager – directly or indirectly – and that adds some infrastructure dependency and makes it more active-record-ish. We did it, no problem.

Now this can be done in JPA with any version and probably with any ORM. The trouble is with queries. They require explicit join condition and for that we need ON. For inner joins WHERE is sufficient, but any outer join obviously needs ON clause. We don’t have dog.breed path to join, we need to join breed ON dog.breedId = breed.id. But this is no problem really.

We really enjoyed this style while still benefiting from many perks of JPA like convenient and customizable type conversion, unit of work pattern, transaction support, etc.

I’ll write a book!

Having enough experiences and not knowing I’m already outside of JPA specification scope I decided to conjure a neat little book called Opinionated JPA. The name says it all, it should have been a book that adds a bit to the discussion about how to use and tweak JPA in case it really backfires at you with these eager fetches and you don’t mind to tune it down a bit. It should have been a book about fighting with JPA less.

Alas, it backfired on me in the most ironic way. I wrote a lot of material around it before I got to the core part. Sure, I felt I should not postpone it too long, but I wanted to build an argument, do the research and so on. What never occurred to me is I should have tested it with some other JPA too. And that’s what is so ironic.

In recent years I learned a lot about JPA, I have JPA specification open every other day to check something, I cross reference bugs in between EclipseLink and Hibernate – but trying to find a final argument in the specification – I really felt good at all this. But I never checked whether query with left join breed ON dog.breedId = breed.id works in anything else than EclipseLink (reference implementation, mind you!).

Shattered dreams

It does not. Today, I can even add “obviously”. JPA 2.1 specification defines Joins in section 4.4.5 as (selected important grammar rules):

join::= join_spec join_association_path_expression [AS] identification_variable [join_condition]
join_association_path_expression ::=
  join_collection_valued_path_expression |
  join_single_valued_path_expression |
  TREAT(join_collection_valued_path_expression AS subtype) |
  TREAT(join_single_valued_path_expression AS subtype)
join_spec::= [ LEFT [OUTER] | INNER ] JOIN
join_condition ::= ON conditional_expression

The trouble here is that breed in left join breed does not conform to any alternative of the join_association_path_expression.

Of course my live goes on, I’ve got a family to feed, I’ll ask my colleagues for forgiveness and try to build up my professional credit again. I can even say: “I told myself so!” Because the theme that JPA can surprise again and again is kinda repeating in my story.

Opinionated JPA revisited

What does it mean for my opinionated approach? Well, it works with EclipseLink! I’ll just drop JPA from the equation. I tried to be pure JPA for many years but even during these I never ruled out proprietary ORM features as “evil”. I don’t believe in an easy JPA provider switch anyway. You can use the most basic JPA elements and be able to switch, but I’d rather utilize chosen library better.

If you switch from Hibernate, where to-one seems to work lazily when you ask for it, to EclipseLink, you will need some non-trivial tweaking to get there. If JPA spec mandated lazy support and not define it as mere hint I wouldn’t mess around this topic at all. But I understand that the topic is deeper as Java language features don’t allow it easily. With explicit proxy wrapping the relation it is possible but we’re spoiling the domain. Still, with bytecode manipulation being rather ubiquitous now, I think they could have done it and remove this vague point once for all.

Not to mention very primitive alternative – let the user explicitly choose he does not want to cascade eager fetches at the moment of usage. He’ll get a Breed object when he calls dog.getBreed(), but this object will not be managed and will contain only breed’s ID – exactly what user has asked for. There is no room for confusion here and at least gives us the option to break the deadly fetching cascade.

And the book?

Well the main argument is now limited to EclipseLink and not to JPA. Maybe I should rename it to Opinionated ORM with EclipseLink (and Querydsl). I wouldn’t like to leave it in a plane of essay about JPA and various “horror stories”, although even that may help people to decide for or against it. If you don’t need ORM after all, use something different – like Querydsl over SQL or alternatives like JOOQ.

I’ll probably still describe this strategy, but not as a main point anymore. Main point now is that JPA is very strict ORM and limited in options how to control its behavior when it comes to fetching. These options are delegated to JPA providers and this may lock you to them nearly as much as not being JPA compliant at all.

Final concerns

But even when I accept that I’m stuck to EclipseLink feature… is it a feature? Wouldn’t it be better if reference implementation strictly complained about invalid JPQL just like Hibernate does? Put aside the thought that Hibernate is perfect JPA 2.1 implementation, it does not implement other things and is not strict in different areas.

What if EclipseLink reconsiders and removes this extension? I doubt the next JPA will support this type of paths after JOINs although that would save my butt (which is not so important after all). I honestly believed I’m still on the standard motorway just a little bit on the shoulder perhaps. Now I know I’m away from any mainstream… and the only way back is to re-introduce all the to-one relations into our entities which first kills the performance, then we turn on the cache for all, which hopefully does not kill memory, but definitely does not help. Not to mention we actually need distributed cache across multiple applications over the same database.

In the most honest attempt to get out of the quagmire before I get stuck deep in it I inadvertently found myself neck-deep already. ORM indeed is The Vietnam of Computer Science.

Building Windows VirtualBox machines

I started this post in January originally, but after a couple of paragraphs I realized I’m writing a more generic post – Believe in build automation. Now you know why I believe in automation and we can get straight to it. I’m a Linux guy, I’d rather work with Linux, I always prefer UNIX/Linux on servers, but I run Windows desktop to be conformant. After all I can run anything in VirtualBox when I need it.

And sometimes what I need is just another Windows. But I don’t want to manually prepare the box all over again after it expires (evaluation), not to mention I want a repeatable process (because I believe in it :-)). You may snapshot your virtual machines, but you cannot avoid eventual end of evaluation period.

State of affairs in Windows automation

Couple of years ago I got a new computer with Windows and I wanted to put all my favourite tools on it. Of course I didn’t have a list. But I had a feeling I’m repeating myself. I also wanted to disable some Windows features. I had experimented with PowerShell before, so I turned to it with faith. I found out that there are some PowerShell modules that allow to add/remove features or applications, but they are limited only to Windows Server. Couple of ugly words ran through my head and I postponed my dream.

Now I know this was a hasty decision, because Microsoft does not offer just one good standard way how to do it. As explained here, you can use one of two PowerShell modules. ServerManager module was the one that made a bit angry because of its restrictions to server versions of Windows, but there is also Dism module available on any recent platform, not to mention dism.exe itself, that works for older Windows incarnations as well.

While this all is just a minor episode, it documents how difficult it may be to find the right way how to perform various tasks on the command line (and preferably PowerShell) for a newcomer. And I wasn’t even that new on Windows.

But after this it was easier and easier to use the right words and ask the right questions the right way to get my answers. Most of them were on StackOverflow, but I have to praise Microsoft’s sites too. Sure, sometimes you have to go over couple of Microsoft pages, but in overall you can find the answers.

Back to Windows virtual machine creation

Here you go, I nearly did it again! Wrote a different post than I wanted, that is. So back to the topic. Of course, you need to know your options for automation, so learning more about PowerShell and about ways how to (un)install various Windows features is still important. But we also need to know the general workflow how to bring Windows virtual machine to life. I decided to use Vagrant because it aims for developers and is praised by them.

When it comes to Windows there is one big trouble – because Windows is big. The same trouble exists for Linux too, but is smaller. We’re talking about automated installation. Good news is that both systems can be installed automatically. It comes as no surprise for Linux, but Windows also features so-called “unattended installation” which aims for corporate world where admins don’t want to sit through installations for all the computers in a big company.

It works in a simple way – you provide an XML file with this unattended configuration for the computer and Windows finds it during installation. It can be found on a floppy or USB drive.

I don’t know all the options how to install Windows in an automatic fashion, but this one is good for virtual machines. All you need to do is provide the booting virtual machine DVD with Windows ISO image and Autounattend.xml file on a virtual floppy disk.

Can Vagrant do it? Ok, now you got me. 🙂 Probably it can, but a brief investigation on the Internet revealed that instead of doing this installation with Vagrant I should first use Packer. And – what a surprise – both tools are developed by the very same company/guy (HashiCorp/Mitchell Hashimoto, sorry for leaving out any other participants from the company). I was not the only one confused about the differences between Vagrant and Packer.

Shortly (and maybe not 100% precise), Packer is good in creating virtual machine base images and Vagrant is good to use them in your development process. Base image is not installation ISO, it’s rather a snapshot of a virtual machine after installation with everything you want to have there for the start. Not with everything possible though, that’s why it’s called base image. Packer can build base images for various virtualization platforms, but we will focus on VirtualBox only.

My idol Matt Wrock

I decided to install the Windows somehow automatically using Vagrant and prefer PowerShell as much as possible. Searching for a solution I somehow found the post called In search of a light weight windows vagrant box by Matt Wrock. I read a bit of it but I was also intrigued by the link to the updated version using Packer and Boxstarter. Already by this time I was overflowed by new terms, but it was worth it, I promise!

Matt definitely knows his stuff, his Windows and automation experience is extensive and he can explain it properly as well. Just a day before that I wasn’t ready to add Packer to my arsenal, not to mention Boxstarter – and soon I learned about Chocolatey as well. Now, honestly, I still don’t get Boxstarter, so for me it’s just “some extension for PowerShell” (disregard at will), but I absolutely fell in love with Chocolatey, because with it the management of programs feels like on Linux.

Matt’s instructions how to use Packer and Boxstarter were pretty cool, he provides Packer files (configurations, or sort of recipes, for Packer, written in JSON format) for Windows Server 2012, Windows Nano (very interesting addition to Microsoft’s arsenal) and Windows 7 (here you need license key, as there is no evaluation ISO, shame). I definitely utilized Windows Server 2012R2, as the server edition always comes handy during development when you want to experiment with a domain controller, etc. But I also wanted packer template for Windows 10 – and I had to create the one myself.

Windows 10 experiments

Actually, the biggest problem with Windows 10 wasn’t the Packer template, but with the Autounattend.xml file. I found some generator, but it didn’t deliver without some errors. I’m still pretty sure that XML is far from flawless, it’s not cleaned up properly and so on – but it works. Diving into every detail in a field that is mostly new for me (the whole world of Windows automation) would probably stop me before I got to the result, so take it as it is, or make it better if you can.

I highly recommend to read that Matt’s article Creating windows base images using Packer and Boxstarter as it is a very good introduction into the whole pipeline. His packer templates also provided a great starting point for more experiments. I also liked the way how he minimized the images by removing many Windows components, defragging the disk, zeroing empty space, etc.

I summed up my experiments in a markdown file and looking at it, it definitely is not perfect and finished. But does it have to be? In a week I played with it I probably installed Windows 10 forty times. Most of these test I commented out the slow parts that were mere optimization steps mentioned above (minimizing the size of the image). I played a lot with some preinstalled software (using Chocolatey, of course) and tried to pre-configure it using registry changes where necessary. This is, however, in vain, as the sysprep.exe step wipes registry changes for vagrant user. Talking about vagrant user, be extra careful to spell it everywhere the same way. Once I messed it up, had vagrant in Autounattend.xml and Vagrant in postunattend.xml (which is used as C:\Windows\Panther\Unattend\unattend.xml by sysprep.exe) and had two Vagrant accounts – you don’t want that. 🙂

I tried hard to perform some installation and configuration steps after sysprep, I tried to change it from Packer’s shutdown_command to windows-restart provisioner step but I wasn’t able to overcome some errors. After a while I settled with a script I just copied to my Vagrant working environment directory and then ran it from initialized box where it appeared in c:\vagrant directory.

Sure I could do even better with full automation, but when things resist too much sometimes it’s good to step back, rethink the strategy and focus on quick wins.

Other options?

There are definitely more ways how to prepare Windows 10 box, or any Windows for that matter, with or without Packer, but even when we focus on Packer solutions there’s a wide spectrum of approaches. Some don’t bother to use sysprep.exe to generalize their installation – after all if it’s only for personal needs it really is not needed. People on GitHub seem to agree on using PowerShell as a Packer provider, but one of the solutions used no provider at all (all part of the build section). Also widen the search for other Windows versions and you’ll see much more variability.

Conclusion

Using Packer as a first step in the pipeline is very practical. You can prepare base image once and save a couple of hours any time you need a fresh environment (partially you can do the same with snapshots in VirtualBox, but it’s not the same).

I use Boxstarter in the process as recommended (although I’m not able to appreciate it fully) and Chocolatey to install/remove programs – during Packer steps and also anytime later. When my evaluation Windows runs out I simply refresh the packer image from ISO and I’m done for the next 90 or 180 days (depending on the OS version).

Following the installation enthusiasm I went on to install SQL Server 2014 Express from Chocolatey package, configure it using PowerShell bits and pieces found across many blogs and stackoverflow questions and wrote it down on GitHub. I actually got used to writing technical notes I may need later into these markdown files and it works very well for me. Now I have automation hiatus, but I’m sure I’ll get back to it and it’s good to find all the notes at hand, including unresolved problems and ideas.

Good luck with Windows automation!