Converting 96kHz 24-bit FLAC to OGG with ffmpeg

Lately my son Robin asked for Peter Gabriel’s song The Tower That Ate People in a car. I like OGGs, although recently it may have been pointless with MP3 patents being expired. But 15+ years ago it was an obvious choice for me, especially because most encoded MP3 files had also clearly cut out high frequencies and generally lower quality at the same bitrate. Again – not a problem I encountered with newer MP3s. But I stayed true to OGG and I honestly don’t need anything better than its Q7 level.

The song is on Peter’s OVO album but the version Robin likes is from Back to Front show in London. So I browsed it, played it and – all the songs were skipped. Darn! I knew it must be because of the quality being very high because the digital download, companion to the Blu-ray Deluxe Book Edition (yeah, I’m a fan), was in 96kHz for both FLAC and OGG. So I had to recode the OGG, or better FLAC to OGG in normal sample rate (44.1kHz).

FFmpeg for the rescue!

I previously transcoded OGGs to MP3 for a little radio that didn’t support OGGs (I never understand why this happens) and I was very satisfied with FFmpeg because when I can do something from a command line I prefer that. So today I downloaded Windows build of FFmpeg and tried to figure out the switches.

After some Googling I tried -codec:a libvorbis and it told me there is no such a codec. So I tried ffmpeg -codecs to find out what (and if) there is any OGG support. There was just vorbis decoder, so I tried that one. Then ffmpeg told me that it’s just experimental and I must add -strict -2 switch to enable it. It worked afterwards but the warning was strange so I investigated further.

The trouble was that the build from FFmpeg site did not have libvorbis compiled in. Every time you run ffmpeg it prints the configuration it was compiled with and mine didn’t show –enable-libvorbis in the output. It was by an accident I found out I’ve got ffmpeg also on my PATH – which was strange considered I didn’t put the downloaded version there. It was part of ImageMagick which I was pretty sure was installed using Chocolatey (most recommended!), I don’t even remember why. But now it came handy, because, behold, this one had libvorbis with it!

If you have Chocolatey already, just cinst -y imagemagick and then start a new console to find ffmpeg on your path. Or do it the hard way.

Those damn spaces!

I use spaces in the filenames, replacing them with underscores or something does not make much sense, not in 21st century I believe. I respect bash (and prefer it in Windows as well, as delivered by Git) and I consider myself more or less a power-user (more less than more I guess). I understand that long time ago text was the thing and objects were not. But all this white-space escaping is sometimes killing me. Just look at all the effort that went into escaping white-spaces – IFS, quoting, print0, etc.

Actually, using NUL character (print0) as a separator seems most logical but obviously it’s difficult to put it into plain text then. But plain text is so awkward to represent anything anyway (except for the actual text). I believe some richer environment where lists are true lists is the logical thing to have. I’m not hinting on PowerShell necessarily, not sure they have it right, but they tried to be progressive for sure.

When I quote the name containing spaces on the input it’s a single argument (let’s say $1). But when I use ffmpeg -i “$1″… in the script the program complains that the first word from the filename is not a valid name. I encountered this problem many times before, passing the arguments from a command line to the script and there to other commands. Today I learned that “${1}” is different from “$1”. I always used curlies only to separate name of a variable from potentially colliding surrounding. But the first one keeps $1 as a single parameter even for another executable called from a script. Handy. Not intuitive. And definitely not something you learn in this section, for instance.

If this was all more “object-oriented” (broader meaning) it would be a filename, String or even File object from the start all the way to where it should be used. Spaces would not matter.

Sample rate and unexpected “video” stream

Because the source flac file had sampling rate of 96kHz – and I suspected this was the main reason the car audio system didn’t play it – I wanted to resample the audio to more traditional CD quality. That’s what option -ar 44100 does. Because OGG seems to have a single sample format, I didn’t have to care about bringing 24bits down to 16.

But I was surprised that my OGG didn’t play in foobar2000 and loading it actually created two entries in a playlist. I checked the output of a command more carefully and noticed it also converted some JPEG image embedded in that FLAC to a “video” stream. Not interested, thank you, said I – and that’s what -vn (no video) switch does.

And the script is…

Add setting the quality of the output OGG and -y to overwrite the output (I experimented repeatedly, you may not want it, of course) and you get a script like this:

#!/bin/sh

ffmpeg.exe -i "${1}" -ar 44100 -vn -codec:a libvorbis -qscale:a 7 -y "${1%flac}ogg"

It only encodes one file. Last thing I wanted is to treat input arguments for a for loop, although I guess I could have used shift too. Anyway, the command is easy:

find . -name \*.flac -exec ./anything-to-ogg-44k1-q7.sh {} \;

I guess it doesn’t care about the input format as long as it recognizes it, hence the “anything”. Of course, ffmpeg can do much more – I just wanted to show one recipe, that’s all.

Advertisements

IntelliJ IDEA and yFiles diagrams

I have a long history with IntelliJ IDEA and in overall I love it. Sure there are tons of minor (and also more than minor) issues but it’s just like with a marriage – you have to find a partner who’s flaws you can tolerate. IntelliJ IDEA is such a partner for me.

But there is a thing I really don’t understand at all – for years I hoped JetBrains reconsiders, but it seems to be a lost cause. I believe their diagramming is just plain terrible. And the reason is not how the diagrams look like – even though while some UMLs are nice most dependency diagrams generated from any bigger POM file are virtually unusable.

The worst part of the whole experience is mouse control over the diagram:

  • Mouse wheel scrolls vertically, but terribly slow. Zoom would be much better. Use Ctrl+wheel. But even if you unzoom as much as possible, you often don’t see anything because the canvas is much bigger and the diagram gets lost somewhere. Tip: Try to find it in the middle using scrollbars for orientation.
  • If you ever tried to drag the viewport (aka hand tool or something) you find out that all the intuitive ways don’t work. Even worse, they stand in your way. Use Ctrl+left mouse, BTW.
  • If you try middle mouse on its own you’ll experience the most weird thing – after you drag with middle mouse, nothing happens. But the moment you move the mouse cursor a magnifier appears on the spot where you ended dragging. No clicking makes it go away. Tip: Just press Alt, seriously. Don’t ask. With Alt you can actually move the magnifier around.

If you’ve ever seen dependency diagram in Eclipse you probably can’t understand the layout of IDEA’s diagrams – or anything else around them for that matter. In the aforementioned dependency diagrams you either see a cobweb of lines with pixel-sized boxes, or hardly more than couple of boxes you can actually read at the same time. I tried various layouts and I don’t like either really – and it’s not my visual taste, it’s how unhelpful their layout typically is.

There are some things to like, I guess. I like how you can navigate them with quick search (typing). But that’s about it. Why the visualization and layout engine is how it is and why the controls are so counter-intuitive is beyond me.

Of course, if you’re more regular user of these diagrams you’ll probably get familiar with the controls. Not sure about the layout though. I don’t know how much these flaws are related to the used commercial yFiles and how much is IDEA’s integration.

Conclusions? Don’t forget that Alt to get rid of that magnifier. 🙂 I personally believe the diagrams could have been much better and easier to navigate, perhaps even without a commercial engine. That would also enable the community around the free IntelliJ Platform to build something on top of them.

Opinionated JPA with Querydsl book finished!

I’m not sure I’ve ever had such a long pause in blogging – not that I blog that often, but still. I either didn’t want to blog about how not to do things (current project I’m working on), or made various notes for myself in GitHub markdown – or, slowly but surely, working on my book. And this post is about the book.

The book is complete

I finally finished the first edition (and perhaps the last, but not necessarily) of my first technology book named Opinionated JPA with Querydsl. I started the book on December 16th, 2015. I planned to do it sometime during 2016. But September 2017 isn’t that late after all – especially with so little happening around JPA nowadays.

When I started I imagined a book around 100 pages but the thing grew over 200 in the end. Sure, I made font a bit bigger so it’s easy to read on ebook readers even in PDF format which still seems to be superior in presentation although less flexible on small readers. And even in this volume I didn’t cover all the things that are not covered in traditional JPA/ORM books.

Don’t mess with JPA, will ye?

I have to admit that I still don’t know JPA in and out although I can navigate the specification pretty well when I need to find something. There are features I simply refused to use, but for most of these I know they don’t solve the problems I typically have. If I must put it into a single point it would be better control over generated SQL.

Now I can hear those ORM purists and I believe I understand this topic reasonably well. I’ve heard about ORM being leaky abstraction, heard why it’s bad and when it’s actually good, I’ve read many articles on the topic and worked many hours using Java ORM solutions. If you want something extensive, there is always Ted Neward’s The Vietnam of Computer Science which was written in 2006 and hardly anything is out of date.

But I don’t care about academic ideas here, ORM is real, it’s used and I actually like many of its features. The least I like its effort to hide SQL from us though. I like its type conversion when compared to very poor low-level JDBC. I can live with unit-of-work as well but there are cases when it’s simply not suitable. And then you’re left on your own.

Streaming long query straight to a file or socket? Expect out of memory if you’re querying entities that fill up your persistence context eventually, even so you don’t need them there at all. Even without persistence context, it simply tries to create the whole list first before you can work with it. No cursor, nothing. Is this really such an unexpected and rare need?

Not compliant, not knowing it

I always firmly believed that if you work with SQL database one should know SQL. Whatever blanket you put over it, ignorance is hardly ever a good thing. I wrote quite a lot of articles on JPA. I saw first-hand what happens when you consider open-session-in-view a pattern instead of what it really is (antipattern). I tacked N+1 problem in context of pagination or thought about repeating problem of mapping enums to arbitrary database values. I realize that all the theory about specification crumbles in practice when you get into crossfire of various bugs in various JPA providers. I tried to modularize single entity model (persistence unit).

However I also liked improvements in JPA 2.1 and ORM still made my life easier in most situations. When I discovered that I can actually join on arbitrary value – e.g. map a foreign key as a plain value and then use join with on clause explicitly – I was blown away. That’s when I asked myself: “Why other people don’t try it too? Why we keep fighting ins and outs of relation mappings? Why we rely on convoluted configurations or particular providers to give us lazy to-one mapping?”

And then I decided to write a book about it. There was more to it – I wanted to lump more of my rogue ideas about JPA/ORM, staying still more or less concerned user of JPA, not a hater. I also wanted to see whether I can pull it off, all the way. I wanted to see how it is to self-publish a book on something like Leanpub. I didn’t expect much of a profit from that though, I realized this is no Perennial Seller as it’s too technology related and really niche, destined to be out-of-date rather soon.

But then during writing the book while I was testing my ideas both with Hibernate and EclipseLink, I found out that Hibernate does not support JOIN on root entity (e.g. join Dog d on …), only on entity paths (e.g. join person.dog). What the… how could they miss this thing?! And then it dawned on me… this is not part of a specification. My book more or less stopped for a couple of months, but eventually went on admitting openly that I’m not JPA compliant anymore. Good thing is that Hibernate eventually joined the club and since 5.1 they support this so called “ad hoc joins”.

Here we are

I’d just like to return to abstractions we talked about previously before I end this post. Right now I’m reading Patterns of Software by Richard P. Gabriel, written in 1996. We can argue that some of the problems are solved already, but I’d not be that sure. There’s a chapter called Abstraction Descant. I found out it really relates to me. Abstractions are important tools in our arsenal, but not everything can be solved by abstraction.

After reading this I realized I care even less whether ORM is leaky abstraction or not – it should be practical and not really that much how clean or perfect abstraction it is. Especially ORM being quite a big beast. It’s not a low level where abstractions shine best – like data structures, etc. I’m not going to say more, read that part from the book and make your own mind.

So – I’ve finished the book, hopefully not in vein. Kind of a longer blog post if you will. If you’re interested but not sure about it, you can grab it for free (and you can eventually pay later if you like it and feel it helped, I’m sure it’s possible :-)).

Bratislava World Usability Day 2016 and government IT

I wrote about sustainability and design takeaways from Bratislava World Usability Day 2016 in my previous post. World Usability Day (WUD) 2016 was organized on November 10th, 2016 in many places around the world. Theme for this year was Sustainability, but for us, working with and for the public sector, it was even more attractive thanks to the guest from UK and Estonia government agencies that implement or oversee the government services – services for real people, citizens. Services that should serve – just like the state itself should. And that is very touchy topic here in Slovakia.

Videos from Bratislava event can be found here, the page is in Slovak, but videos are easy to find and are in English.

Estonia: pathfinder or e-Narnia?

Risto Hinno came to us from Estonia, the state renown for it’s level of e-government. But if you imagined their world as a perfect place with flawless services you’d be wrong. Risto came to talk about their approach to the services and the problems they had to overcome and are overcoming.

Estonia and Slovakia are both countries from the Eastern Bloc, Slovakia is the successor of Czechoslovakia, while Estonia is one of the post-Soviet states. Both states are in NATO and EU and both use Euro, but there are also some important differences. I may not be historically accurate, but while in Slovakia we still have plenty of “old guard” people in their posts (like judges) and plenty of old-thinking politicians, many of them previously members of Communist party, now often using the sticker saying “social democrat”. In Estonia most of these were Russians and they simply were gone after Estonia became independent. And that allowed for deeper change, change that is much needed here in Slovakia but haven’t happened. Some ask: “Will it ever?”

But back to the services. As Risto put it, what we (citizens) want is value, but what we typically get is bureaucracy. The answer to this problem is to make everything smaller and simpler and really focus on the value.

Problems small and big

But just as with value-vs-bureaucracy problem there are opposite forces in play here. Even when the stakeholders agree on delivering maximum value for the money they often don’t agree on how to do it.

Very often the expectations are big and the budget follows them. Very often we don’t respect the systems our users already work with. And very often we deliver little value for a lot of money afterwards. Or worse, we often make the life of our users harder and they simply can’t understand what are the new system advantages we are talking about.

It is very important to understand that we need to deliver value in small chunks. Many times in my career I’ve heard: “…but we can’t deliver this useful system in small!” Really? How do you know you can deliver it on a bigger scale then? History shows us time after time that megalomaniac plans crumble. And, to make matters worse, they crumble often over many, many years.

Managers often expect that developers can plan their work while the developers have trouble to account for all the complexity in advance – often the accidental (that is “not essential”) complexity. And the accidental complexity always gets higher with bigger system, there is simply no remedy for that. Analyse as much as you want, you find out something unexpected the minute you start coding. Or when you meet with a customer. These are truths known for decades now, but still they seemingly make no sense to many managers and other key decision makers.

And so far we’ve only talked about mismatch in beliefs how to build complex systems. What does it matter whether you want to “build it” or “let it grow”, whether you are forced to “fixed time, fixed price” contract or you can do it really incrementally using whatever agile is currently chic – this all is not important at all when the true reason to spend the money is… well, to spend the money!

Yes, public money, aka nobody’s money – who cares? People care, of course, people who are in the chain somewhere. People who decide who should participate and have some piece from that big cake – competent or not, doesn’t matter. There are always subcontractor that will do it in the end. Money talks. And value is just standing aside. Just as users and their needs do.

It can be scaled down

Of course, it can, the question is whether we dare to be accountable and flexible to deliver clear value for the money. Value that is easy to see and evaluate whether it’s worth it or not. In Estonia they are also far from perfect, but they try hard to keep it small and simple (KISS principle). They limit their evaluation/analysis projects to 50k Euro and implementation projects to 500k.

I saw people laugh at this but 500k in these countries is a reasonable cost for 8-10 person team for a year. Yes, you have to mix juniors and seniors, which is pretty normal – and no, you can’t pay for 3 more levels of management above the team. Get out of their way and they will likely deliver more value than a similar team in a typical corporate environment that has to spend 20% of their time with reporting and other overhead (and that’s a low estimate).

If the cost calculation doesn’t work for you, take less people and make the project last half a year, not full. I’m not to be convinced that there is no way to deliver visible value within 500k Euro.

Risto Hinno also mentioned another very interesting thing. They decide how many services – or how much work if you will – they want implemented at a time. This way they prevent IT market in Estonia from heating up too much because that leads to very low quality. Companies start hiring everyone and anyone, a lot of code is written by external workers who often don’t care and everything is also done at way too high pace. These are all recipes for disaster. Things they seem to know in Estonia, but not here in Slovakia.

Problems with services

Risto talked also about typical problems they faced. The learned the hard way that services must have an owner. He also presented the maturity model of the services. Using my notes and not necessarily exactly his words the levels are:

  1. ad hoc services,
  2. simple list of services is managed,
  3. services have their owners,
  4. services are measured (including their value),
  5. service management is a daily routine.

He talked about building measurement in the services. This part of the talk rang a lot of devops/continuous delivery bells. And he also talked about the future visions they have:

  • Base future services on life events. This makes them more obvious to their consumers, that is citizens.
  • Aggregated services – many simple services can collaborate to achieve more complex scenarios. Risto actually mentioned some crazy number of services they have, but also noted that many of them are really simple. Still – it’s easier to put together simple blocks than to cut and slice big blocks.
  • Link between public and IT services.

So Estonia seems to have started well and they keep improving. I wish they keep on track because I loved the ideas presented – and many of them were familiar to me. I just needed to hear that it actually somewhere works. And now it’s time to get to the next level.

Designing the next generation of government services around user needs

That was the title of the presentation by Ciara Green who came to tell us how they do it in the United Kingdom. She works for GDS, Government Digital Service and she talked about the transformation of government services that, if we simplify it, started around 2010 with quite a short letter (11 pages) by Martha Lane Fox after she was asked to oversee a review of the state of the government services at the time. Sure, 11 pages seems long for a letter, but it was short in a world where you likely get hundred(s) pages of analysis that is not to the point in the end. The letter was.

After this government services all came under a single domain gov.uk and many other good things happen. UK is way ahead of Slovakia, historically, mentally of course (despite the Brexit and all the lies leading to it) – so it doesn’t come as a surprise that they decided to focus on value and they also used current agile methodologies.

They knew what happens if you deliver over many years and then surprise your customer or users – invariably not a good surprise. So they started to deliver fast and often, tested a lot, tested with real users including seniors, focused on UX. Just as Risto, Ciara too argued for making things simple. It is very easy to do things complex and longer and we should do the opposite. We should start with needs, real world needs, remind us these often. And we should do less    (reminds me the powerful “less but better” mantra).

Another interesting point was Good services are Verbs. Bad services are names. Of course there are also other components, various registers, but in the end the focus should be on services and on the activities (e.g. life situations) they cover. Sure, the verbs are a bit unusual sometimes. One very important service is called Verify and it verifies the identity of the user with various partners (banks, Royal Mail, and more) because in UK there is no central register of citizens. So they can do this without keeping personal data (I don’t know the details) and here in Slovakia we build various registers for years and they often add more problems than they solve.

Funny enough, when she talked about some services and the name was used, it still functioned as a noun in the sentence – quite naturally. So I believe the word class used for names may not be the most important thing but using verbs may remind us what we try to solve.

Back to Slovak reality

Ciara’s talk was pure sci-fi for us. She works for government agency where they develop services in an agile way. How crazy is that? Pretty much, if you say it in Slovakia. Slovak services are portioned between many companies, most of them with political background (not official, of course), and we spent around 800 million Euro for government IT that looks like a very bad joke. Each ministry takes care of its own sandbox and if there is some initiative to unify how IT is done it is executed extremely bad.

For example, there is some central portal for public services that acts as a hub and connects various parties in government IT. However, this “service” is good mostly for the provider of the service, not for its users. The protocols are crazy complicated, if you need to connect to it (you want to or is forced to, which is more likely) you need to conform to some strict plan for testing, etc. There is no way to do it agile, it only separates you from the service you want to talk to. It adds another barrier between you and the other party, not only technically but also organizationally.

It is said that one minister mentioned to a young woman working at the ministry, horrified how the state works, that she should not be naive, that sometimes things are as they are and we have to be realistic. He, reportedly, pointed at government IT and the bad companies who suck money out of it. Now this is all a matter of speculation, but the words could have been said. The tragedy is twofold.

First: The companies do what they are allowed to do. It is not that bad companies do whatever they want, they do it with connections to the officials of the government and various bureaus. As crazy as it sounds, stories that someone who worked for some company now works for the state and manages projects his previous employer delivers. Stories like this are uncovered on virtually a daily basis now.

Second: Even if it was true and the bad companies did whatever they wanted… then the state totally failed to do its basic job. It actually did fail in the first case too, but here it seems to be a very weak state, not the state our officials depict to us.

Final words

While the Slovak reality is pretty bleak, it was very refreshing to see that it can be done and how it’s done somewhere else. It’s nice to see that agile can work, even more – it can work in a state agency. And that state agencies can deliver true value, when they really focus on it. We have also learned that state can regulate how much he wants at once. This can – and should – be done in IT, but also in infrastructure projects like motorways (another anti-example from Slovakia). It gives you better quality for lower price and surprisingly it still gets done sooner in the end!

In any case, there is a long way for Slovakia and Slovaks to get to the point when we can focus on value and don’t have to fight with elemental lack of political culture (to which I include wasting/misusing/frauding public money as well).

Neither Risto nor Ciara brought any political topics in, but some of the Slovak political “folklore” obviously affected the questions that were asked afterwards. Corruption was mentioned not once. But these were areas where our speakers couldn’t help us (oh, how I envy them).

The presented topics were so interesting for us that UX parts were often left aside a bit – although focusing on value and user from the start is pretty useful recommendation. But as with anything simple, it is much harder to do it than to do something complicated and big.

Bratislava World Usability Day 2016 and the future of design

By a lucky coincidence I visited the World Usability Day (WUD) event here in Bratislava. It was November 10th, 2016 – as any other event of the same name around the world. Theme for this year was Sustainability, but for us, working with and for the public sector, it was even more attractive thanks to the guest from UK and Estonia government agencies that implement or oversee the government services – services for real people, citizens.

I will talk about government services in the followup post. This one will be more about design and how I feel about it. Mind you, I’m a software developer with some experiences with real users – I always prefer to hear from them although listening blindly to your users is also not a recipe for success. I’m not a designer. But I’m also a user of many things – and not only modern technology gadgets. Maybe I have some twisted programmer’s perspective but that doesn’t make me less a user.

Design of everyday things

Before going on, let me divert to a book I’m just reading – The Design of Everyday Things. I’ll probably never be a great designer but there are many basic aspects of design we can learn about and use them every day in software development. In the book I also found many funny examples of frustrating experiences – experiences we all have to go through sometimes.

I’m personally torn between the progress and stability. I understand the progress is inevitable – and in many cases it doesn’t affect the design. Technology performance and capacities get higher and it all gets smaller at the same time – this doesn’t mean we have to change how we interact with computers or computer-based devices like smartphones. On the other hand we can – and we even should because previous UIs were insufficient and current performance allows us to do so much better. Are we doing better?

Everybody now experiments with design but I doubt they test it properly. I wonder how Google – that definitely has facilities and resources – tested when they changed “add document” button to the bottom-right corner. Anyone I met who used computers and not tiny screens couldn’t find that button. Then you have products developed by a single developer – how should they experiment in design? How much should then learn before? How much of they learn can inhibit their creativity?

One of the ideas of the book is that the importance of the design will only grow. I have to agree. How is it possible that you need to set the current time on your oven to be able to bake a cake (not just one brand)? If we screw ovens after decades they worked already how can we design revolutionary devices? But maybe we can – perhaps the problem is not with designing new types of devices where we expect some searching. Perhaps we’re just too meddlesome and can’t resist redesigning what doesn’t need redesign anymore.

Role of Sustainability or Sustainability of Roles

Back to the WUD 2016 and the presentation that had the designated theme in its title, presented by Lukáš “Bob” Mrvan (with Avast). Videos from Bratislava event can be found here – and while the page is in Slovak, it is easy to find the videos there – and most of them are in English (all I mentioned are at least). Pity they are not made as a split screen between the slides and the presenter or that they don’t take the slide more often.

Sustainability definitely resonated throughout the presentation. This may seem annoying to some but not to me as I’m convinced our current lifestyle is unsustainable.

Another interesting idea was that too often we focus on technical part of the design and not on the whole experience. E.g. Bob was talking about their call centre – they needed to replace their insufficient application, but the most important change might have been designing their call scripts properly. Of course this wasn’t the first time I’d heard about this more holistic approach. So, just as the book says the importance of the design will grow, Bob claims the role of designer will change. And I agree.

But this all raises more questions, obviously. Maybe we need dedicated design experts on big projects, but what about small ones? How much of the design essentials must we take in to deliver useful software? How much an analyst and developer and tester should know about the design? And how to keep track of it when it develops like crazy nowadays? How to distinguish lasting advices from fashion trends?

Focus on people…

Part of the presentation discussed the speed of progress and its acceleration, talking about exponential Moore’s law vs our slow linear improvements in IQ. I take these only as visualization aids for the idea that the change is indeed inevitable. But when someone puts exponential curve on a linear scale and says “look at the pace of change since 2000” then I can move myself to 2000 and say “look at the crazy pace of change since 1985”. The rate is still the same it just affects more and more of our lives, that’s all.

Yes, society changes, design of things should get better and easier. The exponential curve doesn’t tell us anything different now than any time before. But right now it governs lives of virtually everyone (or soon it will). What to do with that is beyond the discussion about design, but the design is affected too.

…not just users, but workers as well

But there is one positive about these facts. Knowing that people evolve slower than technology we can focus on them – learn how we work, something about psychology (and psychopathology) of design, how we interact with things. This knowledge will last, it’s much better investment than learning something about the newest framework. Learning the technology is also necessary, of course, but we should find time for learning more important bigger ideas as well.

Bob mentioned it can be difficult to persuade our managers to give us time for learning and added a chart of performance of the top organizations vs average. The top organizations have also higher levels of employee satisfaction and learning culture is part of it. These are all known facts documented in many books, some of them decades old.

Some believe that in our line of work we should educate in our free time – and while I agree with this to a degree I refuse the idea that we should just be prepared anytime for anything at work. If organization doesn’t want us to practice at work at all, it can’t expect we will do it home, especially later in our lives with families. It’s also different to have a solo practice and a team practice.

To wrap it up

Bob’s presentation was much more cultural than technical. This seems to be the trend at the conferences nowadays. This is a good shift in overall although not all presentations are quality. This one was one of the better ones, definitely on the inspiring side of a spectrum. Bob also organized an exhibition about design, he is active in the community – so he’s got experiences of his own to present on the topic.

One of the questions about the design is – do we need revolutionary changes or will evolutionary suffice? Bob was more on the revolutionary side, it seemed to me. I understand the need for these in new areas. But revolutionary changes make me personally tired in many existing devices – especially the phones and web applications.

Productivity is directly tied to the design of things. If we need to relearn how to work with a phone every other year I don’t call that good progress. Like switching back and menu buttons? I have two phones with each of the buttons on the opposite sides!

Applications come and go and nothing is developed for reasonable time. Smart TVs are called a failure because people refused them, but producers refuse the idea that their Smart hubs (or whatever they call it) suck. They don’t improve the applications there. It’s been reported years ago that YouTube on Samsung smart TV does not use external keyboard – and it still doesn’t. If we don’t care about improving applications evolutionary as well, revolution will not bring anything good.

With this I’ll finish this post – mostly about design – and in the next one I’ll talk about government services. Those should also be about the design but are much more about politics, especially here in Slovakia.

AWS did it again, this time with Lightsail

I still remember how AWS (Amazon Web Services) re:Invent 2015 conference impressed me. AWS is extremely successful but they keep pushing innovations at us. I played with their smallest EC2 instances couple of times, considering whether or not I want my own server.

This year at re:Invent 2016 happening during this week Andy Jassy, CEO of Amazon Web Services, announced many new services in his keynote. One that resonated with me immediately was Lightsail. It is extremely affordable server with plenty to offer even in its weakest configuration. For $5 per month you’ll get 512MB of memory, 1 vCPU, 20GB SSD and 1TB data transfer. See also this blog post for more information.

With such a reasonable base price I decided to try it – by the way, the first month is free! My plans were nothing big, I actually just wanted to move my homepage there. But you have that flexibility of a Linux box ready for you anytime you fancy.

I spun my Lightsail instance in no time, choose AMI Linux just for a change (considering my Ubuntu preference) and with Google’s help I got nginx and PHP 7 up and running very quickly indeed. I used the in-browser SSH but because it’s not quite the console I like (but close, even Shift+PgUp/Down works as expected) I wanted to put my public key in ~/.ssh/authorized_keys. I didn’t know how to copy/paste it from my computer, but when you press Ctrl+Alt+Shift in the web SSH it will open a sidebar where you can paste anything into the clipboard and right-click will do the rest.

I really liked this experience, I like the price as well and in-browse SSH comes very handy when you are at a place where port 22 is blocked for whatever reason. (I’m sure it has something with compliance, but don’t want me to understand.) I’m definitely keeping my new pet server although I know that cattle is more common now. Especially in the cloud.

Eclipse 5 years later – common formatter quest

It was 5 years ago when I compared IntelliJ IDEA and Eclipse IDE in a series of blog posts (links to all installments at the bottom of that post). I needed to migrate from IDEA to Eclipse, tried it for couple of months and then found out that I actually can go on with IDEA. More than that, couple of years later many other developers used IDEA – some in its free Community edition, others invested into the Ultimate (comparison here).

Today I have different needs – I just want to “develop” formatter for our project that would conform to what we already use in IDEA. I don’t know about any automatic solution. So I’ll install Eclipse, tune its formatter until reformat of the sources produces no differences in my local version of the project and then I’ll just add that formatter into the project for convenience of our Eclipse developers.

Importing Maven project in Eclipse

I went to their download page, downloaded, started the executable and followed the wizard. There were no surprises here, Eclipse Mars.2 started. With File/Import… I imported our Maven project – sure that wizard is overwhelming with all the options, but I handled. Eclipse went on with installing some Maven plugin support. This is unknown for IDEA users – but it’s more a problem of Maven model that doesn’t offer everything for IDE integration, especially when it comes to plugins. It also means that plugins without additional helper for Eclipse are still not properly supported anyway. In any case, it means that Eclipse will invade your POM with some org.eclipse.m2e plugins. Is it bad? Maybe not, Gradle builds also tend to support IDEs explicitly.

Eclipse definitely needed to restart after this step (but you can say no).

SVN support

We use Subversion to manage our sources. I remembered that this was not built-in – and it still is not. Eclipse still has this universal platform feeling, I’m surprised it knows Java and Maven out of the box.

But let’s say Subversion is not that essential. I wasn’t sure how to add this – so I googled. Subversive is the plugin you may try. How do I install it? Help/Install New Software… does the trick. I don’t know why it does not offer some reasonable default in Work with input – this chooses the software repository which is not clear to me at all from that “work with”. I chose an URL ending with releases/mars, typed “subv…” in the next filter field and – waited without any spinner or other notification.

Eventually it indeed found some Subversive plugin…s – many of them actually. I chose Subversive SVN Team Provider, which was the right thing to do. Confirm, OK, license, you know the drill… restart.

But this still does not give you any SVN options, I don’t remember how IDEA detects SVN on the project and just offers it, but I definitely don’t remember any of torturing like this. Let’s try Subversive documentation – I have no problem reading. Or watching Getting started video linked therein. 🙂

And here we go – perspectives! I wonder how other IDEs can do it without reshuffling your whole screen. Whatever, Eclipse == perspectives, let’s live with it. But why should I add repository when the URL to it is already somewhere in the .svn directory in the root of the project? Switching to SVN Repository Exploring perspective, Eclipse asked for SVN connector. Oh, so much flexibility. Let’s use SVN Kit 1.8.11, ok, ok, license, ok. Restart again? No, not this time, let’s wait until it installs more and do it at once. This was wrong decision this time.

I followed the video to add the SVN repository, but it failed not having the connector. Were I not writing this as I go, I’d probably remember I have installed one. 🙂 But I wasn’t sure, maybe I cancelled it, so let’s try SVN Kit, sounds good. It failed with “See error log for details.” Ok, I give up, let’s try Native JavaHL 1.8.14 (also by Polarion). This one works. Restart needed. Oh, … this time I rather really restarted as my mistake dawned on me.

I checked the list of installed software, but SVN plugins don’t seem to be there – this is rather confusing. But if you go to Windows/Preferences, in the tree choose Team/SVN, then tab SVN Connector – there you can find the connectors you can use. Sure I had both of them there. My fault, happy ending.

So I added SVN repository, but as the Getting started video went on, I knew I’m in trouble. It doesn’t say how to associate existing SVN project on my disk with a repo. I found the answer (stackoverflow of course). Where should I right click so that Team menu shows enabled Share project…? In Package explorer of course. I added one project, hooray! Now it shows SVN information next to it. But I noticed there is Share projects…, I don’t want to do it one by one, right? Especially when Eclipse does not show the projects in the natural directory structure (this sucks). I selected it all my projects but Team menu didn’t offer any Share at all now!

Ok, this would throw me out of balance at 20, but now I know that what can go wrong goes wrong. That only project associated with SVN already – I had to deselect to let Eclipse understand what I want. Strictly speaking there is some logic in eliminating that menu item, but as a user I think otherwise. So now we are SVN ready after all!

I updated the project (not using other perspective), no information whether it did something or not – IDEA shows you updated files without getting into your way. Should have used synchronize, I know…

Oh, and it’s lunch time, perfect – I really need a break.

Quick Diff

This one is for free with IDEA, with Eclipse we have to turn it on. It’s that thing showing you changes in a sidebar (or gutter).

Windows/Preferences, filter for “quick” and there you see it under General/Editors/Text Editors. Well it says enabled, but you want to check Show differences in overview ruler too. In my case I want to change colours to IDEA-ish yellow/green/red. (Who came with these Sun/enterprise violetish colours for everything in Eclipse?) What to use as reference source? Well, with IDEA there is no sense for “version on disk” option. I chose SVN Working Copy Base in hope it does what I think it does (shows my actual outgoing changes that are to be committed).

Outgoing changes contain unmanaged files!

Ah yeah, I recall something like this. This is the most stupid aspect of SVN integration – it does not respect how SVN work. After seeing my outgoing changes in Team Synchronizing perspective (probably my least favourite and most confusing one for me) I was really afraid to click on Team/Commit… But as the three dots indicate, there is one more dialog before any damage is done – and only managed files are checked by default. So commit looks good, but disrespect of outgoing changes to the SVN underlying principles is terrible. Eclipse users will tell you to add files to ignore, but that is just workaround – you can then see in the repository all the files people needed to ignore for the sake of stupid team synchronization. Just don’t auto-add unmanaged files, show some respect!

Eclipse Code Style options

With quick diff ready I can finally start tuning my formatter. There are some good news and some bad news. Well, these are no news actually, nothing has changed since 2011. Code Style in IDEA is something you can set up for many languages in IDEA. It also includes imports. In Eclipse when you filter for “format” in Preferences you see Formatter under Java/Code Style and more options for XML/XML Files/Editor. These are completely separated parts and you cannot save them as one bunch. For Imports you have Java/Code Style/Organize Imports.

In my case it doesn’t make sense to use project specific settings. What I change now will become workspace specific, which is OK with me – but only because I don’t want to use Eclipse for any other projects with different settings (that would probably either kill me or I’d have to put them into separate workspaces).

And then we have Java/Code Style/Clean Up configuration (this is what Source/) and Java/Editor/Save Actions to configure and put into project(s) as well. Plenty of stuff you need to take care of separately.

Line wrapping and joining

One of the most important thing we do with our code in terms of readability is line wrapping – and one thing I expect from any formatter is an option that respects my line breaks. Eclipse offers “Never join lines” on Line Wrapping and Comment tab. It seems you have to combine them with “Wrap where necessary” option for most options on Line Wrapping tab, but it still does not allow you to split line arbitrarily – it joins the lines back on reformat, to be precise.

Sometimes I want to say “wrap it HERE” and keep it that way. In IDEA I can wrap before = in assignment or after – and it respects it. I don’t know about any specific line-break/wrapping option for this specific case. Eclipse respects the wrap after, but not the one before – it re-joins the lines in the latter case. Sure I don’t mind, I prefer the break after = as well. But obviously, Eclipse will not respect my line breaks that much as IDEA.

Just to be absolutely clear, I don’t mind when a standalone { is joined to the previous line when rules say so. There are good cases when control structures should be reformatted even when wrapped – but these are cases revolving mostly around parentheses, braces or brackets.

When I investigated around “Never join lines” behaviour I also noticed that people complain about Eclipse Mars formatter when compared to Luna one. Do they rewrite them all the time or do they just make them better? Do they have good tests for them? I don’t know. Sure, formatters are beasts and we all want different things from them.

Exporting Eclipse settings

Let’s say you select top right corner link Configure Project Specific Settings… in particular settings (e.g. Organize Imports). This opens dialog Project Specific Configuration – but do I know what is the scope of it when I select my top-level project? Actually – I don’t even see my top level project (parent POM in our case), only subset of my open projects based on I don’t know what criteria. That is ridiculous.

I ended up exporting settings using Export All… button – but you have to do it separately for whatever section you want. In our case it’s Clean Up, Formatter, Organize Imports and Save Actions. I simply don’t know how to do it better. I’ll add these XML exported configs into SVN, but everybody has to import them manually.

IDEA with its project (where project is really a project in common sense) and modules (which may be Maven “project”, but in fact just a part of the main project) makes much more sense. Also, in IDEA when you copy the code style into the project you feel sure we’re talking about all of the code style. If I add it to our SVN everybody can use it.

You can also export Code Style as XML – but a single one. Or you can export all of IDE settings and choose (to a degree) what portions you want to import. While this is also manual solution you need to do it once with a single exported config.

(This situation is still not that bad as with keybinds where after all these years you still can’t create your own new Scheme in a normal fashion inside the Eclipse IDE.)

Conclusion

Maybe the way of Go language, where formatting is part of the standard toolchain, is the best way – although if it means joining lines anywhere I definitely wouldn’t like it either.

I can bash IDEA formatter a bit too. For me it’s perfectly logical to prefer annotations for fields and methods on separate line, but NOT wrapping them when they are on the same line. Just keep the damn lines a bit different when I want it this way! Something like soft format with prefered way how to do the new code. This is currently not possible all the way. I can set IDEA formatter in such a way that it keeps annotations on separate lines and respects them at the same line as well – but all the new code has annotations by default on the same line.

This concept combining “how I prefer it” with “what I want to keep preserved even if it’s not the way I’d do it” is generally not the way formatters work now. I believe it would be a great way how they should work. This can partially be solved by formatting only changed lines, but that has its own drawbacks – especially when the indentation is not unified yet.