Pre-christmas technology review 2017

It’s been two years since my last technology overview and I feel the need for another one. Especially after otherwise pretty weak blog-year.

Notebook HP Stream 14” with 32GB eMMC (“Snow White”) (verdict: 2/5)

I got this for my son to teach him some HTML or similar entry-level computer stuff. He loves doing it. However, even nearly bare pre-installed Windows is not able to update itself (Anniversary Update I guess) with over 7GB free disk. And there is no way to make it any free-er. At least not without some hacking. There is just a single local user (admin) because any other user takes additional gigabyte(s), especially with Microsoft account, OneDrive, etc. There is only a couple of apps, I’ve got Visual Studio Code (175 MB) and K-Lite Codec Pack (111 MB).

That’s it. Virtually no data except some pretty tiny HTML files we work on and some pictures (few megas). Anything else is beyond my control. I tried compressing OS. I wanted to switch to Linux, but I’ve read eMMCs are not well supported there (in general). The disk is also incredibly slow – don’t let the fact it’s a flash drive fool you. It’s not SSD. Simply said, it’s something slow connected over a super-slow interface.

I believe there are devices where it’s acceptable, but please, let’s make it official. Notebook with eMMC is just a stupid idea. Especially with 32 gigs and Windows on top of it – that should be considered a crime. Or a scam at least.

Without the problem combining Windows with eMMC, I’d take the machine as OK for the price. It’s light, thin, looks good, runs long on battery (5-8 hours no problem), bright enough display. But it simply does not update anymore. Shame.

Next day update: It didn’t boot the next time and kept finishing in the BSoD. Recovery didn’t work anymore, although I’ve never touched any recovery partition that was there. Windows bootable USB didn’t help (you need to upgrade from your booted system and install didn’t work because there wasn’t free room anymore). Ubuntu 17.10 installed, but booted to black screen – however after enabling Compatibility support module (CSM) in BIOS (F10 to get there), it booted fine.

Bluetooth speaker Bose SoundLink Mini II (5/5)

I wanted a speaker for a party and I wanted it on short notice (not that I needed it, just wanted). I was surprised how many options there is for Bluetooth speaker, but Bose’s one was available and I’d seen it couple of weeks before – and heard it too. Rather pricey just under 200 Eur, but I took the risk.

I’m not sure how much the speaker is responsible for “catching” the bluetooth signal but sometimes I had to keep it pretty close to the device (especially with my HP ProBook). I settled on using a mobile phone. What I love is both way control – I can pause or skip the song using speaker’s button. I also like how it speaks at me and it was easy to pair it with devices.

And then there is the sound. Perhaps too deep to my liking, but definitely satisfying. I expect the speaker to play nicely when quiet – that is I hear enough music details yet I can talk with people around. This is that kind of speaker. Second revision also has micro USB on the speaker itself in case you don’t have the cradle with you (or it gets broken or what) which is nice. Some say it’s not powerful enough, but for me it is more than adequate. No regrets.

Windows package manager Chocolatey (5/5)

I encountered Chocolatey when I was experimenting with Windows unattended installation for VirtualBox. Since then I use it and it’s one of the first things I install on fresh computers. Then I just check my notes about setting up Windows (warning, it’s a live document for personal use, not a blogpost) and copy paste commands to install what I want. This is what good operating system should have at its core – and in a simple matter. Sure you can add/remove things with various PowerShell commands, but the situation there is messy – some features are available only on Enterprise edition so it simply does not do what any Linux package manager does. Chocolatey is a nice plaster for this hole in Windows. Nuff said.

CSS grid layout

Shortly – go and try it. It’s awesome, especially for responsive design when combined with media queries. I first heard about it in this video (highly recommended). While playing with it I also used this tutorial. While still just a candidate recommendation it’s widely supported with recent browsers (although there are some problems with IEs where it originally came from). And you can make it fallback gracefully, just in case. I’m no expert, but this is technology not only for experts – and that’s good.

JavaScript framework ExtJS 4.x (3/5)

I encountered this JavaScript/UI framework on the most recent project and it was selected for us long time ago. It for sure is comprehensive and you can achieve a lot with it – that’s why the verdict went all the way up to 3. I simply can’t deny the results one can do with it. But for us it was too much. It all starts with incredibly steep learning curve. And you pay dearly for anything you don’t understand. Talking about paying – it’s commercial and the license is pretty expensive. It may be good for all-in ExtJS shops, but not for occasional project where people may change and they need to learn it.

Story of our project? We started using Architect, then some “raw JS” devs walked through and original Sencha workspace was unusable and it was coded by hand since then until I came and saw the resulting mess. During the development it produced off-1px glitches like missing right borders of panels until you ran the Sencha CMD (build) that fixes it – but it takes a long time. In general, the whole process is pretty heavyweight and we couldn’t figure out the way how to build the result into completely separate directory tree (e.g. target when Maven exec plugin was used) without breaking some path references.

So you probably can do a lot with it, but it’s all-in or rather not at all. It’s proprietary and I’d rather invest in some mainstream OSS framework instead. That would be better for both my career and future evolution and maintenance of the project too.

Note: Current version is 6.5.x.

Fitness band Xiaomi Mi Band 2 (4/5)

I bought this step counter (as that was the main reason I wanted it) just before previous Christmas and it worked well for the purposes I wanted it for. It counted steps. How precise it is? I personally think it doesn’t matter. For couple of days I carried some Garmin device on the same hand and it counted 12k steps where Mi Band counted 10k. But this doesn’t really matter. On your average day 10k is more than 8k, you know which one is better. On days you do something completely different you can’t compare the number with the day with different routine. But both devices counted actual steps just fine – the difference happens in between the long walks – like during typing on the computer. Mi Band also clearly differentiates between walking and running as it indicates more energy output for the same step count. So far so good, that’s what I wanted.

How good is Mi Band 2 as a “smart band”? I don’t know. I don’t use it to measure my heart rate as it doesn’t work that well while running anyway. I don’t have it constantly paired with a mobile phone either. Only occasionally I pair it with a phone with the application. Ah, the application – it runs only on newer Androids, my old Samsung Galaxy 3S was out of question. It also doesn’t run on tablets in general, neither on PC. That just sucks. I wasn’t able to connect the application and my Xiaomi account either. Instructions are unclear or unavailable or something fails – I haven’t tried since then, I simply use the app on the phone to see the history and don’t care about the bigger picture (yet). I saw recommendations to use it with different app (like Google Fit or so).

Finally – after roughly a half year the device started to fall out of the wristband. Happens to many people so it seems. I was quite lucky I always picked the device up before I bought some cheap replacements. These feel similar – they are rougher for sure but after a while I don’t feel bothered by them at all. I like they feel firmer than the original, but it’s been just 3 months yet so I’ll see whether they last longer or not. But for this price I have no problem to replace them often.

So, why not 5/5? The device was pretty expensive in Slovakia a year ago (50 Eur or so), it got into normal range later. The application is not available on many devices (not running on tablets for arbitrary reasons?!) and it didn’t connect easily (=at all for me) with an online account. Display is good but not in strong sunshine. Otherwise it does what I want – but let’s face it, I don’t want that much from it. 🙂

Amazon Prime (verdict cancelled)

OK, this was rather a hasty affair. Lasted less than an hour for me. I tried to use my amazon.com account for that – and this is the first gripe. Amazon should seriously help us to decide what account we should (or even could) use for the full benefit in a particular country. Prime in Slovakia is official only for couple of weeks (if I understand it correctly) but I have no idea whether I should use COM, co.uk or DE account for it. I tried COM then – trial is for free, shouldn’t hurt.

I’m trying a link from registration email to a movie included with the membership. Does not work in my region. So much for the confusion of the customer. They seriously should employ their AI to offer me available films. There is no chance in understanding all the limitations before you actually start the trial.

Then I tried Amazon Music – as the music is by far the most important thing for me. The site asked for billing information – why? Amazon has it already! Funny that their video site didn’t. But I filled it in… and the wait indicator just keeps spinning and spinning. Hm, after a while I reloaded and retried – the same result. I couldn’t go further without providing the info and they didn’t accept it (or reject for that matter). Terrible first experience. Anyway… as I browsed around it dawned upon me that there is this Music Prime (included) and Music Unlimited – for additional 8 Eur/month.

Is even my kind of music available on Prime? I don’t need a million of songs… couple of thousand of the right ones would be enough. Let’s check. Our newest hit in the family – Steven Wilson! Nope… Pink Floyd? Nope. Yes? Mike Oldfield? U2?! Unlimited only… I couldn’t find anything I liked on Prime!

I could as well cancel the trial period immediately. And I did. (It will run out after a month, but after that the service will be cancelled.) Shame, as I’m rather a fanboy, especially when it comes to what they do with AWS. But first the Music site must actually work, especially at at its literally first step and then offer some reasonable music for reasonable price. Total price, not add-on price.

I feel no urge to buy Echo Dot for christmas either… at least not this year. The whole landscape moves pretty fast anyway and I’m curious where it all is next December.

Advertisements

Converting 96kHz 24-bit FLAC to OGG with ffmpeg

Lately my son Robin asked for Peter Gabriel’s song The Tower That Ate People in a car. I like OGGs, although recently it may have been pointless with MP3 patents being expired. But 15+ years ago it was an obvious choice for me, especially because most encoded MP3 files had also clearly cut out high frequencies and generally lower quality at the same bitrate. Again – not a problem I encountered with newer MP3s. But I stayed true to OGG and I honestly don’t need anything better than its Q7 level.

The song is on Peter’s OVO album but the version Robin likes is from Back to Front show in London. So I browsed it, played it and – all the songs were skipped. Darn! I knew it must be because of the quality being very high because the digital download, companion to the Blu-ray Deluxe Book Edition (yeah, I’m a fan), was in 96kHz for both FLAC and OGG. So I had to recode the OGG, or better FLAC to OGG in normal sample rate (44.1kHz).

FFmpeg for the rescue!

I previously transcoded OGGs to MP3 for a little radio that didn’t support OGGs (I never understand why this happens) and I was very satisfied with FFmpeg because when I can do something from a command line I prefer that. So today I downloaded Windows build of FFmpeg and tried to figure out the switches.

After some Googling I tried -codec:a libvorbis and it told me there is no such a codec. So I tried ffmpeg -codecs to find out what (and if) there is any OGG support. There was just vorbis decoder, so I tried that one. Then ffmpeg told me that it’s just experimental and I must add -strict -2 switch to enable it. It worked afterwards but the warning was strange so I investigated further.

The trouble was that the build from FFmpeg site did not have libvorbis compiled in. Every time you run ffmpeg it prints the configuration it was compiled with and mine didn’t show –enable-libvorbis in the output. It was by an accident I found out I’ve got ffmpeg also on my PATH – which was strange considered I didn’t put the downloaded version there. It was part of ImageMagick which I was pretty sure was installed using Chocolatey (most recommended!), I don’t even remember why. But now it came handy, because, behold, this one had libvorbis with it!

If you have Chocolatey already, just cinst -y imagemagick and then start a new console to find ffmpeg on your path. Or do it the hard way.

Those damn spaces!

I use spaces in the filenames, replacing them with underscores or something does not make much sense, not in 21st century I believe. I respect bash (and prefer it in Windows as well, as delivered by Git) and I consider myself more or less a power-user (more less than more I guess). I understand that long time ago text was the thing and objects were not. But all this white-space escaping is sometimes killing me. Just look at all the effort that went into escaping white-spaces – IFS, quoting, print0, etc.

Actually, using NUL character (print0) as a separator seems most logical but obviously it’s difficult to put it into plain text then. But plain text is so awkward to represent anything anyway (except for the actual text). I believe some richer environment where lists are true lists is the logical thing to have. I’m not hinting on PowerShell necessarily, not sure they have it right, but they tried to be progressive for sure.

When I quote the name containing spaces on the input it’s a single argument (let’s say $1). But when I use ffmpeg -i “$1″… in the script the program complains that the first word from the filename is not a valid name. I encountered this problem many times before, passing the arguments from a command line to the script and there to other commands. Today I learned that “${1}” is different from “$1”. I always used curlies only to separate name of a variable from potentially colliding surrounding. But the first one keeps $1 as a single parameter even for another executable called from a script. Handy. Not intuitive. And definitely not something you learn in this section, for instance.

If this was all more “object-oriented” (broader meaning) it would be a filename, String or even File object from the start all the way to where it should be used. Spaces would not matter.

Sample rate and unexpected “video” stream

Because the source flac file had sampling rate of 96kHz – and I suspected this was the main reason the car audio system didn’t play it – I wanted to resample the audio to more traditional CD quality. That’s what option -ar 44100 does. Because OGG seems to have a single sample format, I didn’t have to care about bringing 24bits down to 16.

But I was surprised that my OGG didn’t play in foobar2000 and loading it actually created two entries in a playlist. I checked the output of a command more carefully and noticed it also converted some JPEG image embedded in that FLAC to a “video” stream. Not interested, thank you, said I – and that’s what -vn (no video) switch does.

And the script is…

Add setting the quality of the output OGG and -y to overwrite the output (I experimented repeatedly, you may not want it, of course) and you get a script like this:

#!/bin/sh

ffmpeg.exe -i "${1}" -ar 44100 -vn -codec:a libvorbis -qscale:a 7 -y "${1%flac}ogg"

It only encodes one file. Last thing I wanted is to treat input arguments for a for loop, although I guess I could have used shift too. Anyway, the command is easy:

find . -name \*.flac -exec ./anything-to-ogg-44k1-q7.sh {} \;

I guess it doesn’t care about the input format as long as it recognizes it, hence the “anything”. Of course, ffmpeg can do much more – I just wanted to show one recipe, that’s all.

Building Windows VirtualBox machines

I started this post in January originally, but after a couple of paragraphs I realized I’m writing a more generic post – Believe in build automation. Now you know why I believe in automation and we can get straight to it. I’m a Linux guy, I’d rather work with Linux, I always prefer UNIX/Linux on servers, but I run Windows desktop to be conformant. After all I can run anything in VirtualBox when I need it.

And sometimes what I need is just another Windows. But I don’t want to manually prepare the box all over again after it expires (evaluation), not to mention I want a repeatable process (because I believe in it :-)). You may snapshot your virtual machines, but you cannot avoid eventual end of evaluation period.

State of affairs in Windows automation

Couple of years ago I got a new computer with Windows and I wanted to put all my favourite tools on it. Of course I didn’t have a list. But I had a feeling I’m repeating myself. I also wanted to disable some Windows features. I had experimented with PowerShell before, so I turned to it with faith. I found out that there are some PowerShell modules that allow to add/remove features or applications, but they are limited only to Windows Server. Couple of ugly words ran through my head and I postponed my dream.

Now I know this was a hasty decision, because Microsoft does not offer just one good standard way how to do it. As explained here, you can use one of two PowerShell modules. ServerManager module was the one that made a bit angry because of its restrictions to server versions of Windows, but there is also Dism module available on any recent platform, not to mention dism.exe itself, that works for older Windows incarnations as well.

While this all is just a minor episode, it documents how difficult it may be to find the right way how to perform various tasks on the command line (and preferably PowerShell) for a newcomer. And I wasn’t even that new on Windows.

But after this it was easier and easier to use the right words and ask the right questions the right way to get my answers. Most of them were on StackOverflow, but I have to praise Microsoft’s sites too. Sure, sometimes you have to go over couple of Microsoft pages, but in overall you can find the answers.

Back to Windows virtual machine creation

Here you go, I nearly did it again! Wrote a different post than I wanted, that is. So back to the topic. Of course, you need to know your options for automation, so learning more about PowerShell and about ways how to (un)install various Windows features is still important. But we also need to know the general workflow how to bring Windows virtual machine to life. I decided to use Vagrant because it aims for developers and is praised by them.

When it comes to Windows there is one big trouble – because Windows is big. The same trouble exists for Linux too, but is smaller. We’re talking about automated installation. Good news is that both systems can be installed automatically. It comes as no surprise for Linux, but Windows also features so-called “unattended installation” which aims for corporate world where admins don’t want to sit through installations for all the computers in a big company.

It works in a simple way – you provide an XML file with this unattended configuration for the computer and Windows finds it during installation. It can be found on a floppy or USB drive.

I don’t know all the options how to install Windows in an automatic fashion, but this one is good for virtual machines. All you need to do is provide the booting virtual machine DVD with Windows ISO image and Autounattend.xml file on a virtual floppy disk.

Can Vagrant do it? Ok, now you got me. 🙂 Probably it can, but a brief investigation on the Internet revealed that instead of doing this installation with Vagrant I should first use Packer. And – what a surprise – both tools are developed by the very same company/guy (HashiCorp/Mitchell Hashimoto, sorry for leaving out any other participants from the company). I was not the only one confused about the differences between Vagrant and Packer.

Shortly (and maybe not 100% precise), Packer is good in creating virtual machine base images and Vagrant is good to use them in your development process. Base image is not installation ISO, it’s rather a snapshot of a virtual machine after installation with everything you want to have there for the start. Not with everything possible though, that’s why it’s called base image. Packer can build base images for various virtualization platforms, but we will focus on VirtualBox only.

My idol Matt Wrock

I decided to install the Windows somehow automatically using Vagrant and prefer PowerShell as much as possible. Searching for a solution I somehow found the post called In search of a light weight windows vagrant box by Matt Wrock. I read a bit of it but I was also intrigued by the link to the updated version using Packer and Boxstarter. Already by this time I was overflowed by new terms, but it was worth it, I promise!

Matt definitely knows his stuff, his Windows and automation experience is extensive and he can explain it properly as well. Just a day before that I wasn’t ready to add Packer to my arsenal, not to mention Boxstarter – and soon I learned about Chocolatey as well. Now, honestly, I still don’t get Boxstarter, so for me it’s just “some extension for PowerShell” (disregard at will), but I absolutely fell in love with Chocolatey, because with it the management of programs feels like on Linux.

Matt’s instructions how to use Packer and Boxstarter were pretty cool, he provides Packer files (configurations, or sort of recipes, for Packer, written in JSON format) for Windows Server 2012, Windows Nano (very interesting addition to Microsoft’s arsenal) and Windows 7 (here you need license key, as there is no evaluation ISO, shame). I definitely utilized Windows Server 2012R2, as the server edition always comes handy during development when you want to experiment with a domain controller, etc. But I also wanted packer template for Windows 10 – and I had to create the one myself.

Windows 10 experiments

Actually, the biggest problem with Windows 10 wasn’t the Packer template, but with the Autounattend.xml file. I found some generator, but it didn’t deliver without some errors. I’m still pretty sure that XML is far from flawless, it’s not cleaned up properly and so on – but it works. Diving into every detail in a field that is mostly new for me (the whole world of Windows automation) would probably stop me before I got to the result, so take it as it is, or make it better if you can.

I highly recommend to read that Matt’s article Creating windows base images using Packer and Boxstarter as it is a very good introduction into the whole pipeline. His packer templates also provided a great starting point for more experiments. I also liked the way how he minimized the images by removing many Windows components, defragging the disk, zeroing empty space, etc.

I summed up my experiments in a markdown file and looking at it, it definitely is not perfect and finished. But does it have to be? In a week I played with it I probably installed Windows 10 forty times. Most of these test I commented out the slow parts that were mere optimization steps mentioned above (minimizing the size of the image). I played a lot with some preinstalled software (using Chocolatey, of course) and tried to pre-configure it using registry changes where necessary. This is, however, in vain, as the sysprep.exe step wipes registry changes for vagrant user. Talking about vagrant user, be extra careful to spell it everywhere the same way. Once I messed it up, had vagrant in Autounattend.xml and Vagrant in postunattend.xml (which is used as C:\Windows\Panther\Unattend\unattend.xml by sysprep.exe) and had two Vagrant accounts – you don’t want that. 🙂

I tried hard to perform some installation and configuration steps after sysprep, I tried to change it from Packer’s shutdown_command to windows-restart provisioner step but I wasn’t able to overcome some errors. After a while I settled with a script I just copied to my Vagrant working environment directory and then ran it from initialized box where it appeared in c:\vagrant directory.

Sure I could do even better with full automation, but when things resist too much sometimes it’s good to step back, rethink the strategy and focus on quick wins.

Other options?

There are definitely more ways how to prepare Windows 10 box, or any Windows for that matter, with or without Packer, but even when we focus on Packer solutions there’s a wide spectrum of approaches. Some don’t bother to use sysprep.exe to generalize their installation – after all if it’s only for personal needs it really is not needed. People on GitHub seem to agree on using PowerShell as a Packer provider, but one of the solutions used no provider at all (all part of the build section). Also widen the search for other Windows versions and you’ll see much more variability.

Conclusion

Using Packer as a first step in the pipeline is very practical. You can prepare base image once and save a couple of hours any time you need a fresh environment (partially you can do the same with snapshots in VirtualBox, but it’s not the same).

I use Boxstarter in the process as recommended (although I’m not able to appreciate it fully) and Chocolatey to install/remove programs – during Packer steps and also anytime later. When my evaluation Windows runs out I simply refresh the packer image from ISO and I’m done for the next 90 or 180 days (depending on the OS version).

Following the installation enthusiasm I went on to install SQL Server 2014 Express from Chocolatey package, configure it using PowerShell bits and pieces found across many blogs and stackoverflow questions and wrote it down on GitHub. I actually got used to writing technical notes I may need later into these markdown files and it works very well for me. Now I have automation hiatus, but I’m sure I’ll get back to it and it’s good to find all the notes at hand, including unresolved problems and ideas.

Good luck with Windows automation!