Bridgetown2022-12-04T08:49:22+01:00https://jkraemer.net/atom.xmlJens KrämerJens Krämer is a freelance software developer with more than 20 years of experience in web development and Linux systems administration. He is author of several Ruby libraries, contributor to the Redmine project and creator of several Redmine plugins.Debian Buster on a Thinkpad X2702020-02-29T03:08:00+01:002020-02-29T03:14:00+01:00repo://posts.collection/_posts/2020-02-29-debian-buster-on-a-thinkpad-x270.md<h2 id="tldr">TL;DR</h2>
<p>It just works, except for the Intel WiFi which requires proprietary firmware. Use an <a href="https://cdimage.debian.org/cdimage/unofficial/non-free/images-including-firmware/current-live/amd64/iso-hybrid/">‘unofficial’ image</a> that includes the non-free firmware stuff and you’re all set.</p>
<h2 id="the-what-and-why">The what and why</h2>
<p>I have this fine Notebook for nearly two years now, but so far somehow got along with the pre-installed Windows 10 pro.</p>
<p>First, I tried out this <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">WSL thing</a>, and it just worked. It basically gave me a Debian shell on Windows, which is really neat and doesn’t leave much to be desired if you’re working with <code class="highlighter-rouge">tmux</code> and <code class="highlighter-rouge">vim</code> 80% of the time anyway. Due to relatively poor filesystem performance (which lead to really poor git performance) at some point I started to use a Debian machine in Virtual Box instead for most things, but somehow still sticked to Windows for reason number two:</p>
<p>The Macbook Air which I used for photo and video processing was getting too cramped for handling both of these tasks. So I had started to use Lightroom on the X270 instead, and since I don’t like to spread out my raw files on external disks, there simply wasn’t enough room left on the 1TB SSD to add a Linux partition of any useful size.</p>
<p>But then, I got an iPad pro and it very quickly became the only device I wanted to do any kind of video or photo processing on. You’re probably not here for reading praise of Apple products, but let me say just this: the display is gorgeous, there’s plenty of processing power, Lumafusion blows iMovie out of the water and Lightroom on iPad, while still limited feature-wise, is getting better with every release. The iPad pro is just <em>the</em> device for doing the kind of video / photo editing I’m doing. In fact, I’m sometimes using it for <em>real</em> work as well, but that’s another blog post actually. Back to Linux on the Thinkpad X270.</p>
<h2 id="uefi--dual-boot-setup">UEFI / Dual boot setup</h2>
<p>I found <a href="https://www.ctrl.blog/entry/dual-boot-bitlocker-device.html">this</a> to be a good introduction for how to tackle this whole dual boot / UEFI / Bitlocker situation. Turns out it’s not all that bad, in fact I did not have any problems other than being able to…</p>
<h2 id="shrink-that-windows-partition">Shrink that windows partition</h2>
<p>It did not work. I followed any advice out there but got stuck with Windows refusing to shrink the system partition by more than a measly 7.5GB.</p>
<p>If you’re trying to do the same, two things:</p>
<ol>
<li>Check out <a href="https://superuser.com/questions/1017764/how-can-i-shrink-a-windows-10-partition/1175556#1175556">this answer on superuser.com</a> which conveniently has power shell commands for everything while most other answers to the problem send you on a click-hunt through Windows’ settings dialogs, some of which still look like straight out of Windows NT.</li>
<li>Good luck!</li>
</ol>
<p>At some point I gave up, made sure my backups were OK, disabled Bitlocker, and hoped the Debian Installer would do a better job with that.</p>
<h2 id="booting-buster">Booting Buster</h2>
<p>I put a <a href="https://cdimage.debian.org/debian-cd/current/amd64/bt-cd/">Gnome-Live CD Image</a> on a USB stick and fired it up - that worked like a charm, except, uh, no WiFi. Thank you Intel for requiring some non-free firmware blob to make your WiFi chips work.</p>
<p>An ethernet cable came in handy (and of course the fact that this thing has a real ethernet port), but the second time around (see below…) I resorted to an <a href="https://cdimage.debian.org/cdimage/unofficial/non-free/images-including-firmware/current-live/amd64/iso-hybrid/">‘unofficial’ image</a> that includes the non-free firmware stuff.</p>
<p>I was impressed with the whole Gome 3 look and feel. Previously I felt Gnome and KDE were constantly getting in my way to the point that I ran <a href="https://www.bunsenlabs.org">Bunsenlabs</a> with Openbox on my last Thinkpad but this time with Gnome 3 I think it’s fine. Maybe it’s me getting old, maybe it’s really that much better, I don’t know.</p>
<h2 id="first-installation-attempt">First installation attempt</h2>
<p>In the left-hand application launcher of the Live CD Desktop is a link to a graphical installer which I promptly used. Language, timezone, keyboard, ah, partitioning. Fancy looking UI, I resized the Windows partition, created a new one for <code class="highlighter-rouge">/</code> in the resulting space, entered an encryption passphrase and told it to mount the 260GB UEFI partition to <code class="highlighter-rouge">/boot/efi</code>. <em>Next</em> and it started to work on that. I went away from the machine, assuming this would take a while. When I came back few minutes later, damn, the installer window was gone.</p>
<p>What happened? I have no idea, but I restarted the installer, and it looked like the partitioning went well. I continued using the new partition layout as is and couple minutes later I booted Debian from the notebook’s SSD. Even Windows still worked although it complained about some file system problems which were easily fixed by a checkdisk run.</p>
<p>Installing <code class="highlighter-rouge">firmware-iwlwifi</code> allowed me to move from crouching in front of the router back to sitting at my desk, and this was that. I set up a fully encrypted Debian along Windows 10 (encrypted as well after re-enabling Bitlocker) in less than an hour.</p>
<h2 id="problems">Problems</h2>
<p>Of course, there is always <em>something</em> to complain about, so here we go:</p>
<h3 id="slow-boot-process-due-to-encrypted-boot">Slow Boot Process due to Encrypted /boot</h3>
<p>It bothered me quite a bit that I had to enter the LUKS passphrase before the Grub screen. Not only is it useless in case I want to boot into Windows, it also took ages (as in, more than ten seconds) to unlock the disk. Apparently that is a problem with encrypting <code class="highlighter-rouge">/boot</code> and Grub using some low-tech hashing implementations that are <em>much</em> slower than what is available later in Linux land.</p>
<p>It is possible to somewhat fix that by reducing the number of iterations used for the passphrase, but that also reduces the strength of the whole encryption. Also there would still be the general problem of having to enter the LUKS passphrase even when booting into Windows.</p>
<p>Since the merits of encrypting <code class="highlighter-rouge">/boot</code> are debatable anyway (someone able to mess with your kernel in <code class="highlighter-rouge">/boot</code> can most probably as well mess with your boot loader in the still unencrypted EFI partition) I decided to start over, this time setting up a separate <code class="highlighter-rouge">/boot</code> that would stay unencrypted, in the hope this would fix both the slow boot time and the timing when I actually had to enter the LUKS passphrase.</p>
<h3 id="hibernation--no-swap">Hibernation / no Swap</h3>
<p>Another reason for starting over was that I didn’t create a swap partition the first time, totally forgetting that hibernation usually depends on that. While it appears to be possible to <a href="https://wiki.debian.org/Hibernation/Hibernate_Without_Swap_Partition">hibernate to a swap file</a>, that seemed to be a bit more manual setup work than I was willing to invest here for something that would (hopefully) just work if only I had a swap partition.</p>
<h2 id="second-attempt">Second Attempt</h2>
<p>In the meantime I had learned that this particular setup with the encrypted boot is the result of using the Gnome based <em>Calamares</em> installer that I used. Second time around I decided to use the graphical installer offered in the boot menu of the live CD image.</p>
<p>I created two partitions, one for <code class="highlighter-rouge">/boot</code> and one to be encrypted which would then serve as a physical volume for LVM. On top of that, one logical volume for swap, and one for the root partition holding the system itself. Since this is just a personal development environment I didn’t bother with setting up separate volumes for <code class="highlighter-rouge">/home</code> etc.</p>
<p>In theory, one could layer the encryption on top of LVM instead, but that would lead to separate decryption prompts for each partition at boot time which I wanted to avoid. There are ways to work around that (essentially by unlocking one partition and getting the keys for the other(s) from that), but why make things more complicated than necessary.</p>
<h2 id="the-verdict">The Verdict</h2>
<p>Everything works like a charm, including fancy things like all the thinkpad specific <code class="highlighter-rouge">Fn</code> key combinations for volume, screen brightness, think light and keyboard illumination.</p>
<p>Really the only difficult thing was figuring out how to get the WiFi firmware blob, everything else was standard Debian installation procedure. This is the third Thinkpad I’m installing Debian or one of it’s derivatives on, and while I always managed to make everything work as intended, it’s been a much more pleasant experience this time around.</p>Faster Redmine Search With PostgreSQL Trigram Indexes2018-08-01T03:24:00+02:002018-08-01T03:24:00+02:00repo://posts.collection/_posts/2018-08-01-faster-redmine-search-with-postgresql-trigram-indexes.md<p>Here’s an easy way to speed up Redmine’s search if you’re using PostgreSQL.</p>
<h2 id="how-redmine-search-works">How Redmine Search Works</h2>
<p>If you know Redmine, you know there’s a slew of different kinds of objects, from projects and issues over commit messages and news to forum messages and wiki pages. Each of these data types is stored in their own table. When you run a search, Redmine builds SQL queries that look like this:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="k">DISTINCT</span> <span class="nv">`issues`</span><span class="p">.</span><span class="nv">`created_on`</span><span class="p">,</span> <span class="nv">`issues`</span><span class="p">.</span><span class="nv">`id`</span>
<span class="k">FROM</span> <span class="nv">`issues`</span>
<span class="k">INNER</span> <span class="k">JOIN</span> <span class="nv">`projects`</span> <span class="k">ON</span> <span class="nv">`projects`</span><span class="p">.</span><span class="nv">`id`</span> <span class="o">=</span> <span class="nv">`issues`</span><span class="p">.</span><span class="nv">`project_id`</span>
<span class="k">WHERE</span> <span class="p">(</span><span class="n">projects</span><span class="p">.</span><span class="n">status</span> <span class="o"><></span> <span class="mi">9</span>
<span class="k">AND</span> <span class="k">EXISTS</span> <span class="p">(</span><span class="k">SELECT</span> <span class="mi">1</span> <span class="k">AS</span> <span class="n">one</span> <span class="k">FROM</span> <span class="n">enabled_modules</span> <span class="n">em</span> <span class="k">WHERE</span> <span class="n">em</span><span class="p">.</span><span class="n">project_id</span> <span class="o">=</span> <span class="n">projects</span><span class="p">.</span><span class="n">id</span>
<span class="k">AND</span> <span class="n">em</span><span class="p">.</span><span class="n">name</span><span class="o">=</span><span class="s1">'issue_tracking'</span><span class="p">))</span>
<span class="k">AND</span> <span class="p">(</span><span class="n">issues</span><span class="p">.</span><span class="n">project_id</span> <span class="k">IN</span> <span class="p">(</span><span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">))</span>
<span class="k">AND</span> <span class="p">((</span>
<span class="p">(</span><span class="n">subject</span> <span class="k">LIKE</span> <span class="s1">'%query%'</span><span class="p">)</span> <span class="k">OR</span> <span class="p">(</span><span class="n">issues</span><span class="p">.</span><span class="n">description</span> <span class="k">LIKE</span> <span class="s1">'%query%'</span><span class="p">)</span>
<span class="p">))</span>
<span class="k">ORDER</span> <span class="k">BY</span> <span class="nv">`issues`</span><span class="p">.</span><span class="nv">`created_on`</span> <span class="k">DESC</span><span class="p">,</span><span class="nv">`issues`</span><span class="p">.</span><span class="nv">`id`</span> <span class="k">DESC</span>
</code></pre></div></div>
<p>As you can see, besides checking for project ids and status and whether the project actually has the <em>Issue Tracking</em> module enabled, the query has <code class="highlighter-rouge">LIKE</code> statements for the subject and description fields of the <code class="highlighter-rouge">issues</code> table.</p>
<p>That is just the query for issues, there will be one like this for every entity you are searching through, plus one for custom field values, and if you have ever worked with relational databases you’re starting to see the problem:
Search is done using <code class="highlighter-rouge">LIKE '%query%'</code>, and such queries usually cannot make use of any indexes, leading to a full scan of all rows that match the other criteria in the query. Ouch!</p>
<h2 id="how-to-make-it-faster">How to make it faster?</h2>
<p>There are a few possible solutions to the problem:</p>
<h3 id="fulltext-search">Fulltext search</h3>
<p>You could rework the way search is implemented and use a fulltext indexing engine, either built into your database (MySQL and PostgreSQL both can do that), or an external engine like Solr or ElasticSearch.</p>
<p>I took the DB-Engine approach with the <a href="https://jkraemer.net/redmine-plugins#redmine-postgresql-full-text-search">Redmine PostgreSQL Full Text Search Plugin</a> before.
It works great in some scenarios where it can return more relevant search results in shorter times, but it comes with a few problems as well.
First of all, it does not support multi-lingual data sets well because you have to decide on a language definition for tokenization / stemming when configuring it.
Whatever language you pick, chances are words from another language will get the wrong treatment, leading to false results.</p>
<p>This approach in general does not play nicely with new data types introduced by other plugins and, because rewriting / monkey patching of the core search is necessaary, is brittle when core Redmine changes anyhting about the search in new versions.</p>
<h3 id="get-rid-of-the-postfix-part-in-the-like">Get rid of the postfix part in the LIKE</h3>
<p>A somewhat creative solution to the problem is to have two conventional B-tree indexes per column, i.e. one over <code class="highlighter-rouge">subject</code>, and one over the reverse value of it.
Searching for <code class="highlighter-rouge">subject LIKE 'foo%' OR reverse(subject) LIKE 'oof%'</code> will then lead to fast index based searches results.</p>
<p>One problem here is that MySQL cannot index expressions but only ‘real’ columns, so you have to actually store the reverse value in it’s own column, which, together with the additional index, will roughly double the size of your database.
Regardless of the database being used, Redmine’s search code will have to be changed or monkey patched, leading to the same problems with other plugins and Redmine updates I already mentioned above.</p>
<h3 id="trigram-indexes">Trigram Indexes</h3>
<p>A better approach would be to find a way to speed up that exact query on the database side, without having to change anything about the way Redmine search is implemented.
That would solve the performance issue, while keeping full compatibility with other plugins and future Redmine updates.</p>
<p>Guess what, PostgreSQL indeed comes with a solution to this problem: The trigram index, implemented in the <a href="https://www.postgresql.org/docs/9.6/static/pgtrgm.html">pg_trgm module</a>.
<a href="https://en.wikipedia.org/wiki/Trigram">Trigrams</a> are the result of breaking up text into 3-letter groups like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>redmine=# select show_trgm('query');
show_trgm
---------------------------------
{" q"," qu",ery,que,"ry ",uer}
</code></pre></div></div>
<p>After enabling the <code class="highlighter-rouge">pg_trgm</code> extension for your DB, you can create a trigram index like that:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">index_issues_on_description_trgm</span>
<span class="k">ON</span> <span class="n">issues</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">description</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
</code></pre></div></div>
<p>That’s it.
From now on, whenever there’s an <code class="highlighter-rouge">issues.description LIKE '%query%'</code> kind of query, that index can and will be used by PostgreSQL to speed up the search.</p>
<p>Armed with that knowledge, you’re ready to…</p>
<h2 id="speed-up-your-redmine-search-in-two-minutes">Speed Up Your Redmine Search in Two Minutes</h2>
<p>All you have to do is add the <code class="highlighter-rouge">pg_trgm</code> extension to your database and create trigram indexes for all the columns that are queried by Redmine search:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="n">EXTENSION</span> <span class="s1">'pg_trgm'</span><span class="p">;</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_changesets_on_comments_trgm</span>
<span class="k">ON</span> <span class="n">changesets</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">comments</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_documents_on_title_trgm</span>
<span class="k">ON</span> <span class="n">documents</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">title</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_documents_on_description_trgm</span>
<span class="k">ON</span> <span class="n">documents</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">description</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_issues_on_subject_trgm</span>
<span class="k">ON</span> <span class="n">issues</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">subject</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_issues_on_description_trgm</span>
<span class="k">ON</span> <span class="n">issues</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">description</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_projects_on_name_trgm</span>
<span class="k">ON</span> <span class="n">projects</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">name</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_projects_on_identifier_trgm</span>
<span class="k">ON</span> <span class="n">projects</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">identifier</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_projects_on_description_trgm</span>
<span class="k">ON</span> <span class="n">projects</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">description</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_messages_on_subject_trgm</span>
<span class="k">ON</span> <span class="n">messages</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">subject</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_messages_on_content_trgm</span>
<span class="k">ON</span> <span class="n">messages</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">content</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_news_on_title_trgm</span>
<span class="k">ON</span> <span class="n">news</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">title</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_news_on_summary_trgm</span>
<span class="k">ON</span> <span class="n">news</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">summary</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_news_on_description_trgm</span>
<span class="k">ON</span> <span class="n">news</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">description</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_attachments_on_filename_trgm</span>
<span class="k">ON</span> <span class="n">attachments</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">filename</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_attachments_on_description_trgm</span>
<span class="k">ON</span> <span class="n">attachments</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">description</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_wiki_pages_on_title_trgm</span>
<span class="k">ON</span> <span class="n">wiki_pages</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="n">title</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
<span class="k">CREATE</span> <span class="k">INDEX</span> <span class="n">CONCURRENTLY</span> <span class="n">index_wiki_contents_on_text_trgm</span>
<span class="k">ON</span> <span class="n">wiki_contents</span> <span class="k">USING</span> <span class="n">gin</span> <span class="p">(</span><span class="nb">text</span> <span class="n">gin_trgm_ops</span><span class="p">);</span>
</code></pre></div></div>
<p>There’s also a <a href="https://gist.github.com/jkraemer/a812cbbd711f98a995009d82c9e6f95b">rake task</a> which generates and runs the above statements.
Just put the file into <code class="highlighter-rouge">lib/tasks</code> of your Redmine installation, run <code class="highlighter-rouge">bundle exec rake redmine:pg_trgm:setup</code> and you’re done.</p>
<p>Note: While not strictly necessary I added the <code class="highlighter-rouge">concurrently</code> keyword which will run the actual index creation in the background.
Thus, on large datasets, it might take a few minutes or more until the indizes are actually available.</p>
<p>If you are using any plugins that come with their own searchable data stuctures, you can of course add similar indexes for these as well.</p>The Proper Way to Add a Project Settings Tab for Your Redmine Plugin2018-05-10T04:00:00+02:002018-11-15T02:59:00+01:00repo://posts.collection/_posts/2018-05-10-add-a-project-settings-tab-for-your-redmine-plugin.md<p><img src="screenshot-2018-5-1-settings-my-new-project-redmine-1!full" alt="" /></p>
<p>Redmine’s project settings consist of a series of tabs for things like general project information, project members, issue categories and so on.
If your plugin has any project specific configuration of any kind, it might be a good thing to add an entry there for it, as well. But how? Let’s see how the row of tabs is rendered:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">project_settings_tabs</span>
<span class="n">tabs</span> <span class="o">=</span> <span class="p">[{</span><span class="ss">:name</span> <span class="o">=></span> <span class="s1">'info'</span><span class="p">,</span> <span class="ss">:action</span> <span class="o">=></span> <span class="ss">:edit_project</span><span class="p">,</span> <span class="ss">:partial</span> <span class="o">=></span> <span class="s1">'projects/edit'</span><span class="p">,</span> <span class="ss">:label</span> <span class="o">=></span> <span class="ss">:label_information_plural</span><span class="p">},</span>
<span class="p">{</span><span class="ss">:name</span> <span class="o">=></span> <span class="s1">'modules'</span><span class="p">,</span> <span class="ss">:action</span> <span class="o">=></span> <span class="ss">:select_project_modules</span><span class="p">,</span> <span class="ss">:partial</span> <span class="o">=></span> <span class="s1">'projects/settings/modules'</span><span class="p">,</span> <span class="ss">:label</span> <span class="o">=></span> <span class="ss">:label_module_plural</span><span class="p">},</span>
<span class="c1"># [ I cut a couple similar-looking lines...]</span>
<span class="p">{</span><span class="ss">:name</span> <span class="o">=></span> <span class="s1">'activities'</span><span class="p">,</span> <span class="ss">:action</span> <span class="o">=></span> <span class="ss">:manage_project_activities</span><span class="p">,</span> <span class="ss">:partial</span> <span class="o">=></span> <span class="s1">'projects/settings/activities'</span><span class="p">,</span> <span class="ss">:label</span> <span class="o">=></span> <span class="ss">:enumeration_activities</span><span class="p">}</span>
<span class="p">]</span>
<span class="n">tabs</span><span class="p">.</span><span class="nf">select</span> <span class="p">{</span><span class="o">|</span><span class="n">tab</span><span class="o">|</span> <span class="no">User</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">allowed_to?</span><span class="p">(</span><span class="n">tab</span><span class="p">[</span><span class="ss">:action</span><span class="p">],</span> <span class="vi">@project</span><span class="p">)}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>This method resides in the <code class="highlighter-rouge">ProjectsHelper</code> module and as such is automatically available in any views rendered by <code class="highlighter-rouge">ProjectsController</code>. At first sight, not very extension-friendly, especially when compared to the other menus in Redmine which make it very easy to add new entries through the <code class="highlighter-rouge">MenuManager</code> interface.</p>
<h2 id="old-and-busted-alias_method_chain">Old and busted: alias_method_chain</h2>
<p>Many plugins extend that method using the <code class="highlighter-rouge">alias_method_chain</code> mechanism like this:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">IssueTemplates</span>
<span class="k">module</span> <span class="nn">ProjectsHelperPatch</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">included</span><span class="p">(</span><span class="n">base</span><span class="p">)</span>
<span class="n">base</span><span class="p">.</span><span class="nf">class_eval</span> <span class="k">do</span>
<span class="n">alias_method_chain</span> <span class="ss">:project_settings_tabs</span><span class="p">,</span> <span class="ss">:issue_templates</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">project_settings_tabs_with_issue_templates</span>
<span class="n">tabs</span> <span class="o">=</span> <span class="n">project_settings_tabs_without_issue_templates</span>
<span class="n">action</span> <span class="o">=</span> <span class="p">{</span> <span class="ss">name: </span><span class="s1">'issue_templates'</span><span class="p">,</span>
<span class="ss">controller: </span><span class="s1">'issue_templates_settings'</span><span class="p">,</span>
<span class="ss">action: :show</span><span class="p">,</span>
<span class="ss">partial: </span><span class="s1">'issue_templates_settings/show'</span><span class="p">,</span>
<span class="ss">label: :project_module_issue_templates</span> <span class="p">}</span>
<span class="n">tabs</span> <span class="o"><<</span> <span class="n">action</span> <span class="k">if</span> <span class="no">User</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">allowed_to?</span><span class="p">(</span><span class="n">action</span><span class="p">,</span> <span class="vi">@project</span><span class="p">)</span>
<span class="n">tabs</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>That example stems from the <a href="https://github.com/akiko-pusu/redmine_issue_templates/blob/master/lib/issue_templates/projects_helper_patch.rb">issue templates</a> plugin and was slightly shortened for clarity.</p>
<p>So why is this bad? <code class="highlighter-rouge">alias_method_chain</code> calls once could were all over the place in the Rails code base, but nowadays it is frowned upon. Deprecated in Rails 4, removed from Rails 5 (which Redmine 4 will be using), this is definitely not the way to go. Of course you could ship your own implementation of <code class="highlighter-rouge">alias_method_chain</code> or simply do the aliasing by hand, but luckily there’s a better way. But first, another wrong approach that looks good at first:</p>
<h2 id="maybe-prepend">Maybe prepend()?</h2>
<p>Instead of using aliasing, we can simply override the <code class="highlighter-rouge">project_settings_tabs</code> method:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">SomePlugin</span>
<span class="k">module</span> <span class="nn">ProjectsHelperPatch</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">apply</span>
<span class="k">unless</span> <span class="no">ProjectsHelper</span> <span class="o"><</span> <span class="no">InstanceMethods</span>
<span class="no">ProjectsHelper</span><span class="p">.</span><span class="nf">prepend</span> <span class="no">InstanceMethods</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">module</span> <span class="nn">InstanceMethods</span>
<span class="k">def</span> <span class="nf">project_settings_tabs</span>
<span class="n">tabs</span> <span class="o">=</span> <span class="k">super</span>
<span class="k">if</span> <span class="no">User</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">allowed_to?</span><span class="p">(</span><span class="ss">:some_permission</span><span class="p">,</span> <span class="vi">@project</span><span class="p">)</span>
<span class="n">tabs</span><span class="p">.</span><span class="nf">push</span><span class="p">({</span>
<span class="ss">name: </span><span class="s1">'some_plugin_settings'</span><span class="p">,</span>
<span class="ss">partial: </span><span class="s1">'projects/settings/some_plugin'</span><span class="p">,</span>
<span class="ss">label: :label_some_plugin</span>
<span class="p">})</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">tabs</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Looks nice and makes use of <code class="highlighter-rouge">prepend</code> which is awesome and modern, right? But in this case it’s not a good choice.</p>
<p>Why? Because modifying <code class="highlighter-rouge">ProjectsHelper</code> (or any other rails helper for that matter) may or may not lead to the desired result, depending on the fact what code has already been loaded and what not.
If you manage to change the helper module early enough (to be precise: before the corresponding controller is loaded), your changed method will make it into the views, but if you are to late, your modification will still be successful but views will use the ‘original’ version of the method.
Since your plugin will probably have to co-exist with a bunch of other plugins, and you never know what these do (i.e. it’s not so uncommon to extend <code class="highlighter-rouge">ProjectsController</code> - once another plugin does that, and it’s loaded before yours, it’s game over for your helper patch), you cannot rely on this.</p>
<p>The fact that <code class="highlighter-rouge">prepend</code> is ineffective when the module has been included already but method aliasing still works has to do with how Ruby internally handles these things - let’s just say <code class="highlighter-rouge">prepend</code> creates a new version of the module under the old name, while method aliasing just switches around handles to methods in the existing module.</p>
<p>So, to come to the point of this article, what <em>should</em> you do to add your plugin’s project settings tab?</p>
<h2 id="declare-a-new-helper">Declare a new helper</h2>
<p>It’s that simple. Just declare a new helper and tell the controller to make use of it:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">SomePlugin</span>
<span class="k">module</span> <span class="nn">ProjectSettingsTabs</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">apply</span>
<span class="no">ProjectsController</span><span class="p">.</span><span class="nf">send</span> <span class="ss">:helper</span><span class="p">,</span> <span class="no">SomePlugin</span><span class="o">::</span><span class="no">ProjectSettingsTabs</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">project_settings_tabs</span>
<span class="n">tabs</span> <span class="o">=</span> <span class="k">super</span>
<span class="k">if</span> <span class="no">User</span><span class="p">.</span><span class="nf">current</span><span class="p">.</span><span class="nf">allowed_to?</span><span class="p">(</span><span class="ss">:some_permission</span><span class="p">,</span> <span class="vi">@project</span><span class="p">)</span>
<span class="n">tabs</span><span class="p">.</span><span class="nf">push</span><span class="p">({</span>
<span class="ss">name: </span><span class="s1">'some_plugin_settings'</span><span class="p">,</span>
<span class="ss">partial: </span><span class="s1">'projects/settings/some_plugin'</span><span class="p">,</span>
<span class="ss">label: :label_some_plugin</span>
<span class="p">})</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="n">tabs</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>As you can see, the implementation of the new <code class="highlighter-rouge">project_settings_tabs</code> method is identical to the previous approach, including the call to <code class="highlighter-rouge">super</code>.</p>
<p>By explicitly referring to the <code class="highlighter-rouge">ProjectsController</code> class we make sure it, and by extension <code class="highlighter-rouge">ProjectsHelper</code>, is loaded.
Since we do not have control what code other plugins load, our safest bet is to load whatever concerns us to create a known state and go on from there.
Depending on code <em>not</em> being loaded on the other hand is never going to work reliably.</p>
<p>The call to super in this case works because the view context, at the time our helper is included, already holds the implementation of <code class="highlighter-rouge">project_settings_tabs</code> from Redmine’s <code class="highlighter-rouge">ProjectsHelper</code>. <code class="highlighter-rouge">super</code> simply refers to the ‘old’, already existing method which we are replacing, even if it comes from another module.</p>
<p><em>Note:</em> In order for the above patch to be effective, you still have to call the <code class="highlighter-rouge">apply</code> method in a <code class="highlighter-rouge">to_prepare</code> block in <code class="highlighter-rouge">init.rb</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="no">Rails</span><span class="p">.</span><span class="nf">configuration</span><span class="p">.</span><span class="nf">to_prepare</span> <span class="k">do</span>
<span class="no">SomePlugin</span><span class="o">::</span><span class="no">ProjectSettingsTabs</span><span class="p">.</span><span class="nf">apply</span>
<span class="k">end</span>
</code></pre></div></div>Bye, Flickr!2018-04-27T04:44:00+02:002018-04-27T04:55:00+02:00repo://posts.collection/_posts/2018-04-27-bye-flickr.md<p><img src="elbe-1!big" alt="River Elbe, Dresden, Germany" title="The first photo I ever uploaded on Flickr." /></p>
<p>I’ve been using Flickr for 11 years.</p>
<p>In the beginning I really liked it, but unfortunately things went downhill from there. I stayed with them through the acquisition by Yahoo (which definitely didn’t do Flickr any good), through user interface changes that can be called debatable at best (endless scrolling, anyone?). I even coped with having to log in through Yahoo, forever having to input my username which definitely is not an email address in a form field labelled ‘email’. Try to explain that to your Dad whom, at some point in the distant past, you also made a Flickr user.</p>
<p>Nowadays I find myself using the Flickr website only when I really have to, just using it for lack of a better option to share photos with friends and family which I uploaded from Lightroom or using the mobile app.</p>
<p>Now <a href="https://blog.flickr.net/en/2018/04/20/together-smugmug-flickr-faq/">Flickr changes owners again</a> and I have a hard time believing that anything will get better soon. Despite what they’re saying now (“There is no plan to merge the products”), in my opinion it’s most likely that in the long run SmugMug will try to migrate users over to their own platform, instead of running two photo platforms in parallel for an indefinite amount of time.</p>
<p>Long story short, I wrote <a href="https://github.com/jkraemer/bye-flickr">an app</a> to download all my photos and any additional information that might be useful from my Flickr account. It’s written in Ruby and available as a Gem so installation is just a straightforward <code class="highlighter-rouge">gem install bye-flickr</code>. All that you have to do is to create an API key on the flickr website. Checkout the <a href="https://github.com/jkraemer/bye-flickr/blob/master/README.md">Readme</a> which has all the details.</p>
<p>With that done and 26GB of photos and JSON files sitting on my disk, the next step is to generate a static site from all that data, along with a plan for how to add future content. More about that another time.</p>Fully Encrypted Headless Debian Stretch Setup2018-04-01T00:58:00+02:002019-09-06T06:51:00+02:00repo://posts.collection/_posts/2018-04-01-fully-encrypted-headless-debian-stretch-setup.md<p>Here’s my field notes from the server I recently set up. It’s a rental server at <a href="http://hetzner.de">Hetzner</a>, which comes with a very basic <em>rescue system</em> pre-installed.</p>
<p>Starting from there, I followed the steps outlined <a href="https://gist.github.com/jkullick/ceb0ba4f4478b48fd5f3f14581a70a8a">here</a>. The differences between the Ubuntu version used there and Debian Stretch kept me busy quite a bit so I’m writing my adopted command chain down here to save you some time.</p>
<h2 id="partition-and-encrypt-the-disks">Partition and Encrypt the disks</h2>
<p>Hetzner servers usually come with at least 2 disks which are partioned identically with 128MB for /boot (which stays unencrypted), and the rest for the encrypted system.</p>
<p>On top of that two Raid 1 arrays are formed, <code class="highlighter-rouge">md0</code> for <code class="highlighter-rouge">/boot</code> and <code class="highlighter-rouge">md1</code> for everything else.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sgdisk <span class="nt">-og</span> /dev/sda
sgdisk <span class="nt">-n</span> 1:2048:+256M <span class="nt">-t</span> 1:fd00 /dev/sda
sgdisk <span class="nt">-n</span> 128:-3M:0 <span class="nt">-t</span> 128:ef02 /dev/sda
sgdisk <span class="nt">-n</span> 2:0:0 <span class="nt">-t</span> 2:fd00 /dev/sda
sgdisk <span class="nt">-R</span> /dev/sdb /dev/sda
sgdisk <span class="nt">-G</span> /dev/sdb
partprobe
mdadm <span class="nt">--create</span> /dev/md0 <span class="nt">--metadata</span><span class="o">=</span>0.9 <span class="nt">--level</span><span class="o">=</span>1 <span class="nt">--assume-clean</span> <span class="nt">--raid-devices</span><span class="o">=</span>2 /dev/sd[ab]1
mdadm <span class="nt">--create</span> /dev/md1 <span class="nt">--metadata</span><span class="o">=</span>1.2 <span class="nt">--level</span><span class="o">=</span>1 <span class="nt">--assume-clean</span> <span class="nt">--raid-devices</span><span class="o">=</span>2 /dev/sd[ab]2
</code></pre></div></div>
<p>Now, <code class="highlighter-rouge">md1</code> will be encrypted, and on top of that encrypted device we set up our root and other partitions using LVM.
I tend to start with reasonable but small partition size and leave most space in the LVM volume group unassigned so I have room to create / enlarge partitions later on as needed.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cryptsetup <span class="nt">-c</span> aes-xts-plain <span class="nt">-y</span> <span class="nt">-s</span> 512 <span class="nt">-h</span> sha512 luksFormat /dev/md1
cryptsetup luksOpen /dev/md1 debian
pvcreate /dev/mapper/debian
vgcreate debian-vg /dev/mapper/debian
lvcreate <span class="nt">-L</span> 20GB <span class="nt">-n</span> root debian-vg
lvcreate <span class="nt">-L</span> 16GB <span class="nt">-n</span> swap debian-vg
lvcreate <span class="nt">-L</span> 40GB <span class="nt">-n</span> home debian-vg
lvcreate <span class="nt">-L</span> 100GB <span class="nt">-n</span> var debian-vg
mkfs.ext2 <span class="nt">-L</span> boot <span class="nt">-I</span> 128 /dev/md0
mkfs.ext4 <span class="nt">-L</span> root /dev/debian-vg/root
mkfs.ext4 <span class="nt">-L</span> home /dev/debian-vg/home
mkfs.ext4 <span class="nt">-L</span> var /dev/debian-vg/var
mkswap <span class="nt">-L</span> swap /dev/debian-vg/swap
mount /dev/debian-vg/root /mnt
<span class="nb">mkdir</span> /mnt/<span class="o">{</span>boot,var,home<span class="o">}</span>
mount /dev/md0 /mnt/boot
mount /dev/debian-vg/var /mnt/var
mount /dev/debian-vg/home /mnt/home
</code></pre></div></div>
<h2 id="set-up-a-basic-debain-stretch-with-debootstrap">Set Up a Basic Debain Stretch With Debootstrap</h2>
<p>The most recent version of debootstrap at the time you are doing this may vary, adjust accordingly.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget http://ftp.de.debian.org/debian/pool/main/d/debootstrap/debootstrap_1.0.89_all.deb
ar <span class="nt">-xf</span> debootstrap_1.0.89_all.deb
<span class="nb">tar</span> <span class="nt">-xf</span> data.tar.gz <span class="nt">-C</span> /
debootstrap <span class="nt">--arch</span><span class="o">=</span>amd64 stretch /mnt http://ftp.de.debian.org/debian
</code></pre></div></div>
<p>Next, edit or create a couple config files.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># make sure resolv.conf is correct (mine was prefilled with Hetzner DNS servers) or do:</span>
<span class="nb">echo</span> <span class="s2">"nameserver 8.8.8.8"</span> <span class="o">></span> /mnt/etc/resolv.conf
<span class="nb">echo</span> <span class="s2">"test"</span> <span class="o">></span> /mnt/etc/hostname
<span class="nb">echo</span> <span class="s2">"debian UUID=</span><span class="si">$(</span>blkid <span class="nt">-s</span> UUID <span class="nt">-o</span> value /dev/md1<span class="si">)</span><span class="s2"> none luks"</span> <span class="o">></span> /mnt/etc/crypttab
<span class="nb">cat</span> <span class="o">></span> /mnt/etc/fstab <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
proc /proc proc defaults 0 0
UUID=</span><span class="si">$(</span>blkid <span class="nt">-s</span> UUID <span class="nt">-o</span> value /dev/md0<span class="si">)</span><span class="sh"> /boot ext2 defaults 0 0
UUID=</span><span class="si">$(</span>blkid <span class="nt">-s</span> UUID <span class="nt">-o</span> value /dev/mapper/debian--vg-root<span class="si">)</span><span class="sh"> / ext4 defaults 0 1
UUID=</span><span class="si">$(</span>blkid <span class="nt">-s</span> UUID <span class="nt">-o</span> value /dev/mapper/debian--vg-var<span class="si">)</span><span class="sh"> /var ext4 defaults 0 2
UUID=</span><span class="si">$(</span>blkid <span class="nt">-s</span> UUID <span class="nt">-o</span> value /dev/mapper/debian--vg-home<span class="si">)</span><span class="sh"> /home ext4 defaults 0 2
UUID=</span><span class="si">$(</span>blkid <span class="nt">-s</span> UUID <span class="nt">-o</span> value /dev/mapper/debian--vg-swap<span class="si">)</span><span class="sh"> none swap defaults 0 0
</span><span class="no">EOF
</span><span class="nb">echo</span> <span class="s2">"deb http://security.debian.org/ stretch/updates main"</span> <span class="o">>></span> /mnt/etc/apt/sources.list
</code></pre></div></div>
<p>And change into the fresh Debain system with <code class="highlighter-rouge">chroot</code>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mount <span class="nt">-o</span> <span class="nb">bind</span> /dev /mnt/dev
mount <span class="nt">-o</span> <span class="nb">bind</span> /dev/pts /mnt/dev/pts
mount <span class="nt">-t</span> sysfs /sys /mnt/sys
mount <span class="nt">-t</span> proc /proc /mnt/proc
<span class="nb">chroot</span> /mnt
</code></pre></div></div>
<p>Set a password, create some symlinks for the Raid devices and create mtab. I’m not sure if the links are actually necessary.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>passwd
<span class="nb">mkdir</span> /dev/md
<span class="nb">ln</span> <span class="nt">-s</span> /dev/md0 /dev/md/0
<span class="nb">ln</span> <span class="nt">-s</span> /dev/md1 /dev/md/1
<span class="nb">cp</span> /proc/mounts /etc/mtab
</code></pre></div></div>
<p>Upgrade the installed packages and install a couple more:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt update
apt upgrade <span class="nt">-y</span>
<span class="nv">DEBIAN_FRONTEND</span><span class="o">=</span>noninteractive apt <span class="nb">install</span> <span class="nt">-y</span> <span class="se">\</span>
vim <span class="se">\</span>
linux-base <span class="se">\</span>
linux-image-amd64 linux-headers-amd64 <span class="se">\</span>
grub-pc <span class="se">\</span>
mdadm <span class="se">\</span>
cryptsetup <span class="se">\</span>
lvm2 <span class="se">\</span>
initramfs-tools <span class="se">\</span>
openssh-server <span class="se">\</span>
busybox <span class="se">\</span>
dropbear <span class="se">\</span>
locales <span class="se">\</span>
net-tools
dpkg-reconfigure locales <span class="c"># select en_US.UTF-8</span>
</code></pre></div></div>
<p>Make sure you can actually log into the new system by putting your SSH public key into <code class="highlighter-rouge">/root/.ssh/authorized_keys</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"YOUR PUBLIC SSHKEY HERE"</span> <span class="o">></span> /root/.ssh/authorized_keys
</code></pre></div></div>
<p>Last but not least, set up networking so your system actually goes online after unlocking. I tried to use the <a href="https://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_the_modern_network_configuration_without_gui">modern way</a>, but for some reason it didn’t work, so I went with the classic <code class="highlighter-rouge">/etc/network/interfaces</code> config style. Next hurdle was to find out the (not so) <em>predictable network interface name</em> the kernel assigns to my server’s ethernet adaptor. I got it from the syslog after accessing the system through the rescue image (see below for the steps to access the machine in case you cannot get into it after reboot).</p>
<p>It might also be possible to force the Kernel to use the old <code class="highlighter-rouge">eth0</code> style names using the <code class="highlighter-rouge">net.ifnames=0</code> kernel boot parameter (<a href="https://serverfault.com/questions/856850/predictable-network-interface-names-in-systemd#856872">link to serverfault</a>).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o">></span> /etc/network/interfaces.d/lo <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
auto lo
iface lo inet loopback
</span><span class="no">EOF
</span><span class="c"># your device name may be different</span>
<span class="nb">cat</span> <span class="o">></span> /etc/network/interfaces.d/enp0s31f6 <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
auto enp0s31f6
iface enp0s31f6 inet static
address YOUR.IPv4.ADDRESS
netmask YOUR.IPv4.NETMASK
gateway YOUR.IPv4.GATEWAY
iface enp0s31f6 inet6 static
address YOUR.IPv6.ADDRESS
netmask YOUR.IPv6.NETMASK
gateway YOUR.IPv6.GATEWAY
</span><span class="no">EOF
</span></code></pre></div></div>
<h2 id="set-up-the-initial-ramdisk-for-remote-password-entry">Set Up the Initial Ramdisk for Remote Password Entry</h2>
<p>Since the whole system (except <code class="highlighter-rouge">/boot</code>) is encrypted, you will have to enter the passphrase at boot time.
This is accomplished by embedding the <code class="highlighter-rouge">dropbear</code> SSH server into the initial ramdisk image, which allows you to SSH into the server before it even finished booting to enter the passphrase.
Luckily this is all easy and straight forward to set up nowadays:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sed</span> <span class="nt">-i</span> <span class="s2">"s/NO_START=1/NO_START=0/"</span> /etc/default/dropbear
<span class="nb">echo</span> <span class="s2">"DEVICE=eth0"</span> <span class="o">>></span> /etc/initramfs-tools/initramfs.conf
<span class="nb">sed</span> <span class="nt">-i</span> <span class="s2">"s/^#CRYPTSETUP=</span><span class="nv">$/</span><span class="s2">CRYPTSETUP=y/"</span> /etc/cryptsetup-initramfs/conf-hook
</code></pre></div></div>
<p>For logging in via SSH, install your SSH key, again. You can use the one you just put into <code class="highlighter-rouge">/root/.ssh/authorized_keys</code>, as long as it is an RSA key. Other key types (I tried my ed25519 key) will most certainly not work.
The options before the key limit what can be done when logging in with that key, and the <code class="highlighter-rouge">command</code> is what will be executed directly after log in.
Since the sole purpose of this is to unlock the disk, the script which does just that is launched directly after log in.</p>
<p>To prevent dropbear from starting when it isn’t needed (that is, outside the initrd in your ‘real’ system), run <code class="highlighter-rouge">systemctl disable dropbear</code>. Thanks to Thomas pointing this out in the comments.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s2">"no-port-forwarding,no-agent-forwarding,no-X11-forwarding,command=</span><span class="se">\"</span><span class="s2">/bin/cryptroot-unlock</span><span class="se">\"</span><span class="s2"> YOUR PUBLIC RSA SSHKEY"</span> <span class="o">></span> /etc/dropbear-initramfs/authorized_keys
</code></pre></div></div>
<p>In order to avoid SSH complaining about changed host keys every time you unlock the server or connect to it when unlocked there are two ways: run Dropbear on a different port than you run the ‘normal’ SSH server on the unlocked system on, or use a separate <code class="highlighter-rouge">known_hosts</code> file for unlocking.
I chose to run Dropbear on another port and set some other options while I’m at it:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sed</span> <span class="nt">-i</span> <span class="s2">"s/^#DROPBEAR_OPTIONS=</span><span class="nv">$/</span><span class="s2">DROPBEAR_OPTIONS=</span><span class="se">\"</span><span class="s2">-p 2323 -s -j -k -I 60</span><span class="se">\"</span><span class="s2">/"</span> /etc/dropbear-initramfs/config
</code></pre></div></div>
<p>With all that done, it’s time to update the initrd and install the Grub bootloader. The <code class="highlighter-rouge">ip</code> kernel boot parameter is essential for setting up the network in the initial ram disk, otherwise dropbear won’t be reachable. Be sure to replace the placeholders with whatever IPv4 config your server has.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>update-initramfs <span class="nt">-u</span> <span class="nt">-k</span> all
<span class="nb">echo</span> <span class="s2">"GRUB_CMDLINE_LINUX_DEFAULT=</span><span class="se">\"</span><span class="s2">ip=IPv4IP::GATEWAY:NETMASK</span><span class="se">\"</span><span class="s2">"</span> <span class="o">>></span> /etc/default/grub
grub-install /dev/sda
grub-install /dev/sdb
update-grub
<span class="c"># exit the chroot and reboot</span>
<span class="nb">exit
</span>umount /mnt/<span class="o">{</span>boot,var,home<span class="o">}</span>
<span class="nb">sync
</span>swapoff <span class="nt">-L</span> swap
reboot
</code></pre></div></div>
<p>At this point, you should be able to connect to the server on the configured port and immediately be asked for your passphrase. Enter it, press enter and the connection is closed, while the server continues booting.</p>
<h2 id="it-doesnt-work">It doesn’t work!</h2>
<p>You cannot connect to unlock the system, or you unlock it but cannot connect after that? Don’t panic.</p>
<p>With Hetzner, there is always a way to reboot the machine into the rescue system, which can then be used to fix things.</p>
<p>After you logged into the system, follow these steps to unlock disks and get into the Debian again:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cryptsetup luksOpen /dev/md1 debian
vgchange <span class="nt">-a</span> y
mount /dev/debian-vg/root /mnt
mount /dev/md0 /mnt/boot
mount /dev/debian-vg/var /mnt/var
mount /dev/debian-vg/home /mnt/home
mount /dev/debian-vg/srv /mnt/srv
mount /dev/debian-vg/data /mnt/opt/data
mount <span class="nt">-o</span> <span class="nb">bind</span> /dev /mnt/dev
mount <span class="nt">-o</span> <span class="nb">bind</span> /dev/pts /mnt/dev/pts
mount <span class="nt">-t</span> sysfs /sys /mnt/sys
mount <span class="nt">-t</span> proc /proc /mnt/proc
<span class="nb">chroot</span> /mnt
</code></pre></div></div>
<p>I needed these quite a few times while fixing my network config. Once you’re ready to give it another go, do the same steps as above, but skip the Grub installation:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>update-initramfs <span class="nt">-u</span> <span class="nt">-k</span> all
update-grub
<span class="nb">exit
</span>umount /mnt/<span class="o">{</span>boot,var,home<span class="o">}</span>
<span class="nb">sync
</span>swapoff <span class="nt">-L</span> swap
reboot
</code></pre></div></div>
<p>That’s it, have fun with your shiny new server!</p>Hong Kong Cuts Corporate Tax Rate in Half2017-11-09T03:57:00+01:002018-08-01T03:37:00+02:00repo://posts.collection/_posts/2017-11-09-hong-kong-cuts-corporate-tax-rate-in-half.md<blockquote>
<p>… the profits tax rate for the first $2 million of profits of enterprises will be lowered to 8.25%, or half of the standard profits tax rate, instead of 10% as proposed in my Election Manifesto.<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup></p>
</blockquote>
<p>Now this is how you encourage small to medium size companies to do business in your country.
Also interesting that, contrary to what is usually happening in some other countries, this government does not only do as promised (lower the tax to 10%), they even overdeliver by cutting the rate down to 8.25%.</p>
<p>The new tax scheme is supposed to be implemented by April 2018.
From then on, even if your company for whatever reason fails to secure the 0% tax rate for offshore businesses, it will enjoy one of the lowest corporate tax rates of the world<sup id="fnref:taxrates" role="doc-noteref"><a href="#fn:taxrates" class="footnote" rel="footnote">2</a></sup> for the first HKD 2 million (USD 250,000 at the time of this writing).</p>
<p><img src="view-of-hong-kong-from-victoria-peak!big" alt="Hong Kong" title="Hong Kong has a lot to offer" /></p>
<p>If you think now’s a good time to start up your own Hong Kong Company, check out <a href="http://www.bridges.hk/">Bridges</a>.
I have written about my experience <a href="/2015/10/how-i-set-up-shop-in-hong-kong">setting up a Hong Kong Limited</a> with them back then, and after working with Bridges for 2 years I can still really recommend them.
Having a competent partner at your side makes the whole process a breeze.
Mention I<sup id="fnref:me" role="doc-noteref"><a href="#fn:me" class="footnote" rel="footnote">3</a></sup> sent you and we’ll both get a nice discount.</p>
<p><small>Photo Credit: <a href="https://commons.wikimedia.org/wiki/File:View_of_Hong_Kong_from_Victoria_Peak.jpg">View of Hong Kong from Victoria Peak</a> by <a href="http://flickr.com/photos/53197929@N00">Dennis Tang</a></small></p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p><a href="https://www.policyaddress.gov.hk/2017/eng/policy_ch03.html">The Chief Executive’s 2017 Policy Address</a>: <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:taxrates" role="doc-endnote">
<p><a href="https://home.kpmg.com/xx/en/home/services/tax/tax-tools-and-resources/tax-rates-online/corporate-tax-rates-table.html">KPMG corporate tax rate comparison</a> <a href="#fnref:taxrates" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:me" role="doc-endnote">
<p>That is, Jens Kraemer / Electric Things Limited <a href="#fnref:me" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Redmine 3.4 Is Out!2017-07-13T08:19:00+02:002017-07-13T08:38:00+02:00repo://posts.collection/_posts/2017-07-13-redmine-3-4-is-out.md<p>The new Redmine version is packed with <a href="http://www.redmine.org/versions/119">200 improvements and bug fixes</a>. In this post, I’ll give you an overview of what has changed.</p>
<h2 id="user-interface-improvements">User Interface Improvements</h2>
<p>A lot of work went into improving Redmine’s user interface, with quite a few patches coming from the <a href="https://plan.io/redmine-hosting">Planio</a> team.
My personal favorites are better support for High Resolution (a.k.a. Retina) displays with user avatars and embedded images (<a href="http://www.redmine.org/issues/24922">#24922</a>, <a href="http://www.redmine.org/issues/24927">#24927</a>) and the various improvements regarding the handling and display of attachments:</p>
<ul>
<li>Markup for images dragged into a text area holding wiki-formatted text is now generated automatically (<a href="http://www.redmine.org/issues/26071">#26071</a>)</li>
<li>All Attachment links lead to a preview page now and feature a trailing <em>Download</em> icon for direct download (<a href="http://www.redmine.org/issues/25988">#25988</a>)</li>
<li>The attachment list got a cleaner layout (<a href="http://www.redmine.org/issues/25988">#25988</a>)</li>
<li>Multiple attachments may be removed at once when editing the container object (<a href="http://www.redmine.org/issues/13072">#13072</a>)</li>
</ul>
<p><img src="2017-07-13-11-11-57-scrot!big" alt="Cleaned up attachments list" title="Cleaned up attachments list" /></p>
<p><img src="2017-07-13-11-11-29-scrot!big" alt="Select attachments to be deleted" title="Select attachments to be deleted" /></p>
<p>Another highlight is the search-as-you-type projects dropdown which replaces the old projects select box that wasn’t very helpful with a large number of projects.</p>
<p><img src="2017-07-13-16-31-49-scrot!big" alt="Project search" title="Project search" /></p>
<h2 id="improved-issue-lists-and-querying">Improved Issue Lists and Querying</h2>
<p>For many users working with issues and issue lists is the most important thing in their use of Redmine, so it’s always great to see improvements in this area.</p>
<p>The issue list is now able to display issue attachments in the new <em>Files</em> column (<a href="http://www.redmine.org/issues/25515">#25515</a>), and you may even filter the list for issues having an attachment whose name matches a given substring (<a href="http://www.redmine.org/issues/2783">#2783</a>).
Also, similar to the description, you can now display the <em>Last Comment</em> from the history of each issue.</p>
<p><img src="2017-07-13-15-45-22-scrot!big" alt="Files column in issue list" title="Files column in issue list" /></p>
<p>You may now search for issues <em>Updated by</em> or <em>Last updated by</em> a specific user (<a href="http://www.redmine.org/issues/17720">#17720</a>.
The latter is also available as a display column now and matches only issues where the <em>last</em> update has been made by a given user, while the former will find any issues a given user has updated at any point in their history.</p>
<p>Filtering issues by target version got an upgrade as well - now it is possible to filter by version status (closed or not) and due date, <a href="http://www.redmine.org/issues/23215">#23215</a>).
In addition to that, version custom fields can now be used for issue queries (<a href="http://www.redmine.org/issues/21249">#21249</a>).</p>
<h2 id="time-tracking-enhancements">Time Tracking Enhancements</h2>
<p>Redmine’s time tracking feature is more visible now through it’s own menu tab, and it comes with quite a few improvements:</p>
<ul>
<li>Time entries may now be grouped by user or date, giving a better overview without the need for external tools like Excel (<a href="http://www.redmine.org/issues/16843">#16843</a>)</li>
<li>The state of the <em>Spent Time</em> query form may now be saved for future use in the same way it is already possible for issue queries (<a href="http://www.redmine.org/issues/14790">#14790</a>)</li>
<li>Administrators may choose to make the <em>Issue</em> and <em>Comment</em> fields on time log entries mandatory (<a href="http://www.redmine.org/issues/24577">#24577</a>)</li>
</ul>
<p><img src="2017-07-13-11-46-45-scrot!big" alt="Time entries grouped by date" title="Time entries grouped by date" /></p>
<h2 id="my-page">My Page</h2>
<p>Your personal overview page can now be customized in place (<a href="http://www.redmine.org/issues/25297">#25297</a>).
The ability to embed issue lists powered by custom issue queries (<a href="http://www.redmine.org/issues/1565">#1565</a>) makes this page much more useful in my opinion.
Furthermore, the existing predefined issue lists (watched, assigned and reported issues) may now be sorted at will and changed in terms of which columns are displayed.</p>
<h2 id="under-the-hood">Under the Hood</h2>
<p>Improved handling of user sessions allows you to stay logged in on several devices at the same time (<a href="http://www.redmine.org/issues/10840">#10840</a>). This makes parallel use of Redmine e.g. on a smartphone much more convenient.</p>
<p>Redmine now runs fine with Ruby 2.4 (<a href="http://www.redmine.org/issues/25048">#25048</a>) and it comes with quite a few performance enhancements in specific areas.
Many translations were improved with the help of numerous patches provided by users.</p>
<p>The REST API was extended to allow modification of attachments (<a href="http://www.redmine.org/issues/22356">#22356</a>) and in various places the data returned by API calls was completed / extended (<a href="http://www.redmine.org/issues/22795">#22795</a>, <a href="http://www.redmine.org/issues/7506">#7506</a>, <a href="http://www.redmine.org/issues/12181">#12181</a>, <a href="http://www.redmine.org/issues/23566">#23566</a>).
There’s also a new API for adding files to a project (<a href="http://www.redmine.org/issues/19116">#19116</a>).</p>
<p>In terms of attachment storage, the checksum algorithm used to determine attachment identity was upgraded to SHA256 (<a href="http://www.redmine.org/issues/25240">#25240</a>), and attachments bearing an identical checksum are only stored on disk once, potentially saving a lot of disk space in situations where the same attachments get added multiple times (<a href="http://www.redmine.org/issues/25215">#25215</a>). Both patches are a direct result of my work at <a href="https://plan.io/redmine-hosting">Planio</a>.</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>Redmine 3.4 is a release full of new features and improvements, and as such for sure worth upgrading to as soon as possible.</p>
<p>However, before you upgrade your production environment, be sure to test the new release together with any plugins you are using in a test environment.</p>
<p>If that sounds like an awful lot of work, check out my posts about <a href="https://jkraemer.net/2016/03/deploy-and-maintain-redmine-the-right-way">Deploying and maintaining Redmine the right way</a> and <a href="https://jkraemer.net/2016/04/deploying-redmine-with-capistrano">Deploying Redmine with Capistrano</a>.</p>Let’s Encrypt SSL Certificates With HAProxy and Stable Keys2017-01-08T05:14:00+01:002017-11-01T02:06:00+01:00repo://posts.collection/_posts/2017-01-08-lets-encrypt-ssl-certificates-with-haproxy-and-stable-keys.md<p>I’ve been a (more or less) happy StartSSL customer for years, but since they are going to lose their status as a trusted CA these days <a href="https://en.wikipedia.org/wiki/StartCom">for various reasons</a>, I finally got around to switching to <a href="https://letsencrypt.org">Let’s Encrypt</a>.
Here’s what I did.</p>
<h2 id="the-situation">The Situation</h2>
<p>I have HAProxy running in front of several nginx instances on different (virtual) machines.
HAProxy handles all the TLS stuff.
This is Debian Jessie, so HAProxy is a rather old v1.5.x, but that’s fine with the approach outlined here. No need to install backported / self-compiled newer versions.</p>
<p>I’ve also been using <a href="https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning">HPKP</a> now for a while, which leads to the following requirements:</p>
<ul>
<li>the keys used for generating certificates should be rather stable, since each change requires changes to the HPKP headers</li>
<li>new keys have to be introduced to clients by inclusion in the HPKP header a long time (at least 2 months, in my case) <em>before</em> they are actually used</li>
</ul>
<p>So I had to find a way to use my existing keys (which my sites are ‘pinned’ to) and generate new Let’s Encrypt certificates from them.
Further I wanted to ensure that future (automatic) certificate renewals also reuse the same private keys. Since Let’s Encrypt only issues certificates with 3 months lifetime, automatic renewal and keeping the same keys both really are a <em>must have</em>.</p>
<p>While the official Let’s Encrypt client, <a href="https://certbot.eff.org">Certbot</a>, goes great lengths to automate the whole certificate generation and renewal process, it unfortunately does not provide this comfort in combination with pre-generated, constant keys and CSRs.</p>
<p>What it does, on the other hand, is integrate with webservers to automatically install certificates and also configure the server for using them.
Way too much to put into a single tool for my taste. <a href="https://code.greenhost.net/open/certbot-haproxy">Support for HAProxy</a> is still in its early stages and requires building stuff yourself, so this is not really an option for now, anyway.</p>
<p>In this article, I’ll show how to setup certbot on Debian under a separate user account and use it in standalone mode, together with HAProxy, to generate and renew certificates from given keys / CSRs, with scripts suitable for unattended operation with any number of certificates / sites.</p>
<p>I won’t cover basic sysadmin tasks, HAProxy SSL setup or HPKP configuration.</p>
<h2 id="openssl-preparations">OpenSSL Preparations</h2>
<p>You can (and should) do all of this on your local machine, we’ll copy the relevant files to the server later.</p>
<p><em>Generate private keys (if you haven’t already)</em></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl genrsa -out yoursite.key 4096
</code></pre></div></div>
<p>Sidenote: If you’re using HPKP, you really should generate at least two backup keys and include their hashes in your headers as well, to avoid bricking your domain through loss of a private key.
Print out your backup keys and/or store them (encrypted) in various different locations.
You do not want to leave them lying around on your server because that would render them useless exactly when you need them - if your server got hacked any keys that were stored on it are useless.</p>
<p>I use gpg to encrypt a tar archive which I distribute to various places. It’s also no mistake to make an offsite backup of your primary key of course.</p>
<p>The HPKP hash of an RSA private key to put into the HPKP header can be generated like that:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl rsa -in yoursite.key -outform der -pubout| openssl dgst -sha256 -binary | openssl enc -base64
</code></pre></div></div>
<p><em>Create an openSSL config file for each certificate</em></p>
<p>This can be a big timesaver later on if you’re using the same certificate for several (sub)domains and want to regenerate the CSR for this certificate.
Use the default file from <code class="highlighter-rouge">/etc/ssl/openssl.cnf</code> as a template and copy it to <code class="highlighter-rouge">yoursite.cnf</code> for example.</p>
<p>In the <code class="highlighter-rouge">[ req ]</code> section, uncomment or add the line</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>req_extensions = v3_req
</code></pre></div></div>
<p>You should also change the <code class="highlighter-rouge">default_bits</code> value to 4096 here.</p>
<p>Then, in the <code class="highlighter-rouge">[ v3_req ]</code> section, add this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>subjectAltName = @alt_names
</code></pre></div></div>
<p>To save further typing later on you should also change or add the <code class="highlighter-rouge">commonName_default</code> and <code class="highlighter-rouge">emailAddress_default</code> values in the <code class="highlighter-rouge">[ req_distinguished_name ]</code> section:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>commonName_default = yoursite.com
emailAddress_default = john.doe@gmail.com
</code></pre></div></div>
<p>And finally, at the end of the file, add a new section holding your domain names:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[ alt_names ]
DNS.1 = yoursite.com
DNS.2 = www.yoursite.com
DNS.3 = yoursite.net
DNS.4 = www.yoursite.net
</code></pre></div></div>
<p><em>Generate a CSR</em></p>
<p>With the config file you just created this is easy:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openssl req -new -key yoursite.key -out yoursite.csr -config yoursite.cnf
</code></pre></div></div>
<p>This will ask several questions about country, location and organization. Simply accept the defaults, these values are ignored by Let’s Encrypt anyway.
Common name and email address should hold your configured defaults and can thus be accepted by pressing enter as well.
Do not enter a passphrase, but just press enter two more times and you’re done.</p>
<p>Whenever you have to regenerate the CSR for this certificate due to a change in the domains it is for, or due to a changed private key, just edit the config file if necessary and re-run the above command.</p>
<h2 id="on-the-server">On the Server</h2>
<h3 id="set-up-certbot-and-add-a-separate-user-account-for-running-it">Set Up Certbot and Add a Separate User Account for Running It</h3>
<p><em>Install certbot</em></p>
<p>The <a href="https://certbot.eff.org">Certbot Site</a> has a nice interactive installation howto.
For Debian Jessie the suggestion is to use the <code class="highlighter-rouge">certbot</code> package from the <code class="highlighter-rouge">jessie-backports</code> repository, which I did.</p>
<p><em>Create the certbot user</em></p>
<p>Using a separate account and not having this run as root is just good practice and helps to keep things nicely contained in a dedicated directory.
I’ll leave the user creation up to you but in the end you should have a user <code class="highlighter-rouge">certbot</code> with <code class="highlighter-rouge">/bin/bash</code> as shell and a home of <code class="highlighter-rouge">/home/certbot</code>.
Note that this user doesn’t need to have a password set at all, for the few times you have to interactively use it (only during setup / while trying things out) you can just use <code class="highlighter-rouge">su - certbot</code> as root.</p>
<p>In <code class="highlighter-rouge">/home/certbot</code>, create a few directories:</p>
<ul>
<li><code class="highlighter-rouge">certs</code> which will hold one directory per certificate / site</li>
<li><code class="highlighter-rouge">logs</code> for certbot logs</li>
<li><code class="highlighter-rouge">config</code> for any files certbot creates at runtime</li>
</ul>
<p><em>Configure certbot</em></p>
<p>To simplify certbot usage, it’s a good idea to create a config file in <code class="highlighter-rouge">/home/certbot/.config/letsencrypt/cli.ini</code> which holds most of certbots config:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Use a 4096 bit RSA key instead of 2048
rsa-key-size = 4096
# update to register with the specified e-mail address
email = john.doe@gmail.com
# use a text interface instead of ncurses
text = True
non-interactive = True
agree-tos = True
# use the standalone authenticator
authenticator = standalone
preferred-challenges = http-01
# this is the same port as in the haproxy letsencrypt backend:
http-01-port = 54321
work-dir = /home/certbot
config-dir = /home/certbot/config
logs-dir = /home/certbot/logs
</code></pre></div></div>
<p>With this, you should be able to successfully run <code class="highlighter-rouge">certbot register</code> as the <code class="highlighter-rouge">certbot</code> user. This should create some files below <code class="highlighter-rouge">/home/certbot/config</code>.</p>
<h3 id="configure-haproxy">Configure HAProxy</h3>
<p>I assume you already have SSL/TLS set up for your sites so I’ll show just the relevant parts here for making the Let’s Encrypt domain validation work.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>frontend http
# more config here...
# direct all letsencrypt requests to a special backend
acl letsencrypt-acl path_beg /.well-known/acme-challenge/
use_backend letsencrypt-backend if letsencrypt-acl
# more config here...
# define the backend referenced above and point it to the certbot validation server
backend letsencrypt-backend
server letsencrypt 127.0.0.1:54321
</code></pre></div></div>
<p>This defines an ACL to recognize LetsEncrypt domain validation requests, and points any such requests to a dedicated backend.</p>
<p>The certbot validation server which will be spawned automatically by <code class="highlighter-rouge">certbot</code> during the certificate creation / renewal process when it is used in <em>standalone</em> mode listens on port 54321, and that’s exactly where we point our <code class="highlighter-rouge">letsencrypt</code> backend to.</p>
<h3 id="install-private-key-and-csr">Install private key and CSR</h3>
<p>Create a <code class="highlighter-rouge">/home/certbot/certs/yoursite</code> directory and place <code class="highlighter-rouge">yoursite.key</code> and <code class="highlighter-rouge">yoursite.csr</code> inside. Ensure proper permissions and file ownerships:</p>
<ul>
<li>the key is to be kept private must be owned by root and also only be readable by root (<code class="highlighter-rouge">chmod 0400 yoursite.key</code>)</li>
<li>the csr can be world readable, it’s not confidential. At the very least it has to be readable by the <code class="highlighter-rouge">certbot</code> user.</li>
</ul>
<h3 id="run-certbot">Run certbot!</h3>
<p>Yay! By now we should be all set and ready to try out the validation process.
For testing it is advisable to use certbot with the <code class="highlighter-rouge">staging</code> argument to prevent using up your daily quota of Let’s Encrypt certificates per domain.</p>
<p>Run this as the certbot user:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>certbot certonly --staging --csr yoursite.csr --cert-path yoursite.crt --chain-path chain.pem --fullchain-path fullchain.pem
</code></pre></div></div>
<p>You should end up with a certificate and the two chain files as specified in the command line options.
These are not really usable as they are only signed by the Let’s Encrypt staging environment, but we now know that our HAProxy setup and certbot config work.
The <em>fullchain</em> file is what we’ll use later on with HAProxy, since it includes both the Let’s Encrypt intermediate certificate and the certificate for <em>yoursite.com</em>.</p>
<h2 id="automate-all-the-things">Automate All the Things!</h2>
<p>With <code class="highlighter-rouge">certonly</code> mode and the <code class="highlighter-rouge">--csr</code> option it doesn’t make a difference to certbot wether you are creating a new certificate or renewing an existing one.
Therefore I also only have one shell script for both tasks. You can have a look at it below. There’s also a <a href="https://gist.github.com/jkraemer/b44971f671f499ffefe6cf5c252d7a54#file-le-renew-haproxy">Gist</a></p>
<div title="/usr/local/sbin/le-renew-haproxy" class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="nv">reload_required</span><span class="o">=</span><span class="nb">false</span>
<span class="c"># attempt to renew when less then 30 days remaining</span>
<span class="nv">exp_limit</span><span class="o">=</span>30<span class="p">;</span>
<span class="c"># returns 0 if no renewal needed, 1 otherwise</span>
check_cert_still_valid <span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">cert_file</span><span class="o">=</span><span class="nv">$1</span>
<span class="nb">local </span><span class="nv">exp</span><span class="o">=</span><span class="si">$(</span><span class="nb">date</span> <span class="nt">-d</span> <span class="s2">"</span><span class="sb">`</span>openssl x509 <span class="nt">-in</span> <span class="nv">$cert_file</span> <span class="nt">-text</span> <span class="nt">-noout</span>|grep <span class="s2">"Not After"</span>|cut <span class="nt">-c</span> 25-<span class="sb">`</span><span class="s2">"</span> +%s<span class="si">)</span>
<span class="nb">local </span><span class="nv">datenow</span><span class="o">=</span><span class="si">$(</span><span class="nb">date</span> <span class="nt">-d</span> <span class="s2">"now"</span> +%s<span class="si">)</span>
<span class="nb">local </span><span class="nv">days_exp</span><span class="o">=</span><span class="si">$(</span><span class="nb">echo</span> <span class="se">\(</span> <span class="nv">$exp</span> - <span class="nv">$datenow</span> <span class="se">\)</span> / 86400 |bc<span class="si">)</span>
<span class="nb">echo</span> <span class="s2">"Checking expiration date for </span><span class="nv">$domain</span><span class="s2">..."</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$days_exp</span><span class="s2">"</span> <span class="nt">-gt</span> <span class="s2">"</span><span class="nv">$exp_limit</span><span class="s2">"</span> <span class="o">]</span> <span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"The certificate is up to date, no need for renewal (</span><span class="nv">$days_exp</span><span class="s2"> days left)."</span>
<span class="k">return </span>0
<span class="k">else
</span><span class="nb">echo</span> <span class="s2">"The certificate </span><span class="nv">$cert_file</span><span class="s2"> is about to expire soon. Starting renewal..."</span>
<span class="k">return </span>1
<span class="k">fi</span>
<span class="o">}</span>
get_certificate<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">domain</span><span class="o">=</span><span class="nv">$1</span>
<span class="nb">local </span><span class="nv">cert_file</span><span class="o">=</span><span class="nv">$2</span>
<span class="nb">local </span><span class="nv">key_file</span><span class="o">=</span><span class="s2">"/home/certbot/certs/</span><span class="k">${</span><span class="nv">domain</span><span class="k">}</span><span class="s2">/</span><span class="k">${</span><span class="nv">domain</span><span class="k">}</span><span class="s2">.key"</span>
<span class="nb">local </span><span class="nv">csr_file</span><span class="o">=</span><span class="s2">"/home/certbot/certs/</span><span class="k">${</span><span class="nv">domain</span><span class="k">}</span><span class="s2">/</span><span class="k">${</span><span class="nv">domain</span><span class="k">}</span><span class="s2">.csr"</span>
<span class="nb">local </span><span class="nv">chain_file</span><span class="o">=</span><span class="s2">"/home/certbot/certs/</span><span class="nv">$domain</span><span class="s2">/chain.pem"</span>
<span class="nb">local </span><span class="nv">fullchain_file</span><span class="o">=</span><span class="s2">"/home/certbot/certs/</span><span class="nv">$domain</span><span class="s2">/fullchain.pem"</span>
<span class="c"># Certbot refuses to overwrite existing files, so remove anything that</span>
<span class="c"># might get in the way.</span>
<span class="c"># The certificate used by haproxy is kept separately, so no harm is done by</span>
<span class="c"># deleting these files:</span>
<span class="nb">rm</span> <span class="nt">-f</span> <span class="nv">$cert_file</span> <span class="nv">$chain_file</span> <span class="nv">$fullchain_file</span>
certbot certonly <span class="nt">--csr</span> <span class="nv">$csr_file</span> <span class="nt">--cert-path</span> <span class="nv">$cert_file</span> <span class="nt">--chain-path</span> <span class="nv">$chain_file</span> <span class="nt">--fullchain-path</span> <span class="nv">$fullchain_file</span>
<span class="k">if</span> <span class="o">[</span> <span class="nv">$?</span> <span class="nt">-eq</span> 0 <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Creating </span><span class="nv">$combined_file</span><span class="s2"> with latest certs..."</span>
<span class="nb">sudo</span> /usr/local/sbin/le-haproxy-bundle <span class="nv">$domain</span>
<span class="nb">echo</span> <span class="s2">"Renewal process finished for domain </span><span class="nv">$domain</span><span class="s2">"</span>
<span class="nv">reload_required</span><span class="o">=</span><span class="nb">true
</span><span class="k">else
</span><span class="nb">echo</span> <span class="s2">"certbot failed, not replacing installed certificate for </span><span class="k">${</span><span class="nv">domain</span><span class="k">}</span><span class="s2">"</span>
<span class="k">fi</span>
<span class="o">}</span>
process_certificate<span class="o">()</span> <span class="o">{</span>
<span class="nb">local </span><span class="nv">domain</span><span class="o">=</span><span class="nv">$1</span>
<span class="nb">local </span><span class="nv">cert_file</span><span class="o">=</span><span class="s2">"/home/certbot/certs/</span><span class="nv">$domain</span><span class="s2">/</span><span class="k">${</span><span class="nv">domain</span><span class="k">}</span><span class="s2">.crt"</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="nv">$cert_file</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span>check_cert_still_valid <span class="nv">$cert_file</span>
<span class="k">if</span> <span class="o">[</span> <span class="o">!</span> <span class="nv">$?</span> <span class="nt">-eq</span> 0 <span class="o">]</span><span class="p">;</span> <span class="k">then
</span>get_certificate <span class="nv">$domain</span> <span class="nv">$cert_file</span>
<span class="k">fi
else
</span><span class="nb">echo</span> <span class="s2">"No certificate for domain </span><span class="nv">$domain</span><span class="s2"> exists yet. Creating one..."</span>
get_certificate <span class="nv">$domain</span> <span class="nv">$cert_file</span>
<span class="k">fi</span>
<span class="o">}</span>
<span class="nv">domain</span><span class="o">=</span><span class="nv">$1</span>
<span class="c"># loop over all domains unless a domain is given</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="k">${</span><span class="nv">domain</span><span class="k">}</span><span class="s2">"</span> <span class="o">=</span> <span class="s2">""</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
for </span>i <span class="k">in</span> <span class="si">$(</span> <span class="nb">ls</span> /home/certbot/certs <span class="si">)</span><span class="p">;</span> <span class="k">do
</span>process_certificate <span class="nv">$i</span>
<span class="k">done
else
</span>process_certificate <span class="nv">$domain</span>
<span class="k">fi
if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$reload_required</span><span class="s2">"</span> <span class="o">=</span> <span class="nb">true</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">echo</span> <span class="s2">"Reloading haproxy"</span>
<span class="nb">sudo</span> /usr/sbin/service haproxy reload
<span class="k">fi</span>
</code></pre></div></div>
<p>If called without arguments, this script will loop over the directories in <code class="highlighter-rouge">/home/certbot/certs</code> and either create a new certificate or renew an existing one if it’s about to expire in the next 30 days.
Since the script only acts on certificates that will expire in the next 30 days, it can be called as often as you want. Once or twice daily through cron should be enough, however.
For manual operation you can also call it with one directory name as an argument, i.e. <code class="highlighter-rouge">/usr/local/sbin/le-renew-haproxy yoursite</code>.</p>
<p>Before this will work there are two more things to do, however.</p>
<p><em>Make the certificate usable for HAProxy</em></p>
<p>The task of taking the generated certificate and combining it with the custom <code class="highlighter-rouge">dhparam</code> and private key file to the certificate bundle to be used by HAProxy is carried out by another script, which is called through sudo by <code class="highlighter-rouge">le-renew-haproxy</code>:</p>
<div title="/usr/local/sbin/le-haproxy-bundle" class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="nv">domain</span><span class="o">=</span><span class="nv">$1</span>
<span class="nv">fullchain_file</span><span class="o">=</span><span class="s2">"/home/certbot/certs/</span><span class="nv">$domain</span><span class="s2">/fullchain.pem"</span>
<span class="nv">key</span><span class="o">=</span><span class="s2">"/home/certbot/certs/</span><span class="k">${</span><span class="nv">domain</span><span class="k">}</span><span class="s2">/</span><span class="k">${</span><span class="nv">domain</span><span class="k">}</span><span class="s2">.key"</span>
<span class="nv">combined_file</span><span class="o">=</span><span class="s2">"/etc/haproxy/ssl/</span><span class="k">${</span><span class="nv">domain</span><span class="k">}</span><span class="s2">.pem"</span>
<span class="nv">dhparam</span><span class="o">=</span><span class="s2">"/etc/haproxy/dhparam"</span>
<span class="k">if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="nv">$fullchain_file</span> <span class="nt">-a</span> <span class="nt">-f</span> <span class="nv">$key</span> <span class="nt">-a</span> <span class="nt">-f</span> <span class="nv">$dhparam</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
if</span> <span class="o">[</span> <span class="nt">-f</span> <span class="nv">$combined_file</span> <span class="o">]</span><span class="p">;</span> <span class="k">then
</span><span class="nb">cp</span> <span class="nv">$combined_file</span> <span class="s2">"</span><span class="k">${</span><span class="nv">combined_file</span><span class="k">}</span><span class="s2">.old"</span>
<span class="k">fi
</span><span class="nb">cat</span> <span class="nv">$fullchain_file</span> <span class="nv">$key</span> <span class="nv">$dhparam</span> <span class="o">></span> <span class="nv">$combined_file</span>
<span class="nb">chmod </span>0400 <span class="nv">$combined_file</span>
<span class="k">fi</span>
</code></pre></div></div>
<p><em>Setup sudo permissions for the certbot user</em></p>
<p>For things to work out non-interactively, you have to give the <code class="highlighter-rouge">certbot</code> user permission to use passwordless sudo for these two commands:</p>
<ul>
<li><code class="highlighter-rouge">/usr/local/sbin/le-haproxy-bundle yoursite</code> to build and install the certificate bundle to <code class="highlighter-rouge">/etc/haproxy/ssl</code></li>
<li><code class="highlighter-rouge">/usr/sbin/service haproxy reload</code> to enable reloading of haproxy</li>
</ul>
<h2 id="disclaimer">Disclaimer</h2>
<p>There might be typos in the scripts / I might have entirely forgotten something.
This article is written after the fact, and my server is provisioned with chef so I had to reconstruct / amend / generalize things. Please let me know of any oversights in the comments.</p>
<p>Be careful with HPKP. Use it in report-only mode and/or with short lifetimes until you’re sure you got it right.</p>
<h2 id="prior-art-and-further-reading">Prior Art and Further Reading</h2>
<p><a href="https://certbot.eff.org">The Certbot Documentation</a></p>
<p><a href="https://letsencrypt.org">The Let’s Encrypt Website</a></p>
<p>The idea to use standalone mode through a HAProxy backend and the initial version of my renewal shell script stem from <a href="https://gist.github.com/T0MM0R/8503d077c5e30030fe41af514bbde9c9">this Gist</a>.</p>
<p>Scott Helmes <a href="https://scotthelme.co.uk/setting-up-le/">introductory article on Let’s Encrypt</a> suggests to use a customized OpenSSL config file for each certificate, which eases CSR generation a lot.</p>
<p>Check out the <a href="https://github.com/pierky/haproxy-ocsp-stapling-updater">haproxy-ocsp-stapling-updater</a> for setting up OCSP stapling with HAProxy.</p>
<p>Another howto on <a href="http://www.whiteboardcoder.com/2016/05/lets-encrypt-haproxy.html">Using haproxy with Let’s Encrypt</a></p>
<p>My Gist with all <a href="https://gist.github.com/jkraemer/b44971f671f499ffefe6cf5c252d7a54">config files and scripts</a></p>
<h2 id="and-finally">And finally</h2>
<p>Consider <a href="https://supporters.eff.org/donate">donating to the EFF</a> to support their awesome work.</p>Offshore Tax Exemption or When Your Company Doesn’t Have to Pay Taxes at All2016-09-03T04:00:00+02:002018-08-01T03:40:00+02:00repo://posts.collection/_posts/2016-09-03-offshore-tax-exemption-or-when-your-company-doesn-t-have-to-pay-taxes-at-all.md<p>Hong Kong’s corporate taxes are already very low compared to most western countries.
On top of that, these already low corporate taxes only apply to profits earned locally in Hong Kong. So if you operate a store there, or an office where you do business, or at least sign relevant contracts for your business on Hong Kong soil, then you’re going to pay the regular corporate tax in Hong Kong.
If, on the other hand, you never work in Hong Kong, and maybe only come over to pick up the mail and do some shopping, and your clients are located outside Hong Kong as well, chances are your profits fall into the <em>offshore</em> category.</p>
<p>And those <em>offshore profits</em> are tax free in Hong Kong.</p>
<p>All you have to do is inform the accountants that you intend to apply for this tax exemption, and tell them which of your profits you think are offshore profits (and which not, if any).
They will take care of the rest and mark the corresponding earnings accordingly in the financial report. If this is the first time you apply for the exemption, the IRD will get back to you at some point with some paper work to fill out before they finally grant (or not) the offshore tax exemption status.</p>
<p>It is important to keep in mind that this whole ‘offshore tax free’ thing is just about the Hong Kong corporate tax. You are still liable for personal income tax in Hong Kong (for your director’s salary, if you pay yourself one) and, depending on your citizenship and residency status, personal taxes on salaries and other earnings (i.e., dividends) of any other countries you have to do with in one way or the other.</p>Accounting and Reporting for a Hong Kong Limited2016-09-02T03:47:00+02:002016-09-02T03:50:00+02:00repo://posts.collection/_posts/2016-09-02-accounting-and-reporting-for-a-hong-kong-limited.md<p>Nearly a year ago, I <a href="/2015/10/how-i-set-up-shop-in-hong-kong">started my Hong Kong Limited</a>.
Hong Kong is famous for low taxes and ease of doing business, but like everywhere else there are rules and regulations you have to follow in terms of book keeping and submission of documents to various government agencies.
This article will deal specifically with accounting, auditing and tax reporting to the <abbr title="Inland Revenue Department">IRD</abbr>.</p>
<h2 id="what-needs-to-be-done-and-when">What needs to be done and when?</h2>
<p>The business year in Hong Kong traditionally ends on March 31st., but it’s also possible to use the calendar year, so you’ll either start preparing your reports at the beginning of the year or in April.</p>
<p>While newly formed companies are exempt from reporting during the first 18 months of their existence, I think it is a good idea to still prepare all the things in time so you get an idea about how it works early on.</p>
<p>The whole process consists of two parts - accounting and auditing.</p>
<p>While in theory you can do all the accounting by yourself, you almost certainly should hire an accountant to do that for you.
The auditing is mandatory and has to be carried out by an accredited auditor.</p>
<h2 id="accounting">Accounting</h2>
<p>I wrote it above and I’ll reiterate it: get a professional accountant.</p>
<p>I’ve been working with <a href="http://bridges.hk">Bridges</a> since the beginning and am very happy with their performance, therefore I chose to let them handle this part of my business as well.</p>
<p>How much you’ll have to pay to have everything prepared and ready for auditing depends on the nature of your business and more specifically on the number of financial transactions you have. I just have very few so there’s not much to do and I got away with the minimal fees of about HKD 540 per month at Bridges.</p>
<p>The whole process couldn’t be much easier:</p>
<p>Simply collect everything relevant (sales invoices, expenses / invoices for any purchases made, all bank statements and any contracts / agreements you entered into), scan what’s not already in digital format anyway, put it all together in a ZIP archive and send it over via email<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>. An accountant at Bridges will then do all the book keeping and get back to you in case of questions.</p>
<p>With a few emails going back and forth for questions and clarifications it took a little bit less than 2 months until I got my accounting report including profit/loss computation, balance sheet, a tax computation and a general ledger with all the details.</p>
<h2 id="auditing">Auditing</h2>
<p>After I confirmed the correctness of the accounting report it was submitted to the auditor.
The auditing itself took about a month, then I got the auditor’s report by email for proof reading and confirmation.
I confirmed and few days later received the final report via FedEx for me to sign and send back.
This is the only step where real paper is involved and you actually have to receive and send something by Courier.
Bridges did everything to make this as easy as possible, all the pages and places where I needed to sign were marked and very easy to find.</p>
<p>Usually the signed auditing report and financial statements would then be submitted together with the profits tax return form to the <abbr title="Inland Revenue Department">IRD</abbr>. Since my company is still less than 18 months old I didn’t receive such a form yet, so this is it for now.</p>
<p>The price of the audit depends on your company’s turnover in the period. To give you an idea, for less than HKD 1,000,000 turnover the audit through Bridges costs HKD 4,500.</p>
<h2 id="wrapping-it-up">Wrapping it up</h2>
<p>Just yesterday I got an email from Bridges to inform me that the bound copy of the audited report (which I’ll have to keep safe for the next 7 years) has been stored in my mailbox waiting for me to pick it up.</p>
<p>The whole thing cost me about HKD 8,000, which is about the same I’d have to pay for similar services in Germany.
All in all a reasonable price for not having to do much more than send over my documents by email, respond to a few questions and sign the final report.</p>
<p>If you, after reading this, are considering to form your Hong Kong company through <a href="http://bridges.hk">Bridges</a> and/or want them to do your accounting, feel free to mention Jens Kraemer from Electric Things Limited sent you and we’ll both earn a nice discount.</p>
<p>Have any questions? Feel free to ask in the comments and I’ll do my best to help.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Scary huh? Encrypt the ZIP file and tell them the password on the phone, or have them at least download it from Dropbox etc via a secure connection. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>