Category Archives: Linux

How to Read Kobo e-books on Your Kindle Devices

Recently I am fortunate to be introduced of a wonderful book about self improvement. As a Kindle user, the first thing I do is to look for it in the Kindle Store. Bummer, the search result returns nothing. Having checked again with my friend, the book was bought at Kobo. That’s not good, because I prefer reading books on my e-ink Kindle device rather than on LCD or LED screens, since I would suffer from eye strain easily after prolonged reading using them. Seems as I am getting older my eyes are becoming more sensitive to strong visible lights.

I have no experience with Kobo before, so I did some research, found that Kobo’s e-books are not compatible with Kindle. The book that I want is in EPub format with DRM protection (bad!), Kindle doesn’t take ePub, let alone DRM-protected ones.

I have no other choice, except to find my way to decrypt the e-book to remove the DRM. Once it is DRM-free, I can simply convert it from EPub to MOBI/AWZ3, which is natively supported by Kindle.

As my main working machine is Ubuntu, what I am going to write are only applicable to Ubuntu. With some tweaking you can achieve the same result on other Linux distributions. I performed all the steps under Ubuntu 19.04. The process is a bit tricky but not impossible. It mainly involves these steps:

  • Use Adobe digital editions under WINE to save Kobo e-book as a DRM-encrypted ePUB file.
  • Use DeDRM plugin in Calibre to remove DRM from the EPUB file.
  • Convert the DRM-free EPUB to MOBI/AWZ3 by Calibre.

Save the e-book into DRM-encrypted EPUB

  • First login to Kobo and download the e-book’s ACSM file. ACSM a very small descriptive file that contains metadata of your e-book.
  • Go to your books library in Kobo, under the book cover there is an icon with 3 dots. Click on it and it will display a menu, the third item is “Download”. Click on it, the website will prompt you to download a file, its filename ends with “.acsm”. Save it for later use.
  • First login to Kobo and download the e-book’s ACSM file. ACSM a very small descriptive file that contains metadata of your e-book.
  • Go to your books library in Kobo, under the book cover there is an icon with 3 dots. Click on it and it will display a menu, the third item is “Download”. Click on it, the website will prompt you to download a file, its filename ends with “.acsm”. Save it for later use.
  • If you do not have WINE installed, install it with other tools by:
    sudo apt install wine-stable winetricks cabextract 
  • Now download Adobe digital editions (ADE) version 2.0 and install it in WINE ADE 2.0 is available at http://download.adobe.com/pub/adobe/digitaleditions/ADE_2.0_Installer.exe, download it for later use.
  • Run the following commands:
  • In ADE, open the acsm file by going to File > Add to Library (or press Ctrl-O). There is a GUI glitch that the menu is not shown when you click “File” until you move your mouse around.
  • ADE will ask you to “Authorize Your Computer”. If you do not have an Adobe ID, click on the “Create an Adobe ID” link, and you will be brought to a web page, where you can sign up. If you already have an Adobe ID, sign in with your existing Adobe ID and password.
  • Voila! After you have signed in, ADE will start loading your e-book. After awhile, your e-book will be shown in ADE. Now, check your ~/Documents/My Digital Editions folder, you should see your e-book there in EPUB format!

Remove DRM from Your E-Book

  • If you do not have Calibre in your computer, install it with apt install calibre.
  • Then you need to import the DeDRM plugin to Calibre, as Calibre doesn’t come with the capability to remove DRM out of the box.
    • Go to https://github.com/apprenticeharper/DeDRM_tools/releases, and download the latest DeDRM_tools zip file.
    • Unzip it. There are many files but what we care is the DeDRM_calibre_plugin folder. 
    • Open up Calibre, click on the Preferences icon at the top bar. From there, locate the Plugins icon under the “Advanced” heading. In my Calibre it is the last row.
      Calibre Preferences dialog
    • At the very bottom of the plugins window, there is a button called “Load plugin from file”. Click on it. Browse to the Calibre plugin folder that you extracted from the zip file, select DeDRM_plugin.zip within that folder. Calibre will warn you that importing an external plugin is a potential security risk. That’s fine, just confirm your choice. Once it is done, Calibre will show you a success message telling you to restart the program for the changes to take effect. Do as you are told.
      DeDRM plugin installed
  • Now fire up Calibre again. Click on “Add books” icon on the top bar. Choose the EPUB e-book from ~/Documents/My Digital Editions. Your e-book will be shown in the main window. Click on the “View” icon in the top bar, you should now be able to view it! If you can’t view it saying that the book is encrypted, you may have done something wrong in importing the DeDRM plugin. Go back and check that the plugin is properly installed.

Convert the DRM-free EPUB to MOBI/AWZ3

  • Since you now have your e-book unencrypted, you can convert the e-book to any format that you desire. If you are like me and prefer Kindle, you should convert it to AWZ3.
  • Select your e-book in the main window, then click on the “Convert books” icon that is on the top bar.
  • A new window will show up for configuring many conversion settings. Since we are converting for Kindle, choose “AZW3” as the “Output format” at the top right corner of the window. Then click on “Page setup” in the left pane, choose an “Output profile” for your device. There are many different settings that you can test with. Once you are satisfied, click “OK”. Calibre will now start converting your e-book to AZW3, wait for a while, it will prompt you when it is done.
  • After the conversion completes, look at the right hand side of the main window. You can see the label “Path” under the book cover, after “Path” there is a link called “Click to open”. Click on it, your file browser will open and you should see a new file with the extension azw3. That is the file that you can import to your Amazon Kindle. Transfer it to your Kindle device just like any other Kindle e-books. Enjoy!

Securing DNS Traffic in China

Overview

DNS poisoning is one of the most common cause of nuisance when accessing websites
that are outside this 1.4 billion-people Oriental country. So far, the best way to protect yourself from this trouble is to route all your DNS traffic through an encrypted channel, and the method I am going to introduce is DNSCrypt. There is not yet a standard for encrypted DNS, DNSCrypt is a project done by OpenDNS. According my experience, DNSCrypt is very reliable and robust, the cryptography of the protocol is called DNSCurve, which is a public-key crypto that employes an extremely strong elliptic-curve cryptography called Curve25519.

If you have read my previous writing, you should know my setup is a Raspberry Pi, and so the rest of this article is based on that, running Raspbian. Dnsmasq will be used as the first DNS caching proxy to serve incoming DNS queries from machines on the network. If the queried domain name is a China one, the request will be served by a China DNS. This is necessary because for some domains, answers from DNS servers in China and global ones could be different. If the requested domain does not belong to any known China domains, the request will be forwarded to dnscrypt-proxy, which will ask a DNSCrypt server for an answer.

After DNSCrypt is used, your DNS traffic will look like this:

Setting up DNSCrypt

As illustrated in the above diagram, dnscrypt-proxy is the piece of software that handles DNSCrypt, but it is not available in Raspbian’s Wheezy and Jessie releases, only in testing (currently Stretch). You can either compile it yourself, or grab the debian package I built and install it. You can find the package here. It is based on the Raspbian package in testing repo, with some modification to debian packaging files, since the one in testing depends on systemd, which had not yet been adopted when Wheezy was released.

If you really want to build the package yourself, first install the libsodium packages. The package are also not available in Wheezy repo but the ones from testing, libsodium13_1.0.3-1_armhf.deb and libsodium-dev_1.0.3-1_armhf.deb, can be installed without any problem. Download and install them, then follow these steps to build your dnscrypt-proxy package:

After dnscrypt-proxy is installed, you have to update the port it uses. Change DNSCRYPT_PROXY_LOCAL_ADDRESS in /etc/default/dnscrypt-proxy to another port other than 53 (as it will be used by dnsmasq later), like this:

You can also change the remote DNSCrypt server, but since the default (cisco) works well for me, I left it unchanged.

Now test it to make sure it works as expected:

Setting up dnsmasq

Dnsmasq is very common and is available in Raspbian, installing it is easy:

Now we have to do some configuration in /etc/dnsmasq.conf. These are my recommended settings. Please note that the interface option is the network interface that dnsmasq will serve, and in my case that is wlan0. You have to change it to the one that applies to your case.

Now comes the interesting part. We are going to tell dnsmasq to use a China DNS server (114.114.114.114 in my example) for China domains and DNSCrypt server for all others. This is done by using the server option in /etc/dnsmasq.conf. Here is an example:

This is pretty straightforward. The last line tells dnsmasq to use your dnscrypt proxy if the domain you query does not match any China domains. In my config file there are 12238 lines for China domains so I’m not going to post them all here, you can get the snippet of my dnsmasq.conf here, and put it into your own dnsmasq.conf. The problem is to maintain the list for all China hosts. I am now using the list from the fqrouter project, it has been serving me well, since most common domains are already there. What’s worrying is due to the abandon of the project by it’s author, the list is now unmaintained. If you know a more updated list, please let me know!

Boost WIFI Speed by Forcing 40 MHz Channels in hostapd on RPi

802.11n can double the channel bandwidth of 802.11g from 20 MHz to 40 MHz, but this operation mode is not recommended in areas that the spectrums are congested and likely interfere with existing WIFI and bluetooth devices. As a result, hostapd will not enable 40 MHz when it finds other channels are being used, like what is seen from the hostapd log below:

However, this is unrealistic in modern cities. You should be grateful the primary channel you choose has not been used already, let alone the additional one for 40 MHz. What can we do? We have to force hostapd to turn on 40 MHz anyway. But since hostapd does not have any configuration options for that, we have to recompile hostapd from source. Actually, this has been done in other distros. For example, OpenWRT and Arch has patched their hostapd with the noscan patch, which adds an option to force 40MHz mode regardless of the environment.

Before turning on 40 MHz, the iwconfig on my laptop is like this, notice the bit rate is only 54 Mb/s:

Now let us start. First you need to add a deb-src source to apt so that we can download source packages. Note that I am still using Raspbian Wheezy. If you use Jessie, change the deb-src line according to your original deb line.

Save the following patch into a file:

Because we are going to create a new hostapd package, it’d be a good practice to increment the version number by adding a new section at the top of debian/changelog, like this:

Now we are ready to compile the packages:

If it succeeds, install the new package with dpkg -i.

Restart hostapd, it should now force 40 MHz mode:

And my iwconfig output now shows Bit Rate is 150 Mb/s, yay!

Deploy Shadowsocks on Raspberry Pi with ChinaDNS and Redsocks

Assume you already have a Raspberry Pi configured as a WIFI router like mine shown below, but you live in China and have to deal with the fact that many websites can’t be accessed due to GFW. Don’t be despair and with some hacking you can get your Internet freedom back.

IMG_20150714_134341

The mechanism is to use shadowsocks on your router which directs any traffic to a shadowsocks server in the free world. It’s simple to get it up and running on a local machine, but on a router you need to use redsocks to redirect traffic to the shadowsocks client running on your Raspberry Pi. DNS traffic has to be routed by redsocks as well otherwise your DNS replies will be contaminated. To remain as fast as normal when accessing China websites you also need to skip routing traffic to Redsocks for anything within the China IP ranges. Even if you don’t care about performance, this is still necessary in some circumstances like geoip restriction such as tv.sohu.com does not deliver contents if you live outside of China.

Finally, we want to improve the performance even further by using ChinaDNS. To avoid DNS poisoning, we can always resolve DNS over our secured shadowsocks connection, but this is not optimal if a China website have CDNs outside China. ChinaDNS queries local DNS servers to resolve Chinese domains and queries foreign DNS servers to resolve foreign domains, and from my testing it is useful to avoid DNS poisoning with the “DNS compression pointer mutation” option. (Update: I have switched from ChinaDNS to dnsmasq+dnscrypt, please read Securing DNS Traffic in China to see how it works.)

Shadowsocks

I assume that you have shadowsocks server running on a public server, so I will skip that part and only talk about the client side.

Installing shadowsocks is very simple, note that it will be installed under /usr/local/.

Start up shadowsocks while listening on local port 1080:

Redsocks

Install redsocks, simply apt-get from the archive:

Then you need to change the START option in /etc/default/redsocks from NO to YES, so that redsocks will start automatically at boot time and also can be started by sudo /etc/init.d/redsocks start:

Then update /etc/redsocks.conf. Most of the default settings work fine, just need to change local_ip in the redsocks section to your address of the network interface that accepts traffic from your local network. The default is 127.0.0.1, but that does not work well if you want to re-route traffic from other machines on your network, so change it to something like:

But we want traffic from other hosts in your network to be redirected by redsocks to your local shadowsocks client, which in turn sent to the the remote shadowsocks server. We need to pay special attention to DNS traffic, as DNS poisoning is prevalent in China. We need to take special care to redirect DNS traffic through redsocks/shadowsocks.

We also want all China traffic NOT to go through shadowsocks for performance. This can be easily done by looking at the destination IP, if it is in the China IP range we skip going through the REDSOCKS china. First we need to get all network segments allocated to China and save it to a file called chnroute.txt:

These all can be accomplished by iptables. You need to run the following iptables commands, or put them in a local script and run it with sudo.

Run iptables -t nat -L -n to make sure the rules have been added correctly. Now start up redsocks by sudo /etc/init.d/redsocks start and let’s test it out by doing some web browsing on another computer in your local network. If that works fine, congratulations and you have set up everything correctly! If not, look at shadowsocks output and also turn on redsock’s log_debug and check if there is anything useful in /var/log/daemon.log.

ChinaDNS

ChinaDNS is not absolutely necessary, but as explained at the beginning it is desirable. There is no pre-built package so we need to compile it. It is simple to do:

After it is successfully compiled, test it out:

If it goes well, run src/chinadns -m -c chnroute.txt when your router boots.

That’s it! I hope these are useful to you.

Update: I have switched from ChinaDNS to dnsmasq+dnscrypt, please read Securing DNS Traffic in China to see how to set it up.

Debian GNU/Linux 4.0 Release Talk

Last night I gave a short presentation at Beijing Linux User Group (BLUG) on the Debian project and the new Etch release. As Roy Chan had given a talk about the new release in the Hong Kong Debian Etch release party, I asked for his permission to use his slides for this talk, and he happily agreed. I made some modifications to his slides and cut out the part about Linux introduction, you can find the presentation slides in OpenOffice.org 2 format here, PDF version here. In the meeting Darren mentioned the idea of using peer-to-peer technology in apt, so that users don’t need to set any repositories in the configuration, this is a very good idea, and coincidentally there is a Google Summer of Code project on this topic.