wallabako

README

Wallabag downloader

This tool is designed to automatically download Wallabag articles into your local computer or Kobo ebook reader.

Features:

• fast: downloads only files that have changed, in parallel
• unattended: runs in the background, when the wifi is turned on, only requires you to tap the fake USB connection screen for the Kobo to rescan its database
• status synchronization: read books are marked as read in the Wallabag instance
• easy install: just drop the magic file in your kobo reader like any other book, edit one configuration file and you're done

The instructions here are mostly for the Kobo E-readers but may work for other platforms. I have tested this on a Debian GNU/Linux 9 ("stretch") system, a Kobo Glo HD and a Kobo Touch.

Table of Contents

1. Wallabag downloader
2. Download and install
3. Upgrade
4. Configuration
5. Usage
   1. Kobo devices
   2. Commandline
   3. On-screen display
6. Support
7. Troubleshooting
   1. Logging
   2. Configuration file details
   3. Configuration file is not found even if present
   4. Some articles are not downloaded or disappear
   5. Unable to open database file
   6. x509: failed to load system roots and no roots provided
   7. Command not running
   8. Uninstalling
8. Known issues
9. Credits
10. Contributing
    1. Design notes
    2. Remaining issues
11. Related projects

Download and install

Quick start for Kobo devices:

1. connect your reader to your computer with a USB cable
2. download the latest KoboRoot.tgz
3. save the file in the .kobo directory of your e-reader
4. create the configuration file as explained in the configuration section
5. disconnect the reader

When you disconnect the reader, it will perform what looks like an upgrade, but it's just the content of the KoboRoot.tgz being automatically deployed. If you connect the reader again, the KoboRoot.tgz file should have disappeared.

When you connect your reader to a Wifi access point, the wallabako program should run, which should create a wallabako.log.txt file at the top directory of the reader which you can use to diagnose problems, see also the troubleshooting section.

Another more advanced installation method, for power users, can done with launchers like KFmon or NickelMenu. See the design document, section "Launchers" for more information for this experimental configuration.

Upgrade

To upgrade wallabako, simply follow install instruction.

Configuration

The next step is to configure Wallabako by creating a .wallabako.js file in the top directory of the reader, with the following content:

{
  "WallabagURL": "https://app.wallabag.it",
  "ClientId": "14_2vun20ernfy880wgkk88gsoosk4csocs4ccw4sgwk84gc84o4k",
  "ClientSecret": "69k0alx9bdcsc0c44o84wk04wkgw0c0g4wkww8c0wwok0sk4ok",
  "UserName": "joelle",
  "UserPassword": "your super password goes here"
}

Make sure you use a plain text editor like Gedit or Notepad, as LibreOffice can cause trouble because it doesn't save the file as plain text by default.

Let's take this one step at a time. First, the weird curly braces syntax is because this is a JSON configuration file. Make sure you keep all the curly braces, quotes, colons and commas ({, }, ", :, ,).

1. The first item is the WallabagURL. This is the address of the service you are using. In the above example, we use the official Wallabag.it service, but this will change depending on your provider. Make sure there is no trailing slash (/).

2. The second and third items are the "client tokens". Those are tokens that you need to create in the Wallabag web interface, in the Developer section. Simply copy paste those values in place.

3. The fourth and fifth items are your username and passwords. We would prefer to not ask you your password, but unfortunately, that is still required by the Wallabag API

Also note that some commandline flags are hardcoded in the wallabag-run script. To modify those, you will need to modify the file in the KoboRoot.tgz file or hack the kobo to get commandline access. There are also more settings you can set in the configuration file, see the troubleshooting section for more information.

Note that there are more configuration settings that can be set, see the configuration file details section for more information.

Usage

Kobo devices

If everything was deployed correctly, Wallabako should run the next time you activate the wireless connection on your device. You will notice it is running because after a while, the dialog that comes up when you connect your device with a cable will come up, even though the device is not connected! Simply tap the Connect button to continue the synchronisation and the library will find the new entries.

When an article downloaded with Wallabako is finished on your reader, it will be marked as read in Wallabag.

Wallabako also downloads a limited numbers of articles from Wallabag, and it will remove extra articles (for example if they are too old or were marked as read in Wallabag).

By default, Wallabako will not delete old files from your reader - you will need to remove those files through the reader interface yourself. This is to avoid unnecessary synchronisations which are distracting to the user.

Commandline

Wallabako can also be compiled installed on a regular computer, provided that you have the go suite installed. Simply do the usual:

go get gitlab.com/anarcat/wallabako

If you are unfamiliar with go, you may want to read up on the getting started instructions. If you do not wish to install golang at all, you can also download the standalone binaries for 64 bits (aka amd64 or x86_64) or ARM (e.g. Raspberry PI).

You also need to create a configuration file as detailed above.

The program looks for the file in the following locations:

1. $HOME/.config/wallabako.js
2. $HOME/.wallabako.js
3. /mnt/onboard/.wallabako.js
4. /etc/wallabako.js

You will probably want to choose the first option unless you are configuring this as a system-level daemon.

Then you can just run wallabako on the commandline. For example, this will download your articles in the epubs directory in your home:

wallabako -output ~/epubs

Use the -h flag for more information about the various flags you can use on the commandline.

The program is pretty verbose, here's an example run:

$ wallabako -output /tmp
2017/01/31 22:16:41 logging in to https://example.net/wallabag
2017/01/31 22:16:41 CSRF token found: 200 OK
2017/01/31 22:16:41 logged in successful: 302 Found
2017/01/31 22:16:42 found 65 unread entries
2017/01/31 22:16:42 URL https://example.net/wallabag/export/23152.epub older than local file /tmp/23152.epub, skipped
2017/01/31 22:16:42 URL https://example.net/wallabag/export/23179.epub older than local file /tmp/23179.epub, skipped
2017/01/31 22:16:42 URL https://example.net/wallabag/export/23170.epub older than local file /tmp/23170.epub, skipped
2017/01/31 22:16:42 URL https://example.net/wallabag/export/23180.epub older than local file /tmp/23180.epub, skipped
2017/01/31 22:16:42 URL https://example.net/wallabag/export/23160.epub older than local file /tmp/23160.epub, skipped
2017/01/31 22:16:42 processed: 5, downloaded: 0
2017/01/31 22:16:42 completed in 1.44s

You can also run the program straight from source with:

go run *.go

Building for the Kobo requires more work, see the contribution guide for details.

On-screen display

Out of the box, wallabako does not know how to give user feedback on ebook readers like the Kobo. It can (and will), however, use the fbink program if it's found. If you have [koreader][] or kfmon installed, it should be able to find it there.

This feature will show wallabako's log in a crude overlay. By default, it piles them up on two lines at the bottom of the screen. This is less intrusive but can be hard to read.

To get the full experience, try to enable FbinkInteractive in the configuration file, which will scroll all the messages, one per line, from the top of the display, completely clobbering the current display. This can be extremely distracting which is why it's off by default.

And, of course, if you haven't modified your reader far enough to have fbink actually installed, you'll never see that good stuff. If you want to only install fbink, it seems the canonical installation instructions are in this forum post.

Support

I will provide only limited free support for this tool. I wrote it, after all, for my own uses. People are welcome to file issues and send patches, of course, but I cannot cover for every possible use cases. There is also a discussion on MobileRead.com if you prefer that format.

Troubleshooting

To troubleshoot issues with the script, you may need to get commandline access into it, which is beyond the scope of this documentation. See the following tutorial for example.

• Hacking the Kobo Touch for Dummies
• Kobo Touch Hacking

Below are issues and solutions I have found during development that you may stumble upon. Normally, if you install the package correctly, you shouldn't get those errors so please do file a bug if you can reproduce this issue.

Table of Contents

• Logging
• Configuration file details
• Configuration file is not found even if present
• Some articles are not downloaded or disappear
• Unable to open database file
• x509: failed to load system roots and no roots provided
• Command not running

Logging

Versions from 0.3 to 1.0 were writing debugging information in the wallabako.log.txt on the reader. This is now disabled by default (see this discussion for why) but can be enabled again by adding a logfile option in the configuration file, like this:

{
  "WallabagURL": "https://app.wallabag.it",
  "ClientId": "14_2vun20ernfy880wgkk88gsoosk4csocs4ccw4sgwk84gc84o4k",
  "ClientSecret": "69k0alx9bdcsc0c44o84wk04wkgw0c0g4wkww8c0wwok0sk4ok",
  "UserName": "joelle",
  "UserPassword": "your super password goes here",
  "logfile": "/mnt/onboard/wallabako.log.txt"
}

This will make a wallabako.log file show up on your reader that you can check to see what's going on with the command.

You can increase the verbosity of those logs with the Debug commandline flag or configuration option (set to true, without quotes). WARNING: this will include your password and authentication tokens, so be careful where you send this output.

Configuration file details

Most commandline options (except -version and -config) can also be set in the configuration file. Here are the configuration options and their matching configuration file settings:

Configuration	Flag	Default	Meaning
Debug	-debug	false	include (lots of!) additional debugging information in logs, including passwords and confidential data
Delete	-delete	false	delete EPUB files marked as read or missing from Wallabag
Database	-database	/mnt/onboard/.kobo/KoboReader.sqlite	path to the Kobo database
Concurrency	-concurrency	6	number of downloads to process in parallel
Count	-count	-1	number of articles to fetch, -1 means use Wallabag default
Exec	-exec	nothing	execute the given command when files have changed
LogFile	N/A	no logging	rotated logfile to store debug information
OutputDir	-output	current directory	output directory to save files into
PidFile	-pidfile	wallabako.pid	pidfile to write to avoid multiple runs
RetryMax	-retry	4	number of attempts to login the website, with exponential backoff delay
Tags	-tags	no tags filtering	a comma-separated list of tags to filter for
Plato.LibraryPath	N/A	/mnt/onboard	For plato document reader integration, the value of [[libraries.path]] in Settings.toml
FbinkInteractive	N/A	false	use full screen interactive fbink mode, if available

Some more details about specific settings:

• The Count option actually defaults to 30 in Wallabag, at the time of writing. You may want to bump that up if you have more than 30 unread articles, see below for details.

• The PidFile is actually written in one of those directories, the first one found that works:

  1. /var/run
  2. /run
  3. /run/user/UID
  4. /home/USER/.

• There's no -logfile flag anymore since this was not really useful: you can just redirect output to a file using shell redirection (> logfile). Also, it was difficult to implement logging for configuration file discovery while at the same time allowing the logfile to be changed when commandline flags are parsed.

Finally, note that some of those settings are hardcoded in the wallabako-run wrapper script and therefore cannot be overridden in the configuration file. Those are:

Flag	Value
-output	/mnt/onboard/wallabako
-exec	/usr/local/bin/fake-connect-usb

Changing those settings could be dangerous. In particular, changing the -output directory while enabling -delete could delete files unexpectedly if they match the magic pattern (N.epub where N is an integer).

Here's an example of a fully-populated configuration file:

{
  "Concurrency": 6,
  "Count": -1,
  "ClientId": "14_2vun20ernfy880wgkk88gsoosk4csocs4ccw4sgwk84gc84o4k",
  "ClientSecret": "69k0alx9bdcsc0c44o84wk04wkgw0c0g4wkww8c0wwok0sk4ok",
  "Debug": false,
  "Delete": false,
  "Database": "/mnt/onboard/.kobo/KoboReader.sqlite",
  "Exec": "/usr/local/bin/fake-connect-usb",
  "FbinkInteractive": false,
  "LogFile": "/mnt/onboard/wallabako.log.txt",
  "OutputDir": "/mnt/onboard/wallabako",
  "PidFile", "wallabako.pid",
  "Plato": {
    "LibraryPath": "/mnt/onboard"
  },
  "RetryMax": 4,
  "Tags": "",
  "UserName": "joelle",
  "UserPassword": "your super password goes here",
  "WallabagURL": "https://app.wallabag.it"
}

Configuration file is not found even if present

This can happen if you have some sort of a syntax error in the configuration file. For example, this can happen if you have a double-quote in your password and you didn't properly escape it.

The configuration file is a JSON file, parsed by the Unmarshal function of the Golang json package. Wikipedia says that:

JSON's basic data types are: [...] String: a sequence of zero or more Unicode characters. Strings are delimited with double-quotation marks and support a backslash escaping syntax.

This means that if you have a password that is, say foo"bar, you can escape the special character with a backslash, like this:

"UserPassword":  "foo\"bar",

According to the JSON specification, strings should be delimited by double-quote (") and unless you have weird control characters in your passwords (e.g. tab or formfeed), double-quote and backslash are the only characters you should need to escape.

Another common error is to add an extra comma (,) on the final entry, or omitting the brackets ({ or }). Files with BOM markers used to cause issues as well, but that has been fixed in the Wallabago library since 0.7.

Some articles are not downloaded or disappear

If you can't seem to synchronize all your articles and you have a large number of unread articles, you may want to change the Count field in the configuration file. By default, Wallabako only downloads a part of the database: it is limited by the number of articles returned by the Wallabag listing (30 at the time of writing).

Also, if the Delete option is set, older articles will be deleted from the Kobo reader as well.

Note that it should be fairly safe to use a larger number here, as only Concurrency (e.g. 6) articles will be downloaded in parallel at a time. It could make the first listing request slower, however, if you have a huge number of articles. We have reports of operation with 60 articles without significant performance issues.

Unable to open database file

If you see this warning message repeated:

2017/03/01 21:06:49 unable to open database file

It is because the database cannot be found. By default, the database path is hardcoded to /mnt/onboard/.kobo/KoboReader.sqlite, which is likely to work only on Kobo readers. If you are running this on your desktop or another reader, you should disable the database by using the -database flag:

wallabako -database ""

... or configuration option in .wallabako.js:

"Database": "",

Such configuration should silence those warnings as Wallabako will not attempt to open a database file.

Note that the warnings can also safely be ignored.

x509: failed to load system roots and no roots provided

You may see this error when running on weird environments;

2017/01/30 14:45:46 logging in to https://example.net/wallabag
2017/01/30 14:45:51 <nil> Get https://example.net/wallabag/login: x509: failed to load system roots and no roots provided
2017/01/30 14:45:51 completed in 5.12s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11280]

goroutine 1 [running]:
panic(0x2558a0, 0x1061e008)
	/usr/lib/go-1.7/src/runtime/panic.go:500 +0x33c
main.login(0x1060ee20, 0x19, 0x1067d958, 0x7, 0x1060ee60, 0x14, 0x0)
	/home/anarcat/go/src/gitlab.com/anarcat/wallabako/main.go:59 +0x280
main.main()
	/home/anarcat/go/src/gitlab.com/anarcat/wallabako/main.go:147 +0x280

This is because your operating system doesn't ship standard X509 certificates in the location the program expects them to be. A workaround I have found is to copy the /etc/ssl/certs/ca-certificates.crt provided by the ca-certificates package in Debian in the machine.

By default, the tarball creation script adds that magick file to the KoboRoot.tgz archive, which should work around this problem. But this was never tested from scratch.

Note: it may be possible to fix the program to ignore the SystemRootsError but I would advise against it, if only for obvious security reasons...

Command not running

If you notice that udev is not running your command, for some reason, you can restart it with --debug which is very helpful. Example:

[root@(none) ~]# ps ax | grep udev
  621 root       0:00 /sbin/udevd -d
 1242 root       0:00 grep udev
[root@(none) ~]# kill 621
[root@(none) ~]# /sbin/udevd --debug
[1256] parse_file: reading '/lib/udev/rules.d/50-firmware.rules' as rules file
[1256] parse_file: reading '/lib/udev/rules.d/50-udev-default.rules' as rules file
[1256] parse_file: reading '/lib/udev/rules.d/60-persistent-input.rules' as rules file
[1256] parse_file: reading '/lib/udev/rules.d/75-cd-aliases-generator.rules' as rules file
[1256] parse_file: reading '/etc/udev/rules.d/90-wallabako.rules' as rules file
[1256] parse_file: reading '/lib/udev/rules.d/95-udev-late.rules' as rules file
[1256] parse_file: reading '/lib/udev/rules.d/kobo.rules' as rules file
[...]
[1276] util_run_program: '/usr/local/bin/wallabako-run' (stdout) '2017/01/31 00:03:50 logging in to https://example.net/wallabag'
[1256] event_queue_insert: seq 859 queued, 'remove' 'module'
[1256] event_fork: seq 859 forked, pid [1289], 'remove' 'module', 0 seconds old
[1276] util_run_program: '/usr/local/bin/wallabako-run' (stdout) '2017/01/31 00:03:50 failed to get login page:Get https://example.net/wallabag/login: dial tcp: lookup example.net on 192.168.0.1:53: dial udp 192.168.0.1:53: connect: network is unreachable'

In the above case, network is down, probably because the command ran too fast. You can adjust the delay in wallabako-run, but really this should be automated in the script (which should retry a few times before giving up).

Uninstalling

Unfortunately, there is no easy way to uninstall Wallabako (bug 36). While Kobo provides a way to automatically install files on the readers, there is no (at least obvious) way of removing those files once they are installed. To cleanup the install, you need to manually remove the files provided in the KoboRoot.tgz file, which are, at the time of writing:

etc/udev/rules.d/90-wallabako.rules
etc/wallabako.js
etc/ssl/certs/ca-certificates.crt
usr/local/bin/wallabako
usr/local/bin/fake-connect-usb
usr/local/bin/wallabako-run

Of those, the one likely to cause problems is possibly only /etc/udev/rules.d/90-wallabako.rules, and only because it calls /usr/local/bin/wallabako-run which in turns calls /usr/local/bin/fake-connect-usb. So removing the .rules file should be enough to keep Wallabako from starting at all.

If you cannot access those files directly (which is likely, unless you have SSH access or have KSM installed), a workaround disable the configuration file. Remove the .wallabako.js configuration file at the root of your Kobo filesystem: this will "unconfigure" Wallabako which will still fire automatically when wifi comes up, but it will do nothing, which shouldn't cause problems anymore.

Known issues

Like any program, Wallabako is imperfect. There is a list of known issues on the main site, but some issues do not depend on Wallabako, as they are problems with Wallabag itself. Most of the issues we have found in Wallabag are documented in issue #2821.

Credits

Wallabako was written by The Anarcat and reviewed by friendly Debian developers juliank and stapelberg. smurf also helped in reviewing the code and answering my million newbie questions about go.

Also thanks to Norbert Preining for pulishing the Kobo firmware images that got me started into this path and allowed me to easily root my reader. This inspired me to start the related kobo-ssh project to build smaller, ssh-only images.

This program and documentation is distributed under the AGPLv3 license, see the LICENSE file for more information.

Contributing

See the contribution guide for more information. In short: this is a free software project and you are welcome to join us in improving it, both by fixing things, reporting issues or documentation.

Design notes

Moved a separate document.

Remaining issues

There are XXX markers in the source code that show other issues that need to be checked. The other known issues previously stored in this file have been moved to the Gitlab issue queue to allow for better visibility and public collaboration.

Related projects

Other Kobo-related software has support for Wallabako, and may be easier to use than this program.

• KOReader includes a native plug-in that fully integrates with Wallabag servers. The KOReader wiki has installation instructions and details about Wallabag integration. That implementation is discussed in the design document as well.
• Plato also includes an article fetcher with support for Wallabag

Note that Wallabako will also automatically fetch read/unread statuses Plato and Koreader metadata, on top of the built-in Nickel interface shipped with the Kobo.

Documentation

Overview

Parse read status from the Nickel UI database.

Nickel is Kobo's builtin and proprietary UI which stores book details in a SQLite database.

We refer to it as "Nickel" here because that's the internal name used by Kobo. This is to distinguish this UI from the Kobo *device* itself, which also happens to run other UIs and programs like koreader, Plato, etc.

The code simply reads the book status from the sqlite database.

Note that this file relies on an external variable named wallabakoSqliteBackend which is defined in one of nickel_*.go files, depending on compile-time tags. Those files are also responsible for importing the correct modules.