Braiins OS Developer Guide

This project is built around a small tool (braiins build system - bb.py) that extends the OpenWrt build system with the following features:

  • automatic clone/pull of all repositories

  • switching between several configurations

  • out-of-tree repositories for development

  • status of all developed repositories

  • cleaning and purging repositories

  • setup environment for using the OpenWrt toolchain in external projects

  • firmware deployment to NAND/SD over ssh connection or to local repository

  • release version control and firmware signing

  • preparation of feed server with firmware updates

Getting Started

These instructions will guide you through the process of setting up a copy of the project and getting it running on your local machine for development and testing purposes. See the Deployment section for notes on how to deploy the project on a live system.

Prerequisites

The build system is written for Python 3.5. The only prerequisites are:

  • Python 3.5.x

  • virtualenv 15.x.x

  • pip3 9.x.x

Installing

Clone the git repository to some directory and setup your system environment:

# clone braiins OS from git
git clone https://github.com/braiins/braiins.git
cd braiins/braiins-os
virtualenv --python=/usr/bin/python3 bos
source bos/bin/activate
python3.5 -m pip install -r requirements.txt

Building Latest Firmware Images in Docker

The description below shows how to build firmware images generating all local targets (feeds, SD card image and transitional firmware images). This is simplified.

Docker install

Details for installing docker are covered here: https://docs.docker.com/install/linux/docker-ce/ubuntu/ Here's a brief extract of the install:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce
sudo usermod -a -G docker $USER

Docker steps

  • Clone the latest release

git clone --branch firmware_latest https://github.com/braiins/braiins.git
cd braiins/braiins-os
  • Prepare a docker container with all tools:

./docker/build-docker-container.sh
  • Runs a full build for the current revision of the braiins-os release using a test signature key.

./docker/build-in-docker.sh zynq-am2-s17

Docker interactive mode

Call the build release script without parameters to run the docker container in an interactive mode:

./docker/build-in-docker.sh

This mode is intended mainly for testing purpose. The build environment is fully initialized and allows running all bb.py script commands. The current host directory is mapped to the docker container so all files generated by the build process are accessible outside the docker container.

Viewing/Building Documentation Locally

Before pushing a new version of the documentation, it can be viewed locally by jekyll rendering all changes on the fly as follows:

cd docs;
docker run -t --rm -v "$PWD":/usr/src/app -p "4000:4000" starefossen/github-pages

Building

Building a firmware image with the configuration is very simple. Just call the bb.py script with the build command.

# build firmware image with default configuration
python3.5 ./bb.py build

# build firmware image with user configuration
python3.5 ./bb.py --config configs/user.yml build

All repositories are stored in the build/<target> directory where target is specified in a YAML configuration file under a build.name attribute.

Platform Selection

The build system supports multiple platforms with the same base configuration. Currently the following platforms are supported:

  • zynq-am1-s9 (Antminer S9)

  • zynq-dm1-g9 (DragonMint v1 with G9 Control Board)

  • zynq-dm1-g19 (DragonMint v1 with G19 Control Board)

The platform zynq-dm1-g19 is specified in the default configuration file but it can be modified from the command line with a platform parameter:

# build firmware image with default configuration for DragonMint G9
python3.5 ./bb.py --platform zynq-dm1-g9 build

# build firmware image with user configuration for DragonMint G19
python3.5 ./bb.py --config configs/user.yml --platform zynq-dm1-g19 build

Firmware Release

The firmware with specific version has tag in a git repository which contains a modified configuration set to exact commit of all dependent repositories. The tag can be checked out for specific firmware version. Afterwards, we can call build command for reproducible firmware release.

Signing

By default the resulting firmware image and packages are signed by a test key which is specified in the default config by a build.key attribute and is stored in a keys directory. The release key is usually stored in a fortified keyring and is securely used during actual release.

To set this key without changing the configuration file, it is possible to use a key parameter of the build command:

# use secret key for signing (the public key is '/path/secret.pub')
python3.5 ./bb.py build --key /path/secret

# secret and public key can be specified in one parameter
python3.5 ./bb.py build --key /path/secret:/path/public

Development

Fetching

The default configuration on master branch has disabled automatic fetching/merging of remote repositories when the build command is executed. The prepare command can be used instead when synchronization with remote is needed.

# force fetching from remote repositories
python3.5 ./bb.py prepare --fetch

Cleaning

It is possible to clean all projects with two options:

Simple execution of clean command runs the OpenWrt make clean to clean the whole build system. It does not guarantee that all files will be in their initial state.

The second option uses git command to clean all repositories. The command after clean also runs the initialization phase again and prepares the repository for its first build. This option removes all untracked files and must be called with caution!

# clean repositories with the OpenWrt make clean
python3.5 ./bb.py clean

# reset repositories with git clean
python3.5 ./bb.py clean --purge

Status

The braiins OS project consists of several git repositories. It is convenient to track the status of all changes in all repositories at once during development. The status command can be used for this purpose. It is similar to git status but it is executed for all repositories.

# get status of all repositories
python3.5 ./bb.py status

Out-of-Tree Build

Rather than executing the whole OpenWrt build system which can be slow, we can run a separate build of a subproject (e.g. CGMiner) with the OpenWrt toolchain. Environment variables must be set correctly for using the OpenWrt toolchain in out-of-tree projects. For this purpose, the toolchain command is provided.

# set environment variables for OpenWrt toolchain out-of-tree use
eval $(./bb.py toolchain 2>/dev/null)

Configuration

The build system supports multiple configurations specified by a configuration file stored in YAML format. The current configuration can be changed from the command line. It is also possible to alter the most important parameters without modifying the underlying configuration file from the command line.

The configuration is divided into two categories. The first one is a target-specific configuration which is handled exclusively by the braiins build system and can be adjusted only in the YAML configuration. The second one is a package configuration used for image content description which is handled mainly in the OpenWrt menuconfig.

YAML Structure

The main configuration file is stored in a standard YAML 1.2 format. The format expects predefined hierarchical structure which is formed by categories on the global level. The categories can be further divided into subcategories or they can directly contain configuration attributes.

The string attributes can use special syntax for parameter expansion which is extension of standard YAML format. The name of the parameter for expansion is enclosed in {} and can be used anytime in the string. The list of supported parameters is as follows:

  • platform - the name defined in a bos.platform attribute (it has form <target>-<subtarget>)

  • target - the name of target architecture (e.g. zynq) that is derived from bos.platform attribute

  • subtarget - the name of target device (e.g. dm1-g19) that is derived from bos.platform attribute

  • subtarget_family - the name of the family of the subtarget (e.g. dm1) that is derived from bos.platform

  • build_dir - build directory e.g. build/<target>

Curly brackets are also used by the YAML for dictionary in an abbreviated form. If a string starts with a curly bracket, then it must be quoted to distinguish meaning:

# use quotes when string starts with {
name: '{target}'

# form without quotes when it is not ambiguous
sd: output/{platform}/sd/

The default configuration file is fully commented, so the following list of global categories is only a short description:

  • bos - the settings concerning one instance of a bOS device (platform, HWID, default firmware pattition); the default configuration is used only for testing and is usually overridden from the command line during release process

  • net - the network configuration of a target bOS device (MAC, hostname, static IP)

  • miner - the settings concerning only miner-specific attributes (default pool)

  • build - the configuration of the build process (path to OpenWrt configuration, build directories, keys, ...)

  • remote - the list of all remote repositories with parameters for fetching; the parameters fetch and branch used as a default value for all repositories could be overridden in a specific repository by parameter of the same name

  • local - the configuration of output directories for local targets for deployment

  • feeds - the settings of feeds fetching and installation

  • uenv - the configuration of uEnv.txt content (this file is used only for SD images)

  • deploy - the list of targets for deployment and configuration of this process (e.g. reset of target environment, remote ssh connection, ...)

Local Configuration File

It is possible to override each parameter specified in the default configuration file with parameters defined in a local file .local.yml stored in the root directory of the braiins build system.

The structure of the local configuration is the same as default configuration file. Typically, only platform or MAC addresses of the target are being overriden:

bos:
  # possible platforms are zynq-dm1-g9, zynq-dm1-g19, zynq-am1-s9
  platform: zynq-am1-s9

net:
  # default bOS device MAC address
  mac: 00:0A:35:FF:FF:00

During the release process, this local configuration is ignored to avoid interference with the default configuration!

CLI Parameters

The braiins build system supports multiple configurations which can be selected by a global parameter --config. When the script is run without this parameter, configs/default.yml is used. The --platform parameter can be used for changing the target platform.

Global configuration parameters must be consistently used with all commands to guarantee predictable results!

The build system commands are described in detail in separate sections. Below is a list of supported commands:

  • prepare - fetch all remote repositories and prepare source directory

  • clean - clean source directory

  • config - change default configuration of OpenWrt project

  • build - build image for current configuration

  • deploy - deploy selected image to target device (NAND/SD over ssh or to local directory)

  • status - show status of all local repositories (git status equivalent)

  • toolchain - set environment for OpenWrt toolchain (out-of-tree build)

  • release - create branch with configuration for release version

  • key - generate build key pair for signing firmware tarball and packages

Some commands have special parameters which are mapped to the default YAML configuration file. The CLI parameters have the highest priority and cannot be overridden by local configuration files.

Remote Repositories Configuration

The list of all remote repositories for fetching by the braiins build system is specified in the remote attribute in the YAML configuration file. The use of all parameters are shown in the following example:

remote:
  # location aliases for remote repositories
  aliases:
    bos: '{meta_repo}'
    openwrt_feed: git://git.openwrt.org/feed

  # default location for remote repositories
  location: bos
  # default branch for repositories
  branch: master

  # list of remote repositories
  repos:
    lede:
      # use default location and branch
      project: lede.git
    lede-packages:
      location: openwrt_feed
      project: packages.git
      branch: cd5c448758f30868770b9ebf8b656c1a4211a240
    cgminer:
      # use different branch for specific patform
      match:
        zynq-dm1:
          project: cgminer.git
          branch: braiins-dm1
        zynq-am1:
          project: cgminer.git
          branch: braiins-am1

Below is a short description of supported parameters:

  • aliases - the list of URI with symbolic name which must be used in location parameter

  • location - the alias which represents URI of remote location (this parameter can be omitted when default location is set)

  • project - the name of remote repository (it is concatenated with the location URI)

  • branch - the name of branch (this parameter can be omitted when default branch is set)

  • match - special syntax for platform specific configuration (the selection is based on pattern with platform prefix - the longest prefix is chosen for the current platform)

Packages

The standard OpenWrt menuconfig is used for firmware image configuration. When some changes are detected, the difference in configuration is saved to the file specified in YAML configuration file under build.config attribute.

# configure image packages
python3.5 ./bb.py config

Multiple firmware images are being built at once (NAND, NAND Recovery, SD, ...). We must be specify which image will contain a particular package. It is done in two ways:

  • When a package is installed to all images without exception then only OpenWrt menuconfig is used where the package must be selected by asterisk symbol <*>

  • When a package is installed only to specific images then the package must be selected as a module <M> and added to an external package list specified in a build.packages attribute.

The package file is just another YAML structured format that stores lists with inheritance support. The lists with image_ prefix are used for description of installed packages in specified image:

  • image_sd - SD image with extroot support (second partition in the ext4 format is used as an overlay)

  • image_nand - standard NAND image

  • image_recovery - special NAND recovery image (it also supports factory reset)

  • image_upgrade - NAND image for generic stage1 upgrade process from different firmwares

The structured list has the following format:

list_name:
  # inheritance is specified as a list of base lists
  # root list has this parameter omitted
  base:
    - child1
    - child2
  # the list items are specified under separate parameter
  # the resulting list is merged with base lists in order:
  # child1.list, child2.list, item1, item2
  list:
    - item1
    - item2

Kernel

The config command can also be used for the Linux configuration when --kernel parameter is specified. The resulting configuration is then saved in the OpenWrt build system in the target directory. It is standard behavior of the OpenWrt.

# configure kernel (Linux) for selected target
python3.5 ./bb.py config --kernel

Deployment

Whenever firmware images are built by the OpenWrt build system, it is possible to deploy them over ssh connection directly to the running machine (when it runs compatible firmware) or store it to a local path. The default configuration builds all local targets and stores its result to predefined location output/<platform>. It is convenient for testing when we want to verify all possible targets. However, for real deployment, it is more useful to specify a target from the command line.

System Upgrade vs. Deployment

Do not confuse the deployment process with the system upgrade! The deployment is used mainly for developers for testing the firmware on running devices or for initial factory NAND programming. For a system upgrade, use standard firmware tarball which can be loaded with the help of a web interface or with the OpenWrt sysupgrade utility. Follow the user manual for standard firmware upgrade procedure

# download latest packages from feeds server
opkg update
# try to upgrade to the latest firmware
opkg install firmware

Remote Targets

Only commonly used remote targets will be described here. Special targets - useful during development of specific firmware parts - will be omitted. With remote targets, it is possible to deploy either NAND image or SD image (in case that the SD card is inserted into the SD slot). The NAND image can be deployed even if the braiins OS is run from NAND and a UBI partition is mounted. The following targets are supported:

  • sd - writes U-Boot and Linux image with a SquashFS root file system to the SD card

  • nand - writes U-Boot and UBI image with the Linux kernel and a SquashFS root file system to the NAND (the writable overlay uses a UBIFS file system)

Let's assume local network with one instance running braiins/OpenWrt firmware and default configuration of the build system. The following command can be used for deployment of SD or NAND image to this machine:

# mount mmc0 partition 1 and copy all images and 'uEnv.txt' to it
python3.5 ./bb.py deploy sd

# write U-Boot, recovery image and configuration to NAND and do factory reset
python3.5 ./bb.py deploy nand

When more than one device needs to be managed, several arguments can be used to specify remote machine. It can be done only by machine's MAC address specification or even with a hostname when local DNS server does not work correctly or when the MAC address does not correspond with the hostname.

Be very cautious with MAC address! If the --mac parameter is omitted, the default MAC address from configuration file is used (00:0A:35:FF:FF:FF) and remote machine is upgraded with it. Therefore, it is recommended to use hostname only in situations when devices MAC address needs to be changed.

The hostname is determined from MAC address when not specified. The machine generates its name based on current MAC in a form of {MACHINE_CLASS}-xxyyzz where MACHINE_CLASS is e.g. miner and xxyyzz are last three numbers from its address.

# upgrade remote device with the hostname 'miner-ffff01'
python3.5 ./bb.py deploy nand --mac 00:0A:35:FF:FF:01
# upgrade remote device on address '192.168.0.1' and change its MAC to '00:0A:35:FF:FF:FF'
python3.5 ./bb.py deploy nand --hostname 192.168.0.1
# upgrade previous device and set its MAC to original value
python3.5 ./bb.py deploy nand --mac 00:0A:35:FF:FF:01 --hostname miner-ffffff

There are also special configuration sub-targets which modify only bOS configuration and do not touch other parts of the NAND or SD partition:

  • sd_config - modify only uEnv.txt file on SD card which is read by the U-Boot

  • nand_config - modify only NAND U-Boot environment and bOS configuration partition

Local Targets

Local targets can be used for deploying images to locations specified by a file path. The default configuration enables all local targets for storing all images to a predefined directory output/<platform>. There are also special local targets for deployment utilities used for upgrading the original firmware to the braiins/OpenWrt one. The other special target is for a feeds server preparation used for upgrading braiins/OpenWrt firmware with a standard OpenWrt opkg utility. The following list specifies main local targets:

  • local_sd - the same function as remote target but target is specified by a local file path

  • local_sd_recovery - writes special SD recovery image to a local file path (e.g. it can be used for repairing a 'bricked' machine that doesn't boot from its flash memory anymore)

  • local_upgrade - various images needed for upgrading an original firmware for target platform

  • local_feeds - sysupgrade tarball with current firmware and packages needed for creating standard OpenWrt feeds server

Similarly to the remote targets there are also configuration targets:

  • local_sd_config - modify only uEnv.txt file (useful for changing parameter sd_boot)

  • local_sd_recovery_config - modify only uEnv.txt file (useful for changing parameters sd_boot, factory_reset and sd_images controlling SD recovery image for factory reset)

The output location is usually specified by the command line. Since more than one target can by specified at once there is special notation for passing local file path to the specific local target:

<local_target>[:<path>]

Device MAC address can also be specified with --mac parameter. However, it is only used for generating the uEnv.txt. This MAC address is used when booting the device from an SD card. The --hostname parameter is ignored for local targets. There are several useful parameters for bOS configuration which will be described in the next section.

Below are a few typical examples of deploy command for local targets:

# create SD card with default MAC address without SD boot parameter
# a hardware jumper on control board have to be connected to boot from this SD card
python3.5 ./bb.py deploy local_sd:/mnt/mmc0

# create SD card with MAC address '00:0A:35:FF:FF:01' and with SD boot enabled
# it can boot from SD card without connecting a hardware jumper if compatible U-Boot is used
python3.5 ./bb.py deploy local_sd:/mnt/mmc0 --mac 00:0A:35:FF:FF:01 --uenv sd_boot

# create recovery SD card which boots from SD and performs NAND factory reset using images stored on this SD
python3.5 ./bb.py deploy local_sd_recovery:/mnt/mmc0 --mac 00:0A:35:FF:FF:01 --uenv sd_boot factory_reset sd_images

# create special SD card only with 'uEnv.txt' which performs factory reset when it is inserted in a device
python3.5 ./bb.py deploy local_sd_config:/mnt/mmc0 --uenv factory_reset

uEnv

When U-Boot finds inserted SD card it tries to load a file uEnv.txt from its first partition formatted with FAT file system. There are environment variables which can alter U-Boot behavior during boot process. There are standard U-Boot variables (e.g. ethaddr) and some additional ones are provided by braiins/OpenWrt firmware. Configuration of these variables can be done in the braiins build system YAML file in uenv section. These parameters can also be passed by command line argument --uenv. The following list shows all supported settings:

  • mac - set device MAC address (generates ethaddr variable)

  • factory_reset - when SD has this variable enabled and is inserted into the device, the device performs factory reset

  • sd_images - used for factory reset images from SD (factory_reset must also be enable)

  • sd_boot - boot kernel image from SD (the U-Boot is still booted from the NAND)

The sd_boot requires compatible and functional U-Boot on NAND. When the NAND is corrupted it may not work. In that case a HW jumper must be used for a miner control board reconfiguration. E.g. J2 pins must be bridged on G9/G19 boards to change boot mode from NAND to SD card.

Default Pool

Each miner with the same firmware can store different default pool. The information is stored in a miner configuration partition in the NAND. For SD version this functionality is not currently supported. The default pool can be changed from command line with corresponding arguments of deploy command. However, these arguments have effect only for remote targets and for special local targets for an original firmware upgrade. The deploy command supports the following arguments:

  • pool-url - the address of pool server in a format <host>[:<port>] (stratum+tcp://stratum.slushpool.com:3333)

  • pool-user - the name of user and worker (braiinstest.worker1)

Release Management

The braiins build system also has tools for firmware versioning which is used in release cycles. It is based on git repository with tags which holds name of a firmware version and configuration for reproducible firmware build. The release cycle has three stages:

  1. new version creation,

  2. signed firmware building,

  3. publication.

Versioning

The first stage is about git branch creation, modification of default configuration file where each repository points to specific commit and tag creation with a name representing current firmware version. All this can be done by one command with a name release. This command requires that the braiins build system repository and all dependent repositories are clean. After successful call of this command, a remote tag is created with the following version format:

firmware_<YYYY-MM-DD>-<patch_level>-<short_sha>

The <YYYY-MM-DD> represents a date of the braiins build system commit from which is a release created. The value of the <patch_level> is usually 0 and is incremented only in situation when more then one release is created in one day. This increment is done automatically and depends on correctly created git tags. The <short_sha> is a SHA prefix of the commit used for the date. The prefix is 8 characters long.

The release command has also --include argument which is used for specification of a firmware tarball content. In a special situation that a new firmware needs to upgrade also a U-Boot or a FPGA bitstream. Occasionally, a bash script (COMMAND) can also be added. It is run before in pre-init phase of the standard system upgrade process. It can contain some control checks or fixes of previous firmware running on a device. The source code of this script is stored in the OpenWrt repository but must be configured externally that it is included to the output image. The following list contains all sysupgrade components supported by the firmware:

  • command - bash script executed during firmware system upgrade

  • uboot - the U-Boot image for upgrading previous one (it can brick the device)

  • fpga - the FPGA bitstream (the device has auto recovery process which can rescue a device when the new bitstream does not work)

# create git tag and push it to the remote repository
python3.5 ./bb.py release

# do the same but also include 'COMMAND' script and new FPGA bitstream
python3.5 ./bb.py release --include command fpga

Building and Signing

The official firmware is signed with publisher key which should be private. Only one key should exist and be stored in some secured keyring. The key can be generated by the braiins build system with the following command:

# generate key pair and store it to the fortified keyring
python3.5 ./bb.py key ~/keyring/secret

This command generates private and public key into the specified path. Where the private key is to be securely stored is beyond the scope of this description. This key is usually generated only once and is used for signing of all the releases firmwares.

After the release has been created with the release command, it can be built and signed with the following command:

# switch braiins build system to specific firmware version
git checkout firmware_2018-05-27-0-16a21b55
# build this version and sign it with a secret key
python3.5 ./bb.py build --key ~/keyring/secret

If everything goes well, all images are prepared for final publishing to the feeds server. This process can be reproduced anytime in the future.

Feeds Server

The final stage of release management is publishing to the feeds server. It is standard OpenWrt feeds server with the Packages.gz file containing list of ipk packages in a text format. All files needed for this feed server can be created by deploy command with local_feeds target:

# initial feeds server is created by deploy command with 'local_feeds' target
python3.5 ./bb.py deploy local_feeds:~/server/initial_feeds

# the other deployments should be created with the previous contents
python3.5 ./bb.py deploy local_feeds:~/server/new_feeds --feeds-base file://~/server/initial_feeds/Packages

The output directory should be empty before calling deploy command to ensure that the directory would not contain any temporary files. If feeds server contains previous firmwares too the --feeds-base should be called to merge previous Packages index file with new firmware. The previous Packages index file can also be edited before new deployment to prune some old firmwares from the server.

All generated files are described in the following list:

  • firmware_<version>.tar - signed tarball with all images for device system upgrade compatible with sysupgrade utility or LuCI web interface (this file can be used directly without OPKG utility)

  • firmware_<version>.ipk - standard OPKG package with firmware metadata used for installing new firmware (it downloads corresponding firmware_<version>.tar from feeds server and initiate system upgrade)

  • Packages - feeds index file with a list of all packages in a text form (it contains references to firmware_<version>.tar)

  • Packages.gz - gzipped Packages file

  • Packages.sig - the file that contains signature for Packages.gz

Upgrade from Original/Factory Firmware

The example below for Dragon Mint DM1 shows how to upgrade the factory firmware to braiins OS firmware:

# create upgrade tarballs for DragonMint control board
python3.5 ./bb.py deploy local_upgrade:~/upgrade

# extract upgrade tarball for new DragonMint with G19 control board
cd ~/upgrade
tar xvf braiins-os_dm1-g19_ssh_<version>.tar.bz2
cd braiins-os_dm1-g19_ssh_<version>

# run generated upgrade script from local host and initiate upgrade over ssh connection
python3 ./upgrade2bos.py 192.168.0.1

There are two versions of original firmware. Therefore, an appropriate deploy target must to be used:

  • braiins-os_dm1-g19_telnet_<version> - initial release of the firmware that had only telnet server for remote access

  • braiins-os_dm1-g19_ssh_<version> - improved release of the firmware that had ssh server instead

You have to get login information for root access over telnet (v1) or ssh (v2) for your DragonMint miner before you start the upgrade process. Without this information you have to open your miner and use SD version for boot and deploy this firmware with the braiins build deploy command with nand target.

Authors

  • Libor Vašíček - Initial work

License

It is released under the GNU General Public License v3.0 (GNU GPLv3).

See LICENSE file.