MicroPython-Ctl - a TypeScript library for talking to MicroPython devices

I’m happy to introduce MicroPython-Ctl: a TypeScript library for talking to MicroPython devices (such as ESP32/8266, Raspberry Pi Pico, Pyboard, WiPy, and many more).

Use micropython-ctl to quickly build apps that interact with MicroPython devices: Websites / webapps, Node.js programs, Electron applications, Visual Studio Code extensions, Mobile apps (eg. with React Native) and more.

  • Connect to devices over serial or network interface
  • Run Python scripts, receive the output
  • Manipulate files and directories
  • Terminal (REPL) interaction
  • mctl command-line utility
  • Mount MicroPython devices locally (with FUSE, experimental)
  • Typed and fully async (you can use await with any command).
  • Works on Linux, macOS and Windows

You can see all the features in the documentation, examples and cli/.

Installation

For Node.js and Electron, install micropython-ctl from npm:

# Install with yarn
$ yarn add micropython-ctl

# or with npm
$ npm install micropython-ctl

To use it in the browser, include micropython-ctl like this:

<script src="https://cdn.jsdelivr.net/npm/[email protected]/dist-browser/main.js"></script>

Usage example

const micropython = new MicroPythonDevice()

// Connect to micropython device over network
await micropython.connectNetwork('DEVICE_IP', 'WEBREPL_PASSWORD')

// Or connect to micropython device over serial interface
await micropython.connectSerial('/dev/ttyUSB0')

// Run a Python script and capture the output
const output = await micropython.runScript('print("Hello world")')
console.log('runScript output:', output)  // -> Hello world

// List all files in the root
const files = await micropython.listFiles()
console.log('files:', files)
/* [
  { filename: '/boot.py', size: 31, isDir: false },
  { filename: '/files', size: 0, isDir: true }
] */

// Get information about the board:
const boardInfo = await micropython.getBoardInfo()
console.log(boardInfo)
/* {
    sysname: 'esp32',
    nodename: 'esp32',
    release: '1.13.0',
    version: 'v1.13 on 2020-09-02',
    machine: 'ESP32 module with ESP32',
    uniqueId: 'c44f3312f529',
    memFree: 108736,
    fsBlockSize: 4096,
    fsBlocksTotal: 512,
    fsBlocksFree: 438
} */

// Set a terminal (REPL) data handler, and send data to the REPL
micropython.onTerminalData = (data) => process.stdout.write(data)
micropython.sendData('\x03\x02')  // Ctrl+C and Ctrl+B to enter friendly repl and print version

// Trigger a hard reset of the device
await micropython.reset()

See also:

mctl

mctl is the accompanying cli tool, to interact with a MicroPython device from your terminal:

# Install
$ npm install -g micropython-ctl

# Print the help
$ mctl help
Usage: index [options] [command]

Options:
  -t, --tty <device>                            Connect over serial interface (eg. /dev/tty.SLAB_USBtoUART)
  -h, --host <host>                             Connect over network to hostname or IP of device
  -p, --password <password>                     Password for network device

Commands:
  devices                                       List serial devices
  repl                                          Open a REPL terminal
  run <fileOrCommand>                           Execute a Python file or command
  info [options]                                Get information about the board (versions, unique id, space, memory)
  ls [options] [directory]                      List files on a device
  cat <filename>                                Print content of a file on the device
  get <file_or_dirname> [out_file_or_dirname]   Download a file or directory from the device. Download everything with 'get /'
  put <file_or_dirname> [dest_file_or_dirname]  Upload a file or directory onto the device
  edit <filename>                               Edit a file, and if changed upload afterwards
  mkdir <name>                                  Create a directory
  rm [options] <path>                           Delete a file or directory
  mv <oldPath> <newPath>                        Rename a file or directory
  sha256 <filename>                             Get the SHA256 hash of a file
  reset [options]                               Reset the MicroPython device
  mount [targetPath]                            Mount a MicroPython device (over serial or network)
  version                                       Print the version of mctl
  help [command]                                display help for command

Examples:

# List serial devices
$ mctl devices

# Get information about the board
$ mctl info

# Enter the REPL
$ mctl repl

# List files
$ mctl ls -r

# Print contents of boot.py
$ mctl cat boot.py

# Mount the device (experimental, doesn't handle binary files well)
$ mctl mount

Links & References


I think MicroPython is a wonderful project, in particular for education and rapid prototyping. I hope this library is a small addition to the MicroPython ecosystem, making it easier to use build apps that interact with live devices!

bringing everything together as a fully working first release took way longer than I anticipated, and I’m very happy that it’s finally arrived. I hope you enjoy using it for building things! If you do, let me know.

Feel free to reach out: [email protected] / twitter.com/metachris. I’d love hearing from you.

Background

I’ve playing with MicroPython for several small projects in recent months, and wanted a better way to interface with MicroPython over the network, in particular for websites and Electron applications. I looked into the the official reference webrepl implementations and started writing some code on the off evenings.

And here we are, after a major push around Christmas to wrap the prototyping up into a usable module. As is typical, the “only last few bits” end up being quite a lot of small and medium things, let alone platform compatibility and creating testsuites which in turn discovered a few more issues.

All in all I spent way too much time on this. But the prototype was already so far along, and I wanted to get it to a state so that people can actually use this too.

And now I’m really happy that this point is reached! :)

Rabbit holes & Challenges

There have been various challenges and rabbitholes, and I wanted to give some of them a honorable mention.

Implementing the REPL/WebREPL protocol

The primary challenge was implementing the MicroPython protocol, and making the library work with both serial and the network connections.

To start, there are several REPL modes:

  1. The friendly REPL that’s typically used when connecting to MicroPython. It prints all entered characters and the script output back to you.
  2. The RAW REPL: You can enter it with Ctrl+A, send scripts there (characters are not printed back), and then have the script execute. The standard output and error output can then be collected. See the code here. The Raw REPL mode is used for executing scripts.
  3. There is a new raw-paste mode coming that might be interesting to support.

Furthermore there are a number of commands specific to WebREPL connections: GET_VER, PUT_FILE and GET_FILE.

The reference implementations in the micropython/webrepl repository were very helpful.

Making everything await’able

One of the challenges is making commands (and runScript(..)) asynchronous, so users can use await to wait for the collected output of the command. This is accomplished by creating, storing and returning a Promise that is resolved or rejected at a later time when the matching response has been received.

See code here and here.

Implementing commands for limited system resources

MicroPython devices have pretty limited resources, in particular memory and a slow network connection. It’s important to work with these constraints when implementing commands:

  • Sending scripts in chunks with delays (I’ve arrived at the correct chunksize and delays by trial and error)
  • Uploading files: we don’t want to send the whole file at once since it may be larger than the available memory. The solution is to write the data in chunks (see code here)
  • Files content is uploaded hex-encoded (in Node: data.toString('hex')) and written to the file in binary (Python: f.write(ubinascii.unhexlify(chunk))) (see code here)

It was great to have reference implementations from tools such as webrepl, ampy and rshell.

Creating bundles for Node.js and browsers

I wanted the same codebase to work for both Node.js and browsers. The Node.js module is built using TypeScript (config file here), and the browser module using webpack (config file here).

Code-wise this had several implications:

Mounting the device into the filesystem (using FUSE)

Mounting a virtual filesystem in Node.js is a mess! Using FUSE is generally the way to go, but there are no maintained Node.js bindings :(

It was an interesting experiment to mount the MicroPython device on the local filesystem, but ultimately it doesn’t seem good enough with the current state of the Node.js FUSE bindings.

An interesting thing here is that I didn’t want to include the fuse bindings libraries by default, so whenever mctl mount is called, it checks if the bindings are installed, and if not asks to install it then. See code here.

You can experimentally mount the MicroPython device onto your local filesystem with mctl mount, see more here. It should work with .py files, but downloading binary files will probably not work.

Electron

Electron discourages to use Node.js integration in renderers (see here). Therefore the SerialPort module needs to be preloaded and attached as window.SerialPort like this (preload.js):

window.SerialPort = require('serialport');
Testing

I invested more and more time in testsuites that can be run automatically to test the major functionality of the library (see testsuite.ts).

To test cross-platform compatibility, I’ve spun up a few VirtualBox VMs, especially Windows 10 and Ubuntu Linux. I’m mounting the project directory and run the tests in there as well. Finally this is easy for Windows as well, with the VM images provided by Microsoft.

The tests run in about 6 seconds when using the serial interface, and about 50 seconds when using the network connection

I’d like to run the tests in CI, but they need an attached MicroPython device (the local build of MicroPython doesn’t support a terminal or webrepl). A workaround might be to run a telnet server on the MicroPython instance and have MicroPython-Ctl connect to that.

Needs further investigation.

Send feedback & comments to twitter.com/metachris, or create an issue in the micropython-ctl repository.

🙏

Chris