neilotoole.io

Web presence of Neil O'Toole.

Tag: Sq

sq passes 500 stars

It’s been a few months since I posted about sq, but a lot has been going on. We’re now on sq v0.48.3, and the project has passed 500 stars ⭐️ on GitHub.

The changes since the last sq post are too numerous to list here (see the extensive CHANGELOG for details), but some highlights include:

  • Large performance improvements to sq diff.
  • Caching mechanism for ingested data sources (such as CSV, JSON, etc.).
  • Significantly improved Excel support.
  • A progress bar implementation, particularly important for long-running operations such as ingesting large document sources. sq_inspect_remote_s3.png

sq package for Void Linux

Thanks to the efforts of Void Linux contributor @icp, there is now a sq package for Void Linux’s XBPS package manager.

# Update to get latest repo info
$ xbps-install -Syu;  xbps-install -yu xbps

# Install sq
$ xbps-install -yu sq

# Run sq
$ sq version
sq v0.42.1

Or, just use the install script, which detects Void Linux and calls xbps-install under the covers:

$ /bin/sh -c "$(curl -fsSL https://sq.io/install.sh)"

sq v0.41.0

sq v0.41.0 has been published. This release focuses heavily on improvements to Excel support, including:

  • An under-the-hood rewrite, and switch to a new backing Excel driver.
  • The ingest process can now handle duplicate column header names, per #99.
  • And it will now also auto-detect header rows, like CSV has done for ages.
  • And a bunch more minor improvements, including better ingest performance.

Download via sq.io.

sq v0.40.0

sq v0.40.0 is a major release, with a completely rewritten join mechanism. Previously, only a single join was supported, but now you chain as many as desired.

$ sq '.actor | join(.film_actor, .actor_id) | join(.film, .film_id) | .first_name, .last_name, .title'

There’s also support for the full family of join types (inner, left, right, cross, etc.).

Download via sq.io.

shelleditor

shelleditor is a tiny package I’ve put together to call out to the system editor from within a Go CLI. Like most everybody in the Kubernetes ecosystem, I spend a lot of time with the kubectl CLI, and more time than I’d like doing kubectl edit X, to directly edit k8s config for debugging.

Recently, I’ve been working on a complete overhaul of sq’s config mechanism. I may have gone a little overboard.

sq is highly configurable, via flags, via its config file, and via sq config set.

However, when making changes to a sq source, who can remember all the flags? Yes, there’s shell completion for those flags, but there’s a lot of flags. Wouldn’t it be easier to be able to see all the available flags in front of you, and edit them as desired? Kinda like what kubectl edit does?

Enter sq config edit:

sq config edit

That idea’s inception led down a deep rabbit hole, worthy of its own post. But for the purposes of this post, I had an immediate problem to solve: how to call out to the system editor, e.g. vi. I found a standalone package, tj/go-editor, but it didn’t seem to be actively maintained, and anyhow, I was curious to see how kubectl did it. After eventually figuring out how to navigate the Kubernetes source code, I decided I may as well just lift the kubectl edit implementation from them. Alas, a simple import pulled in an ungodly amount of Kubernetes dependencies. So, I extracted the essentials, and the shelleditor package was born.

sq v0.5.0

Back in the sq saddle after some downtime: sq v0.5.0 is now available. Finally there’s conditional selects (basically, the SQL WHERE clause), and a number of other improvements. Download via sq.io.

sq v0.4.0

sq v0.4.0 is out. There’s a bunch of additional output formats in this release: Excel, CSV, TSV, and XML. Download via sq.io.

sq v0.3.0

sq v0.3.0 is done. The major new feature is the addition of the sq inspect command, which provides metadata about the datasource (schema, tables, cols, etc). Download via sq.io.

sq v0.2.1

sq v0.2.1 is now available. There’s a bunch of bug fixes, but the big new feature is cross-datasource joins (you can join an Excel spreadsheet with a MySQL DB, etc.) Download via sq.io.

sq v0.1.11

sq v0.1.11 is now available. This is a bug-fix release. Download via sq.io.