Steampipe dashboards and benchmarks for your data

In Steampipe unbundled we confirmed how its plugins, which initially labored solely with the overseas knowledge wrapper loaded into Steampipe’s batteries-included Postgres, are actually additionally obtainable as stand-alone distributions which you could load into your personal cases of Postgres or SQLite. Now Steampipe itself is unbundled: its dashboard server and benchmark runner have migrated to a brand new open-source venture, Powerpipe.

Once you begin Powerpipe it defaults to a neighborhood occasion of Steampipe so current dashboards and benchmarks work as at all times. The mods that drive them supply their knowledge from the identical plugins, utilizing the identical SQL queries. However you should use Powerpipe’s --database argument to specify a Postgres, SQLite, DuckDB, or MySQL connection string.

Why unbundle Steampipe on this means? Unix philosophy holds {that a} software program software ought to do one factor effectively, and mix readily with different instruments that abide by the identical precept. Steampipe started that means, however because the dashboard and benchmark layers appeared it grew to become clear that refactoring was so as. For devops engineers who use the software to visualise and assess cloud infrastructure, it is sensible to decouple these layers into elements with well-defined interfaces. However Powerpipe is not only for them. Among the many set of recent mods launched with Powerpipe, there’s one which visualizes biomedical knowledge from RNACentral by the use of a public Postgres endpoint. To view these dashboards you put in the mod, then begin Powerpipe like so:

powerpipe server --database postgres://reader:NWDMCE5xdipIjRrp@hh-pgsql-public.ebi.ac.uk:5432/pfmegrnargs

The dashboards are actually browseable at http://localhost:9033.

You may in fact join any Postgres shopper to that endpoint. What Powerpipe brings to the social gathering is an as-code method to visualizing and validating knowledge. The widgets that drive dashboards and benchmarks are written in a declarative language, HCL, that enhances the declarative SQL queries that fill these widgets with knowledge. All of your HCL and SQL code lives in repos, below model management, open to the identical type of collaboration that you just count on and luxuriate in for all different code artifacts.

Working with that code is a developer-friendly expertise in one other means too. Powerpipe watches your recordsdata and reacts immediately while you make modifications to both the HCL that configures dashboard widgets or the SQL that populates them.

IDG

SQL-powered relationship graphs

The suite of dashboard elements consists of every part you’d count on, together with tables, charts, infocards, and enter widgets. It additionally consists of some belongings you won’t count on, notably the nodes and edges that kind relationship graphs. Right here’s an instance that makes use of a choose widget to decide on an InfoWorld writer, then builds a graph that relates the chosen writer to articles written and to the (maybe a number of) classes related to every article.

IDG

Right here’s how the nodes and edges are outlined. Two of the nodes question for classes and authors, and the third makes use of the RSS plugin to question the chosen writer’s RSS feed. Then two edges join the nodes. One relates article hyperlinks to authors, the opposite relates the identical hyperlinks to classes. It’s all simply SQL, leveraged in an unconventional means.

-- enumerate classes
node {
  class = class.class
  sql = <<EOQ
    choose
      class as id,
      class as title
    from
      infoworld_categories()
  EOQ
}

-- enumerate authors
node {
  class = class.writer
  args = [self.input.authors.value]
  sql = <<EOQ
    choose
      writer as id,
      writer as title
    from
      infoworld_authors()
    the place 
      writer = $1
  EOQ
}

-- checklist articles by writer
node  '/index.rss' as feed_link
    from infoworld_authors()
  )
  choose
    hyperlink as id,
    title as title,
    writer,
    jsonb_build_object(
      'hyperlink', hyperlink,
      'writer', writer,
      'revealed', revealed
    ) as properties
  from
    rss_item r
  be a part of
    feed_links f
  utilizing (feed_link)
  the place writer = $1
  order by 
    revealed desc
  EOQ


-- relate articles to authors
edge  '/index.rss' as feed_link
      from infoworld_authors()
    )
    choose
      hyperlink as to_id,
      writer as from_id
    from
      rss_item r
    be a part of
      feed_links f
    utilizing (feed_link)
    the place
      writer = $1
  EOQ


-- relate articles to classes
edge {
  sql = <<EOQ
    choose
      hyperlink as to_id,
      class as from_id
    from
      infoworld_category_urls()
  EOQ
}

Extra typical makes use of of those relationship graphs assist devops engineers perceive how the weather of their cloud infrastructures match collectively, as for instance on this graph which exhibits how the elements of a Kubernetes deployment—deployments, replicasets, pods, containers, servers, and nodes—relate to at least one one other. Every entity within the graph is hyperlinked to a different graph that drills into the entity and particulars its surrounding infrastructure.

IDG

These visualizations, that are additionally obtainable for AWS, Azure, and GCP, are a superb solution to discover and perceive your cloud architectures. And the queries you write to try this are reusable. You may circulation the identical knowledge into dashboard charts and tables.

SQL-powered benchmarks and controls

Powerpipe can also be the engine that powers suites of compliance benchmarks, additionally for AWS, Azure, GCP, Kubernetes, and others. Right here’s a benchmark that validates the picture URLs embedded within the extensions subject of InfoWorld RSS feeds.

IDG

And here is the way it’s outlined. The benchmark consists of one management that runs a multistep question to pick feed URLs for authors, drill into every merchandise’s media tag, then verify the HTTP response code for every URL.

management "image_urls" 

benchmark "feeds" {
  title = "Examine InfoWorld feeds"
  youngsters = [
    control.image_urls
  ]
}

Notable factors right here:

• The feed_link column returned from the primary CTE (frequent desk expression) joins with its counterpart within the RSS plugin to fetch the feed for every writer.
• As a result of the again finish is Steampipe, which is Postgres-based, the Postgres JSONB operators can be found to drill into the media tag and extract the URL.
• Every url then joins with the corresponding column of the Internet plugin—an HTTP shopper wrapped as a database desk!—to verify the response code.

A management is only a SQL question that returns the required columns standing, motive, and useful resource. You may consider a management as a unit take a look at for knowledge, with Powerpipe because the testrunner.

These controls sometimes help commonplace compliance suites: CIS, FedRamp, GDPR, HIPAA, NIST, PCI, SOC 2, and extra. Mods like AWS Compliance supply broad and deep help for these, constructing on the equally broad and deep API protection offered by Steampipe plugins just like the one for AWS.

However you’ll be able to construct benchmark mods to validate any type of knowledge, wherever it lives: in ephemeral tables populated by cloud companies through plugins, or in your personal databases as native tables.

Dashboards and benchmarks as code

Whether or not you’re visualizing knowledge with interactive tables, charts, and graphs, or validating knowledge utilizing controls, the mannequin is identical. You employ SQL queries to amass the information, and HCL widgets to show it, with reside enhancing in each circumstances. The code lives in packages known as mods which you could set up, create, and remix.

Just like the Steampipe product from which it was decoupled, Powerpipe is a single binary which you could run domestically, or in a cloud VM, or in a CI/CD pipeline. And like Steampipe, it’s obtainable in hosted kind at Turbot Pipes the place you’ll be able to collaborate together with your staff and share snapshots of dashboards and benchmarks.

The prevailing suites of Powerpipe dashboards and benchmarks deal with what devops people want most: a typical question language, with reside entry to cloud APIs, embedded in HCL wrappers that reside in repositories together with the remainder of your managed code. That’s the candy spot, however with the unbundling of Steampipe now you can use the identical applied sciences extra broadly.