“Shadowfax” Operating System

Named after the mythological horse, Shadowfax.os is our administrative and deployment system for managing Distilled, our root builder and client accounts, and the varying configurations of all these environments.

We deploy net-new accounts by running the Shadowfax setup scripts programmatically behind an API. The Shadowfax processes coupled with our Waterfall system send messages to manage individual client accounts, allowing us to turn features on and off for individual clients as well as send broadcast messages which will allow us to manage software versions, send patches, and deploy security updates across all of our client accounts which then subsequently update simultaneously.

API Waterfall

The waterfall is our proprietary integration and orchestration solution we created in order to build a “programmable” framework for managing our platform data plane, data processes and its foundational infrastructure.

The waterfall is a packaged system of serverless components that can be used to manage the movement of data through systems by providing methods to store/retrieve data from various data stores, enable stateful data tracking, query internal and third-party APIs, and orchestrate all these processes with a programmable router that can make decisions about data as it flows through a system. There are a number of possible use cases for this type of solution which range from data enrichment, system orchestration and managing backend data flows in an application.

ETL inside of Distilled

The Data Onboarding and ETL module in Distilled leverages white-labeled editions of the best-in-class modern tooling on the market. Our application takes advantage of hundreds of prebuilt connectors offered by services such as Fivetran and Airbyte.

Additionally, for sources that are not presently supported by Fivetran or Airbyte, we offer the following options for onboarding:

• Custom connector and API build-out
• Flatfile import for ongoing/recurring file uploads and mapping to uniform schemas that load the client’s file to their data warehouse upon submission
• SFTP programs hosted on AWS for clients who want to automate file delivery versus the import and mapping process.
• S3 bucket mirroring from their AWS environment to their Distilled client account environment
• Custom Untitled baked Waterfall connectors for complex or high-velocity data sources

Data Warehousing

When a client subscribes to Distilled, we include spin up of a Cloud Data Lake AND Warehouse. This dual-approach creates flexibility for every client based upon their compute needs, and begins persisting data over time that can be utilized for data science and ML applications. The warehouse chosen for a client is determined by the size of their data, the complexity of their regularly occurring queries, and the count of client users that need access to the warehouse. Right now, our supported warehouses include AWS Athena, AWS Aurora Postgres, AWS Redshift, and Snowflake (AWS hosted edition). In our roadmap, we plan to support warehouse configurations outside of the AWS ecosystem such as Google BigQuery and Azure Synapse.

Business Intelligence

Distilled clients are given access to our embedded BI application that includes standard dashboards and reports for common-place data sources. By client request (on a professional services basis), we also build out custom dashboards for our customers. This Distilled module provides business stakeholders with intuitive, accessible and interactive data content that is easy to navigate and pulls insights through “one click” analytics.

For customers that already have an analytics team with a preferred BI tool in place, and are primarily interested in exclusively utilizing Distilled for the out-of-the-box data infrastructure, we offer an application that generates their Data Warehouse connection string so they can take to modeling from their preferred visualization and workbook/spreadsheet environment.

Machine Learning

Distilled comes with a selection of a few pre-built model templates that can be configured to client-specific use cases. This includes categorical models for customer segmentation, lead scoring, marketing mix ROAS optimization, product basket-analysis, and forecasting (in relation to sales/revenue, inventory levels, and other econometric-type analysis and forecasting examples).

At this time, we don’t intend to build a large library of homegrown models, but rather, in our roadmap would like to offer the ability to host notebooks created by customers on our infrastructure.