Introduction

The Lumadata platform is designed to provide data certainty - an iron-clad guarantee that the data you’re brokering to your business teams is accurate. Lumadata continuously observes data in your data warehouse to ensure that data in the data warehouse matches source systems and that previously verified data has not changed unexpectedly. By using a catalog of data observation strategies built from the queries that drive your critical dashboards, reports, AI models, business processes, and algorithms, Lumadata provides certainty to your engineering and business teams that the data they’re using is reliable.

When data issues are detected, Lumadata leverages its fully integrated workflow toolset to engage business and engineering teams simultaneously. Notifications are sent out via email and messenger tools (like Slack) to inform engineers and business teams of data that is not accurate. Work tickets are created for engineers (as in Jira) that enable them to focus on the issues affecting systems and not on communicating with numerous stakeholders. Comments and updates are consolidated in Lumadata from Jira and the overall status of data assets is visible on the Lumadata Data Health Dashboard. The Data Health Dashboard provides an at-a-glance view of data health by subject area across the business with the ability to drill into current and historic data issues.

Activity in Lumadata is logged so that it can be referenced later on. Alerts are attached to observation activities and output is stored along with an impact summary that quantifies the impact on the data that was observed. This gives engineers and business teams a basis for understanding real-world issues affecting datasets under their care. Data observation strategies are version controlled so run history is matched to point-in-time configuration.

How to get started

Lumadata offers free trials to allow business and engineering teams to try the platform without any obligation. Our implementation team will help connect data sources, set up several data observations, and guide your team through the automated workflows Lumadata implements. Our team will also demonstrate key application integrations like Slack and Jira to showcase system integration and also show how to build an observation catalog using our UI or YAML markup within your git repository.

Contact our team to get the account creation process started.

Platform architecture

Overview

Lumadata is built in Amazon's world-class cloud ecosystem. Our services are implemented following best practices with respect to network configuration, network and service isolation, headless architecture, and fault tolerance. Our services backend uses highly scalable, headless components that can scale instantly to handle massive spikes in demand and then go away - eliminating an attack vector since they are not persistent. Our public facing infrastructure is isolated from our private infrastructure. These paradigm ensures our systems are safe, reliable, stable, and available.

Security

We take engagements with our clients seriously and recognize that an invitation to join your data ecosystem comes with critically important mandates around security, privacy, and compliance. In this section, we outline some of the methodologies we employ to implement industry best practices regarding data ecosystem and SaaS software management.

1. Our SaaS platform is deployed in the AWS cloud environment. Communication between services in AWS is always encrypted (this is automatic in AWS). Communications (including communications that may be transporting your data) between our backend processors, our application database, and any ancillary services within AWS are therefore always encrypted in transit.
2. We connect to your systems from dedicated IP addresses. This means your network team can very tightly control ingress of our platform traffic into your network. Connections can be routed through firewalls, load balancers, or peering configurations giving your team full control of the connectivity between our systems.
3. Connections to your systems can be via read-only accounts. You can set up these accounts in your systems and manage them within Lumadata directly.
4. Sensitive data stored in our application database is encrypted at rest. This includes connection information used to connect to your data warehouse and source systems, data snapshots, run history output, API keys, and other such data.
5. Engineers do NOT have access to encryption keys used to encrypt information we store. Further, our standard practice is to inject secret keys used for encryption, APIs, service accounts, and other such use cases automatically during the code build process. Engineers do not require these keys in order to operate the platform which greatly reduces the risk of compromise of keys.
6. You can choose to store your data outside of the Lumadata platform. We store run history, data snapshots, and data profiles in the course of operating our platform, but you may choose through a configuration setting to have this data written back to your systems instead of to Lumadata. This setting uses a separate connection that you configure that uses a user with write permissions to a database you manage. We recommend using a separate, non-critical database for write-back in order to provide operational isolation. Whether stored in Lumadata or your database, the data is still encrypted at rest.
7. When Lumadata identifies data issues, the output is stored to display to engineering teams. This run history can be purged periodically via schedule to reduce the footprint of data stored in Lumadata. As already mentioned, this output data is stored encrypted at rest whether stored in Lumadata or your systems.
8. We only act on the data you specify. Our data observation methodology only requires access to aggregated data from queries that power your dashboards, reports, AI models, etc. We do not require unlimited access to all of your data in order to operate the Lumadata platform. For this reason, when it comes to complying with GDPR, CCPA, HIPPA or other data centric regulation, we suggest not including information in aggregate queries that would be protected by regulation. For example, a count of customer_id would not run afoul of any regulation whereas a report showing customer names would certainly be governed by regulation. Our implementation team can provide guidance on how to build observations that comply with regulatory requirements.
9. You can control who has access to your account including our support team. To enable support to assist you from within the Lumadata platform, you have to add support@lumadata.io to your account. You can also delete support@lumadata.io from your account to remove all access to your catalog.

Reliability and scale

Lumadata is offered on AWS cloud architecture to provide maximum availability and scalability. Our application is partitioned across front-end responsive UI components, an auto-scalable API layer that provides integration and transport, a back-end services layer to compartmentalize key features, and an application database that maintains data consistency. AWS is the leading provider of cloud technology and Lumadata chose to build in their ecosystem to ensure our customers are provided the best performing, most reliable experience possible.