Workflow Overview

Data issues happen unexpectedly, without notice, and have far-reaching downstream consequences.

• Teams using data feeds to operate their department may use erroneous data to make wrong decisions.
• Analysts may find patterns that become void when they learn that their work was built on questionable data.
• Predictive models may turn left when they should have gone right if they're fed bad signals.

Data has to be monitored and whenever issues are found, teams throughout the business need to be notified immediately. Not only does this create transparency and build trust, it prevents harm and wasted work effort. Through diligent monitoring and alerting, Lumadata helps businesses build accountability in data quality and certainty that data is accurate.

Whenever Lumadata detects an issue, it suspends the observation to prevent duplicate alerts. Lumadata builds an impact summary describing the impact to the data issue that was found. An alert is also created and sent to business and engineering teams. Notifications are sent through many messaging channels including email and Slack. Jira tickets are created for engineers and activity between Jira and Lumadata is automatically synchronized so business teams have information without disrupting engineering workflows. Lumadata supports your team through its workflow from issue identification to resolution.

Run Logs and Impact Summaries

Each observation that runs in Lumadata is logged. Whenever issues are detected, Lumadata computes an impact summary that describes the impact of the issue that's detected. If values between two datasets don't match, Lumadata will calculate the variance and report it as part of the impact summary. It identifies issues on a dataset and column level meaning it can determine that "6% of all data is affected" but that only 1 column is showing an issue. This allows business teams to make decisions about using data while it is being worked on. It also indicates to engineers where to focus their troubleshooting efforts.

The run logs also record the specific rows that triggered alerts with a description of exactly what went wrong. This speeds up the diagnostic process and quickens teams down the path to resolution. Logs can be exported for further analysis if required, but usually the issue is clear from within Lumadata.

Alerts and Notifications

Alerts are created from run logs in a "Failed" status. To prevent duplicate alerts, Lumadata automatically suspends activities that have triggered an alert. Alerts trigger many integrated functions automatically as well:

• Notifications are sent out via email based on notification profiles set up in Lumadata
• Messages are sent to shared messenging channels in popular messengers like Slack and Discord
• Jira tickets are created in the active sprint so engineers can start the team's support workflow. Comments and status updates in Jira are visible in Lumadata so business teams stay informed. Completing a ticket in Jira automatically activates observations in Lumadata.

Alerts create transparancy into the state of data in the data warehouse and demonstrate accountability for data assets. Alerts support the partnership between business and engineering and allow both teams to get the most from their data products.

Data Health Dashboard

Lumadata provides an at-a-glance view of data health right on its dashboard. This allows business teams and engineering teams to quickly see the status of data subject areas. Since Lumadata's catalog is tagged by subject area, Lumadata can easily identify which subject areas are affected by alerts. The dashboard presents alerts and subject area status in a single view showing current day readiness and historic performance.