AI Log Diagnosis for a Large Data Platform

Background

A large data platform team operated thousands of scheduled workflows across multiple compute engines. Engineers spent significant time reading failed task logs and searching historical tickets.

Challenge

Traditional alerting identified failed tasks but did not explain the root cause. Repeated failures were handled manually, and knowledge reuse depended on senior engineers.

Approach

We designed a diagnosis workflow that normalized task logs, classified errors, retrieved similar historical cases, and generated concise explanations with suggested fixes.

Technical design

The assistant combined workflow metadata, log preprocessing, an error taxonomy, retrieval over historical cases, and a human feedback loop for improving recommendations.

Outcome

The team reduced repeated troubleshooting workload, improved knowledge reuse, and established a foundation for future workflow-aware automation.

Lessons learned

The strongest gains came from consistent error classification and evidence-backed suggestions, not from long generative answers.