[Remote] Staff AI Observability & Telemetry Engineer

Note: The job is a remote job and is open to candidates in USA. Bitdeer is a technology company focused on AI and Bitcoin mining infrastructure. The Staff AI Observability & Telemetry Engineer will architect and scale the telemetry and observability infrastructure for Bitdeer's AI-native NeoCloud platform, enabling real-time monitoring, diagnosis, remediation, and billing across GPU and network resources.

Responsibilities

• Architect and scale a high-cardinality telemetry infrastructure using highly available time-series databases (e.g., VictoriaMetrics, Thanos, or Mimir) capable of handling massive ingestion rates
• Integrate complex hardware-level exporters (NVIDIA DCGM, network switch telemetry, IPMI/Redfish) directly into the Kubernetes observability stack to provide a unified view of the cluster
• Build eBPF-based diagnostic tools to trace network congestion, kernel-level I/O latency, and distributed training bottlenecks across the cluster
• Develop automated dashboards and alerting pipelines that trigger proactive cordoning of degraded hardware before it impacts customer training jobs
• Design the metric pipelines required for accurate, multi-tenant consumption billing based on real-time GPU and network utilization metrics
• Collaborate with the GPU Systems and Scheduling teams to create observability standards for "AI-native" workloads, ensuring deep insight into job efficiency and resource utilization
• Lead technical design reviews for observability architecture, mentoring team members on best practices for high-performance telemetry collection and analysis

Skills

• Bachelor's or Master's degree in Computer Science, Electrical Engineering, or a related field
• 6+ years of software or site reliability engineering, with deep, hands-on expertise in the Prometheus/OpenTelemetry ecosystem
• Advanced proficiency in Go and extensive experience writing custom Kubernetes metric exporters and operators
• Hands-on experience with kernel-level tracing tools (eBPF, BCC) and deep performance tuning of Linux systems
• Strong familiarity with AI hardware metrics (GPU power states, SM utilization, memory bandwidth) and high-performance network telemetry
• Proven track record of operating, debugging, and scaling large-scale telemetry stacks in high-performance computing or cloud environments
• Strong technical leadership skills; ability to influence architectural decisions and align cross-functional teams around observability standards
• Excellent communication skills, with the ability to translate complex system requirements into manageable engineering milestones
• Experience working in high-velocity, high-growth engineering environments is strongly preferred

Benefits

• Full-Time employment
• Remote (within locations)

Company Overview