[Remote] Senior Backend Engineer

Note: The job is a remote job and is open to candidates in USA. Instrumentl is a SaaS platform helping grant-funded organizations discover, track, and win grant funding. The company is seeking a Senior Backend Engineer to own AI features end to end, building APIs, tool-using agents, and RAG pipelines for grant discovery, application drafting, and research. The role also involves production reliability, evaluation and observability, backend development, experimentation, and cross-functional collaboration.

Responsibilities

• Build tool-using LLM agents (task planning, function and tool calling, multi-step workflows, guardrails) for grant discovery, application drafting, and research assistance
• Turn prototypes into resilient, observable services with clear SLAs, rollback and fallback strategies, and cost and latency budgets
• Stand up evaluation and observability so our AI stays grounded, safe, and cost-effective
• Write high-quality, thoroughly tested code across the backend and the data pipelines that power retrieval and evaluation
• Contribute to reliability practices: alerts, dashboards, and incident response
• Partner with Product, Design, and GTM on scoping, UX, and measurement
• Run experiments (A/B, canaries), interpret results, and iterate
• Raise engineering standards through clear, maintainable code, tests, docs, and thoughtful review

Skills

• 7+ years building and shipping production backend systems in Python (FastAPI, Celery, or equivalent), taking features from prototype to production with real reliability practices like tests, observability, and rollback
• Hands-on experience building LLM features in production: tool and function calling, multi-step agent workflows, and the guardrails and evals that keep them grounded, safe, and cost-effective
• Strong data fundamentals: SQL, schema design, and building pipelines that power retrieval and evaluation
• Thrives in a fast, scrappy startup environment with high ownership and a bias for action, speed, quality, and simplicity
• TypeScript and Node, plus familiarity with Ruby on Rails (our core platform) or a willingness to learn it
• Experience with AWS or GCP, Docker, CI/CD, and observability (logs, metrics, traces)
• RAG depth: document ingestion, chunking and windowing, embeddings, hybrid search (keyword plus vector), re-ranking, and grounded citations
• Experience with re-rankers and cross-encoders, hybrid retrieval tuning, or search and recommendation systems
• Evaluation mindset: designing eval suites (RAG/QA, extraction, summarization) using automated and human-in-the-loop methods, with familiarity with frameworks like Ragas, DeepEval, or OpenAI Evals
• Orchestration frameworks: LangChain or LangGraph, LlamaIndex, Semantic Kernel, or custom orchestration

Benefits

• Equity
• 100% covered health, dental, and vision insurance for employees (50% for dependents)
• Generous PTO, including parental leave
• 401(k)
• Company laptop and home-office stipend
• Bi-annual company retreats
• You’ll always have new challenges and opportunities to grow here.

Company Overview

Company H1B Sponsorship