[Remote] Principal AI Software Engineer

Note: The job is a remote job and is open to candidates in USA. Walker & Dunlop is a provider of capital to the commercial real estate industry, with an in-house technology team focused on real estate data, data science, and technology. The Principal AI Software Engineer will lead the design, development, deployment, and operation of scalable AI applications, agentic workflows, and platform capabilities supporting loan origination, servicing, and business operations.

Responsibilities

• Own complex AI and platform initiatives end-to-end, delivering business-facing capabilities while strengthening the underlying context and agent platform
• Design, build and maintain agentic workflows for business-critical processes such as loan intake, document review and servicing
• Help establish new platform patterns and capabilities by defining approaches, proving them through production use cases and scaling them for broader adoption
• Lead model selection and routing, evaluating models based on cost, latency and quality, and evolving solutions as new models and capabilities become available
• Design interfaces and capabilities used by other teams, including MCP tool servers, agent handoffs, subagents, long-running state and human-in-the-loop checkpoints
• Establish evaluation and reliability practices, including evaluation sets, LLM-as-judge approaches where appropriate, regression testing and ongoing performance measurement
• Operate with minimal oversight, proactively identifying risks, making sound technical trade-offs and escalating when needed, while raising the engineering bar through collaboration and mentorship
• Perform other duties as assigned

Skills

• 12+ years of software development experience, including experience as a senior individual contributor delivering production systems
• Bachelor's or master's degree in Computer Science or a related field, or equivalent experience
• Extensive experience delivering production AI/ML systems, including LLM-powered applications and agentic workflows
• Deep experience with modern agent frameworks and SDKs, such as Claude Agent SDK, LangGraph and Pydantic AI, as well as MCP and A2A
• Experience with prompt and context engineering, including RAG, embeddings, hybrid search and reranking
• Experience with model evaluation, observability and guardrails for production AI systems
• Experience using agentic development tools, including skills, subagents and harness configurations
• Deep proficiency in Python and modern backend frameworks such as FastAPI, with a strong focus on scalable and maintainable architecture
• Strong expertise with SQL databases, including schema design, query optimization and performance tuning
• Experience testing non-deterministic systems, including LLM call fixtures, evaluation suites and strategies for controlling flaky behavior
• Strong judgment in determining which capabilities belong within a shared platform versus individual applications
• Ability to operate effectively in ambiguity, establish emerging patterns and independently move from business problem definition through solution delivery
• Experience with Event Sourcing and Domain-Driven Design principles
• Excellent verbal and written communication skills
• Ability to show ownership of your work, take on challenges and acknowledge growth opportunities, and demonstrate patience when learning new processes
• Courtesy, respect, and thoughtfulness in teaming with colleagues and other stakeholders
• Experience in Real Estate, Financial Services or Capital Markets preferred

Benefits

• Discretionary bonus
• The opportunity to join one of Fortune Magazine’s Great Places to Work winners
• Up to 83% subsidized medical payroll deductions
• Competitive dental and vision benefits
• 401(k) + match
• Pre-tax transit and commuting benefits
• A robust health and wellness program – earn cash rewards and gain access to resources that promote health, engagement, and balance
• Paid maternity and parental leave, as well as other family paid leave programs
• Company-paid life, short and long-term disability insurance
• Health Savings Account and Healthcare and Dependent Care Flexible Spending
• Career development opportunities
• Empowerment and encouragement to give back – volunteer hours and donation matching

Company Overview

Company H1B Sponsorship