AI Evaluation Engineer - Planning & Operations

**About Us** Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high\-performing teams by matching them with professionals who truly fit their needs. **Role Overview** We are looking for an **AI Evaluation Engineer specialized in planning and operations** to design and build benchmark tasks that simulate real\-world scenarios such as scheduling, logistics, and resource allocation. This role focuses on **planning, scheduling, and operational optimization problems** , where multiple agents must collaborate to solve constraint\-rich scenarios involving resources, timelines, and dependencies. **Commitments Required: 8 hours per day with an overlap of 4 hours with PST.** **Employment type: Contractor assignment (no medical/paid leave)** **Duration of contract: 4 weeks\+** **Location:** **Bangladesh, Brazil, Colombia, Egypt, Ghana, India, Indonesia, Kenya, Nigeria,Turkey, Vietnam** **Interview: take home assessment (60min) \+ short interview** **Responsibilities** * Design and build multi\-agent benchmark tasks involving: + Planning, scheduling, and resource allocation + Operational decision\-making (logistics, project planning, incident response, capacity planning) * Create constraint\-rich problem statements with multiple interacting variables * Develop verification scripts to evaluate: + Feasibility (all constraints satisfied) + Completeness (all requirements met) + Optimality (efficiency of solutions) * Define task decomposition strategies across specialized sub\-agents (e.g., resource allocation, constraint resolution, optimization) * Model realistic operational systems with dependencies, timelines, and constraints * Implement validation logic and evaluation pipelines using Python * Work with Docker environments for reproducibility and execution * Collaborate with internal teams to improve task quality, coverage, and evaluation rigor **Requirements** * 5\+ years of experience in operations, project management, logistics, or supply chain * Strong ability to formalize constraints, dependencies, and scheduling logic * Proficiency in Python for building validation and verification scripts * Experience with optimization techniques (linear programming, constraint satisfaction, scheduling algorithms) * Strong structured problem\-solving and decomposition skills * Experience with AI benchmarks or evaluation frameworks (e.g., SWE\-bench or similar) * Hands\-on experience with Docker (Dockerfiles, image builds, debugging) ****Nice to Have**** * Background in operations research or optimization\-heavy domains * Experience with simulation or modeling tools * Familiarity with AI planning systems or automated reasoning * Project management experience or certifications (PMP, Agile, etc.)