Custom Software Engineer

Project Role : Custom Software Engineer
Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs.
Must have skills : Grafana
Good to have skills : Apache Kafka
Minimum 7.5 year(s) of experience is required
Educational Qualification : 15 years full time education

Summary:
As a Custom Software Engineer, a typical day involves developing tailored software solutions by designing, coding, and improving various components within systems or applications. The role requires working with modern frameworks and following agile methodologies to ensure the delivery of scalable and efficient software that meets unique business requirements. Collaboration with cross-functional teams and continuous refinement of software elements to enhance performance and functionality are integral parts of the daily workflow.

Roles & Responsibilities:
- Expected to be an SME, collaborate and manage the team to perform.
- Responsible for team decisions.
- Engage with multiple teams and contribute on key decisions.
- Provide solutions to problems for their immediate team and across multiple teams.
- Lead efforts to identify and implement process improvements that enhance team productivity and software quality.
- Mentor junior team members to support their professional growth and integration within the team.
- Coordinate with stakeholders to align software development activities with business goals and project timelines. Design and implement the observability layer for Akura across application, integration, event, agent, audit, and cloud runtime components.
- Define logging, metrics, tracing, alerting, and dashboard standards for platform services.
- Establish end-to-end correlation across Guidewire events, Kafka topics, backend services, agent decisions, ServiceNow actions, and APEX UI views.
- Ensure all services expose health, readiness, latency, throughput, error, and dependency metrics.
- Work with Solution Architect and DevOps Engineer to align observability with cloud deployment and operational readiness needs.
- Configure and support OpenTelemetry instrumentation across backend microservices, Kafka consumers/producers, API services, agent services, and connectors.
- Configure OpenTelemetry Collector pipelines for telemetry ingestion, enrichment, filtering, batching, memory limiting, masking, and routing.
- Implement OTLP receivers and exporters as required by target observability platforms.
- Use OpenTelemetry context propagation to preserve trace IDs and correlation IDs across services.
- Support OpenTelemetry configuration governance and reusable collector patterns.
- Instrument Spring Boot / FastAPI microservices with logs, traces, and metrics.
- Capture API latency, request volume, error rate, dependency failures, and service health.
- Define dashboard views for Guidewire Adapter Service, Incident Evaluation Service, ServiceNow Connector, Agent Orchestration Service, Audit Service, and APEX Backend APIs.
- Ensure application logs include correlation ID, event ID, service name, environment, severity, and processing status.
- Support troubleshooting of service-to-service and API gateway interactions.
- Build dashboards and alerts for Kafka / Amazon MSK event processing.
- Monitor topic throughput, consumer lag, failed events, retry counts, DLQ volume, partition health, and event processing latency.
- Provide visibility into topics such as: guidewire.claim.event.created, incident.evaluation.requested, agent.incident.decision.created, incident.creation.requested, audit.event.recorded
- Support replay and failure investigation by correlating Kafka offsets, event IDs, and audit entries.
- Work with Backend/Kafka Engineers to define observability hooks for producers, consumers, and stream processing applications.
- Implement observability for AI/agent workflows, including agent invocation count, agent latency, decision outcome, confidence distribution, failed tool calls, and human-in-loop routing.
- Ensure agent decision flows are traceable from input event to final action.
- Capture agent execution metadata, decision status, tool-call failures, and audit references.
- Work with Agent Integration Engineer to expose metrics for incident tr…