Package com.netflix.genie.web.agent.apis.rpc.v4.endpoints

Class GRpcAgentFileStreamServiceImpl

java.lang.Object
com.netflix.genie.proto.FileStreamServiceGrpc.FileStreamServiceImplBase
com.netflix.genie.web.agent.apis.rpc.v4.endpoints.GRpcAgentFileStreamServiceImpl
All Implemented Interfaces:
AgentFileStreamService, io.grpc.BindableService
public class GRpcAgentFileStreamServiceImpl extends com.netflix.genie.proto.FileStreamServiceGrpc.FileStreamServiceImplBase implements AgentFileStreamService
AgentFileStreamService gRPC implementation. Receives and caches manifests from connected agents. Allows requesting a file, which is returned in the form of a AgentFileStreamService.AgentFileResource.

Implementation overview: Each agent maintains a single "control" bidirectional stream (through the 'sync' RPC method). This stream is used by the agent to regularly push manifests. And it is used by the server to request files.

When a file is requested, the agent opens a separate "transfer" bidirectional stream (through the 'transmit' RPC method) for that file transfer and starts sending chunks (currently one at the time), the server sends acknowledgements in the same stream.

This service returns a resource immediately, but maintains a handle on a buffer where data is written as it is received.

Since:
4.0.0

  • Constructor Details

    • GRpcAgentFileStreamServiceImpl

      public GRpcAgentFileStreamServiceImpl(JobDirectoryManifestProtoConverter converter, org.springframework.scheduling.TaskScheduler taskScheduler, AgentFileStreamProperties properties, io.micrometer.core.instrument.MeterRegistry registry)
      Constructor.
      Parameters:
      converter - The JobDirectoryManifestProtoConverter instance to use
      taskScheduler - A TaskScheduler instance to use
      properties - The service properties
      registry - The meter registry

  • Method Details

    • getResource

      public Optional<AgentFileStreamService.AgentFileResource> getResource(String jobId, Path relativePath, URI uri, @Nullable org.springframework.http.HttpRange range)
      Returns a Resource for the given job file boxed in an Optional. If the service is unable to determine whether the file exists, the optional is empty. In all other cases, the optional is not empty. However the resource may return false to exist() calls (if the file is not believed to exist on the agent) or false to isReadable() if the file cannot be streamed for other reasons.
      Specified by:
      getResource in interface AgentFileStreamService
      Parameters:
      jobId - the job id
      relativePath - the relative path in the job directory
      uri - the file uri //TODO redundant
      range - the list of ranges requested (RFC 7233) or null if no range is specified
      Returns:
      an optional Resource

    • getManifest

      public Optional<DirectoryManifest> getManifest(String jobId)
      Returns the manifest for a given job, boxed in an Optional. The manifest may not be present if the agent is not connected to this node (for example because execution has completed, or because the agent is connected to a different node).
      Specified by:
      getManifest in interface AgentFileStreamService
      Parameters:
      jobId - the job id
      Returns:
      an optional DirectoryManifest

    • sync

      public io.grpc.stub.StreamObserver<com.netflix.genie.proto.AgentManifestMessage> sync(io.grpc.stub.StreamObserver<com.netflix.genie.proto.ServerControlMessage> responseObserver)
      Overrides:
      sync in class com.netflix.genie.proto.FileStreamServiceGrpc.FileStreamServiceImplBase

    • transmit

      public io.grpc.stub.StreamObserver<com.netflix.genie.proto.AgentFileMessage> transmit(io.grpc.stub.StreamObserver<com.netflix.genie.proto.ServerAckMessage> responseObserver)
      Overrides:
      transmit in class com.netflix.genie.proto.FileStreamServiceGrpc.FileStreamServiceImplBase