Package com.netflix.genie.client

Class JobClient

java.lang.Object
com.netflix.genie.client.JobClient
public class JobClient extends Object
Client library for the Job Service.
Since:
3.0.0

  • Constructor Details

    • JobClient

      public JobClient(@NotNull @NotNull retrofit2.Retrofit retrofit, int maxStatusRetries)
      Constructor.
      Parameters:
      retrofit - The configured Retrofit client to a Genie server
      maxStatusRetries - The maximum number of retries to check for job status

    • JobClient

      @Deprecated public JobClient(@NotEmpty @NotEmpty String url, @Nullable List<okhttp3.Interceptor> interceptors, @Nullable GenieNetworkConfiguration genieNetworkConfiguration) throws GenieClientException
      Deprecated.
      Use JobClient(Retrofit, int)
      Constructor.
      Parameters:
      url - The endpoint URL of the Genie API. Not null or empty
      interceptors - Any interceptors to configure the client with, can include security ones
      genieNetworkConfiguration - The network configuration parameters. Could be null
      Throws:
      GenieClientException - On error

  • Method Details

    • submitJob

      public String submitJob(JobRequest jobRequest) throws IOException
      Submit a job to genie using the jobRequest provided.
      Parameters:
      jobRequest - A job request containing all the details for running a job.
      Returns:
      jobId The id of the job submitted.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • submitJob

      public String submitJob(JobRequest jobRequest, String upstreamSecurityTokenName, String upstreamSecurityTokenValue) throws IOException
      Submit a job to genie using the jobRequest and upstream security token information.
      Parameters:
      jobRequest - A job request containing all the details for running a job.
      upstreamSecurityTokenName - the security token name provided by upstream.
      upstreamSecurityTokenValue - the security token value provided by upstream.
      Returns:
      jobId The id of the job submitted.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • submitJobWithAttachments

      public String submitJobWithAttachments(JobRequest jobRequest, Map<String,InputStream> attachments) throws IOException
      Submit a job to genie using the jobRequest and attachments provided.
      Parameters:
      jobRequest - A job request containing all the details for running a job.
      attachments - A map of filenames/input-streams needed to be sent to the server as attachments.
      Returns:
      jobId The id of the job submitted.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • submitJobWithAttachments

      public String submitJobWithAttachments(JobRequest jobRequest, Map<String,InputStream> attachments, String upstreamSecurityTokenName, String upstreamSecurityTokenValue) throws IOException
      Submit a job to genie using the jobRequest and attachments provided with the upstream security information.
      Parameters:
      jobRequest - A job request containing all the details for running a job.
      attachments - A map of filenames/input-streams needed to be sent to the server as attachments.
      upstreamSecurityTokenName - the security token name provided by upstream.
      upstreamSecurityTokenValue - the security token value provided by upstream.
      Returns:
      jobId The id of the job submitted.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • getJobs

      public List<JobSearchResult> getJobs() throws IOException, GenieClientException
      Method to get a list of all the jobs.
      Returns:
      A list of jobs.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • getJobs

      @Deprecated public List<JobSearchResult> getJobs(String id, String name, String user, Set<String> statuses, Set<String> tags, String clusterName, String clusterId, String commandName, String commandId, Long minStarted, Long maxStarted, Long minFinished, Long maxFinished) throws IOException, GenieClientException
      Deprecated.
      Method to get a list of all the jobs from Genie for the query parameters specified.

      Deprecated: For new search fields

      Parameters:
      id - id for job
      name - name of job (can be a SQL-style pattern such as HIVE%)
      user - user who submitted job
      statuses - statuses of jobs to find
      tags - tags for the job
      clusterName - the name of the cluster
      clusterId - the id of the cluster
      commandName - the name of the command run by the job
      commandId - the id of the command run by the job
      minStarted - The time which the job had to start after in order to be return (inclusive)
      maxStarted - The time which the job had to start before in order to be returned (exclusive)
      minFinished - The time which the job had to finish after in order to be return (inclusive)
      maxFinished - The time which the job had to finish before in order to be returned (exclusive)
      Returns:
      A list of jobs.
      Throws:
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.
      See Also:

    • getJobs

      public List<JobSearchResult> getJobs(@Nullable String id, @Nullable String name, @Nullable String user, @Nullable Set<String> statuses, @Nullable Set<String> tags, @Nullable String clusterName, @Nullable String clusterId, @Nullable String commandName, @Nullable String commandId, @Nullable Long minStarted, @Nullable Long maxStarted, @Nullable Long minFinished, @Nullable Long maxFinished, @Nullable String grouping, @Nullable String groupingInstance) throws IOException, GenieClientException
      Deprecated.
      Use getJobs(String, String, String, Set, Set, String, String, String, String, Long, Long, Long, Long, String, String, Integer, SortAttribute, SortDirection, Integer)
      Method to get a list of all the jobs from Genie for the query parameters specified.
      Parameters:
      id - id for job
      name - name of job (can be a SQL-style pattern such as HIVE%)
      user - user who submitted job
      statuses - statuses of jobs to find
      tags - tags for the job
      clusterName - the name of the cluster
      clusterId - the id of the cluster
      commandName - the name of the command run by the job
      commandId - the id of the command run by the job
      minStarted - The time which the job had to start after in order to be return (inclusive)
      maxStarted - The time which the job had to start before in order to be returned (exclusive)
      minFinished - The time which the job had to finish after in order to be return (inclusive)
      maxFinished - The time which the job had to finish before in order to be returned (exclusive)
      grouping - The grouping the job should be a member of
      groupingInstance - The grouping instance the job should be a member of
      Returns:
      A list of jobs.
      Throws:
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.

    • getJobs

      public List<JobSearchResult> getJobs(@Nullable String id, @Nullable String name, @Nullable String user, @Nullable Set<String> statuses, @Nullable Set<String> tags, @Nullable String clusterName, @Nullable String clusterId, @Nullable String commandName, @Nullable String commandId, @Nullable Long minStarted, @Nullable Long maxStarted, @Nullable Long minFinished, @Nullable Long maxFinished, @Nullable String grouping, @Nullable String groupingInstance, @Nullable @Min(1L) @Min(1L) Integer pageSize, @Nullable SortAttribute sortAttribute, @Nullable SortDirection sortDirection, @Nullable @Min(0L) @Min(0L) Integer pageIndex) throws IOException, GenieClientException
      Method to get a list of all the jobs from Genie for the query parameters specified.
      Parameters:
      id - id for job
      name - name of job (can be a SQL-style pattern such as HIVE%)
      user - user who submitted job
      statuses - statuses of jobs to find
      tags - tags for the job
      clusterName - the name of the cluster
      clusterId - the id of the cluster
      commandName - the name of the command run by the job
      commandId - the id of the command run by the job
      minStarted - The time which the job had to start after in order to be return (inclusive)
      maxStarted - The time which the job had to start before in order to be returned (exclusive)
      minFinished - The time which the job had to finish after in order to be return (inclusive)
      maxFinished - The time which the job had to finish before in order to be returned (exclusive)
      grouping - The grouping the job should be a member of
      groupingInstance - The grouping instance the job should be a member of
      pageSize - The maximum number of results returned
      sortAttribute - The entity attribute used to sort
      sortDirection - The sort direction
      pageIndex - The page index
      Returns:
      A list of jobs.
      Throws:
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.

    • getJob

      public Job getJob(String jobId) throws IOException, GenieClientException
      Method to get a job from Genie.
      Parameters:
      jobId - The id of the job to get.
      Returns:
      The job details.
      Throws:
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.

    • getJobCluster

      public Cluster getJobCluster(String jobId) throws IOException, GenieClientException
      Method to get the cluster on which the job executes.
      Parameters:
      jobId - The id of the job.
      Returns:
      The cluster object.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • getJobCommand

      public Command getJobCommand(String jobId) throws IOException, GenieClientException
      Method to get the command on which the job executes.
      Parameters:
      jobId - The id of the job.
      Returns:
      The command object.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • getJobRequest

      public JobRequest getJobRequest(String jobId) throws IOException, GenieClientException
      Method to get the Job Request for the job.
      Parameters:
      jobId - The id of the job.
      Returns:
      The job requests object.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • getJobExecution

      public JobExecution getJobExecution(String jobId) throws IOException, GenieClientException
      Method to get the Job Execution information for the job.
      Parameters:
      jobId - The id of the job.
      Returns:
      The job execution object.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • getJobMetadata

      public JobMetadata getJobMetadata(String jobId) throws IOException, GenieClientException
      Method to get the metadata information for the job.
      Parameters:
      jobId - The id of the job.
      Returns:
      The metadata object.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • getJobApplications

      public List<Application> getJobApplications(String jobId) throws IOException, GenieClientException
      Method to get the Applications for the job.
      Parameters:
      jobId - The id of the job.
      Returns:
      The list of Applications.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • getJobStdout

      public InputStream getJobStdout(String jobId) throws IOException, GenieClientException
      Method to fetch the stdout of a job from Genie.
      Parameters:
      jobId - The id of the job whose output is desired.
      Returns:
      An input stream to the output contents.
      Throws:
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.

    • getJobStdout

      public InputStream getJobStdout(String jobId, @Nullable Long rangeStart, @Nullable Long rangeEnd) throws IOException, GenieClientException
      Method to fetch the stdout of a job from Genie.

      Range Logic:

      rangeStart but no rangeEnd then go from the start byte to the end of available content

      rangeStart and rangeEnd return that range of bytes from the file if they exist

      If only rangeEnd then return the last number of those bytes from the file if they exist

      Parameters:
      jobId - The id of the job whose output is desired.
      rangeStart - The start byte of the file to retrieve. Optional. Greater than or equal to 0.
      rangeEnd - The end byte of the file to retrieve. Optional. Greater than or equal to 0. Must be greater than rangeStart.
      Returns:
      An input stream to the output contents.
      Throws:
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.

    • getJobStderr

      public InputStream getJobStderr(String jobId) throws IOException, GenieClientException
      Method to fetch the stderr of a job from Genie.
      Parameters:
      jobId - The id of the job whose stderr is desired.
      Returns:
      An input stream to the stderr contents.
      Throws:
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.

    • getJobStderr

      public InputStream getJobStderr(String jobId, @Nullable Long rangeStart, @Nullable Long rangeEnd) throws IOException, GenieClientException
      Method to fetch the stderr of a job from Genie.

      Range Logic:

      rangeStart but no rangeEnd then go from the start byte to the end of available content

      rangeStart and rangeEnd return that range of bytes from the file if they exist

      If only rangeEnd then return the last number of those bytes from the file if they exist

      Parameters:
      jobId - The id of the job whose stderr is desired.
      rangeStart - The start byte of the file to retrieve. Optional. Greater than or equal to 0.
      rangeEnd - The end byte of the file to retrieve. Optional. Greater than or equal to 0. Must be greater than rangeStart.
      Returns:
      An input stream to the stderr contents.
      Throws:
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.

    • getJobOutputFile

      public InputStream getJobOutputFile(String jobId, String outputFilePath) throws IOException, GenieClientException
      Method to fetch an output file for a job from Genie.

      NOTE: If the specified outputFilePath is a directory, then the directory manifest is returned.

      Parameters:
      jobId - The id of the job whose output file is desired.
      outputFilePath - The path to the file within output directory.
      Returns:
      An input stream to the output file contents.
      Throws:
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.

    • getJobOutputFile

      public InputStream getJobOutputFile(String jobId, String outputFilePath, @Nullable Long rangeStart, @Nullable Long rangeEnd) throws IOException, GenieClientException
      Method to fetch an output file for a job from Genie and accepting an range of bytes to return.

      NOTE: If the specified outputFilePath is a directory, then the directory manifest is returned.

      Range Logic:

      rangeStart but no rangeEnd then go from the start byte to the end of available content

      rangeStart and rangeEnd return that range of bytes from the file if they exist

      If only rangeEnd then return the last number of those bytes from the file if they exist

      Parameters:
      jobId - The id of the job whose output file is desired.
      outputFilePath - The path to the file within output directory.
      rangeStart - The start byte of the file to retrieve. Optional. Greater than or equal to 0.
      rangeEnd - The end byte of the file to retrieve. Optional. Greater than or equal to 0. Must be greater than rangeStart.
      Returns:
      An input stream to the output file contents.
      Throws:
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.
      See Also:

    • getJobStatus

      public JobStatus getJobStatus(String jobId) throws IOException, GenieClientException
      Method to fetch the status of a job.
      Parameters:
      jobId - The id of the job.
      Returns:
      The status of the Job.
      Throws:
      GenieClientException - If the response recieved is not 2xx.
      IOException - For Network and other IO issues.

    • killJob

      public void killJob(String jobId) throws IOException, GenieClientException
      Method to send a kill job request to Genie.
      Parameters:
      jobId - The id of the job.
      Throws:
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.

    • waitForCompletion

      public JobStatus waitForCompletion(String jobId, long blockTimeout, long pollTime) throws GenieClientException, InterruptedException, IOException, GenieTimeoutException
      Wait for job to complete, until the given timeout.
      Parameters:
      jobId - the Genie job ID to wait for completion
      blockTimeout - the time to block for (in ms), after which a GenieClientException will be thrown
      pollTime - the time to sleep between polling for job status
      Returns:
      The job status for the job after completion
      Throws:
      InterruptedException - on thread errors.
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.
      GenieTimeoutException - If the job times out.

    • waitForCompletion

      public JobStatus waitForCompletion(String jobId, long blockTimeout) throws GenieClientException, InterruptedException, IOException, GenieTimeoutException
      Wait for job to complete, until the given timeout.
      Parameters:
      jobId - the Genie job ID to wait for completion.
      blockTimeout - the time to block for (in ms), after which a GenieClientException will be thrown.
      Returns:
      The job status for the job after completion.
      Throws:
      InterruptedException - on thread errors.
      GenieClientException - If the response received is not 2xx.
      IOException - For Network and other IO issues.
      GenieTimeoutException - If the job times out.