com.netflix.genie.web.data.services.jpa

Class JpaJobPersistenceServiceImpl

java.lang.Object

com.netflix.genie.web.data.services.jpa.JpaJobPersistenceServiceImpl

All Implemented Interfaces:

JobPersistenceService

@Transactional(rollbackFor={GenieException.class,GenieRuntimeException.class,javax.validation.ConstraintViolationException.class,IdAlreadyExistsException.class,SaveAttachmentException.class})
public class JpaJobPersistenceServiceImpl
extends java.lang.Object
implements JobPersistenceService

JPA implementation of the job persistence service.

Constructor Summary

Constructors

Constructor and Description
JpaJobPersistenceServiceImpl(JpaTagPersistenceService tagPersistenceService, JpaFilePersistenceService filePersistenceService, JpaApplicationRepository applicationRepository, JpaClusterRepository clusterRepository, JpaCommandRepository commandRepository, JpaJobRepository jobRepository, AttachmentService attachmentService) Constructor.

Method Summary

Modifier and Type	Method and Description
void	claimJob(@NotBlank(message="Job id is missing and is required") java.lang.String id, @Valid AgentClientMetadata agentClientMetadata) Set a job identified by id to be owned by the agent identified by agentClientMetadata.
void	createJob(JobRequest jobRequest, JobMetadata jobMetadata, @NotNull Job job, @NotNull JobExecution jobExecution) Save all the initial job fields in the data store.
long	deleteBatchOfJobsCreatedBeforeDate(@NotNull java.time.Instant date, @Min(value=1L) int maxDeleted, @Min(value=1L) int pageSize) This method will delete a chunk of jobs whose creation time is earlier than the given date.
FinishedJob	getFinishedJob(@NotBlank(message="Job id is missing and is required") java.lang.String id) Get a DTO representing a finished job.
java.util.Optional<java.lang.String>	getJobArchiveLocation(@NotBlank(message="Job id is missing and is required") java.lang.String id) Get the location a job directory was archived to if at all.
java.util.Optional<JobRequest>	getJobRequest(@NotBlank(message="Id is missing and is required") java.lang.String id) Get the original request for a job.
java.util.Optional<JobSpecification>	getJobSpecification(@NotBlank(message="Id is missing and is required") java.lang.String id) Get the saved job specification for the given job.
JobStatus	getJobStatus(@NotBlank(message="Job id is missing and is required") java.lang.String id) Get the status for a job with the given id.
boolean	isApiJob(@NotBlank(message="Job id is missing and is required") java.lang.String id) Get whether the job with the given ID was submitted via the REST API or other mechanism.
boolean	isV4(@NotBlank(message="Id is missing and is required") java.lang.String id) Get whether the job is a v4 job.
java.lang.String	saveJobSubmission(@Valid JobSubmission jobSubmission) Save the given job submission information in the underlying data store.
void	saveResolvedJob(@NotBlank(message="Id is missing and is required") java.lang.String id, @Valid ResolvedJob resolvedJob) Save the given resolved details for a job.
void	setJobCompletionInformation(@NotBlank(message="No job id entered. Unable to update.") java.lang.String id, int exitCode, @NotNull(message="No job status entered. Unable to update") JobStatus status, @NotBlank(message="Status message can\'t be blank. Unable to update") java.lang.String statusMessage, java.lang.Long stdOutSize, java.lang.Long stdErrSize) Method to set all job completion information for a job execution.
void	setJobRunningInformation(@NotBlank java.lang.String id, @Min(value=0L,message="Must be no lower than zero") int processId, @Min(value=1L,message="Must be at least 1 millisecond, preferably much more") long checkDelay, @NotNull java.time.Instant timeout) Update the job with information for the running job process.
void	updateJobStatus(@NotBlank(message="Id is missing and is required") java.lang.String id, @NotNull JobStatus currentStatus, @NotNull JobStatus newStatus, java.lang.String newStatusMessage) Update the status of the job identified with id to be newStatus provided that the current status of the job matches newStatus.
void	updateJobStatus(@NotBlank(message="No job id entered. Unable to update.") java.lang.String id, @NotNull(message="Status cannot be null.") JobStatus jobStatus, @NotBlank(message="Status message cannot be empty.") java.lang.String statusMsg) Update the status and status message of the job.
void	updateJobWithRuntimeEnvironment(@NotBlank java.lang.String jobId, @NotBlank java.lang.String clusterId, @NotBlank java.lang.String commandId, @NotNull java.util.List<java.lang.String> applicationIds, @Min(value=1L) int memory) Update the job with the various resources used to run the job including the cluster, command and applications.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

JpaJobPersistenceServiceImpl

public JpaJobPersistenceServiceImpl(JpaTagPersistenceService tagPersistenceService,
                                    JpaFilePersistenceService filePersistenceService,
                                    JpaApplicationRepository applicationRepository,
                                    JpaClusterRepository clusterRepository,
                                    JpaCommandRepository commandRepository,
                                    JpaJobRepository jobRepository,
                                    AttachmentService attachmentService)

Constructor.

Parameters:

tagPersistenceService - The JpaTagPersistenceService to use

filePersistenceService - The JpaFilePersistenceService to use

applicationRepository - The JpaApplicationRepository to use

clusterRepository - The JpaClusterRepository to use

commandRepository - The JpaCommandRepository to use

jobRepository - The JpaJobRepository to use

attachmentService - The AttachmentService implementation to use to persist attachments before converting them to dependencies of the job

Method Detail

createJob

public void createJob(@NotNull
                      JobRequest jobRequest,
                      @NotNull
                      JobMetadata jobMetadata,
                      @NotNull
                      @NotNull Job job,
                      @NotNull
                      @NotNull JobExecution jobExecution)
               throws GenieException

Save all the initial job fields in the data store.

Specified by:

createJob in interface JobPersistenceService

Parameters:

jobRequest - the Job request object to save. Not null

jobMetadata - metadata about the job request. Not null

job - The Job object to create

jobExecution - The job execution object to create

Throws:

GenieException - if there is an error

updateJobWithRuntimeEnvironment

public void updateJobWithRuntimeEnvironment(@NotBlank
                                            @NotBlank java.lang.String jobId,
                                            @NotBlank
                                            @NotBlank java.lang.String clusterId,
                                            @NotBlank
                                            @NotBlank java.lang.String commandId,
                                            @NotNull
                                            @NotNull java.util.List<java.lang.String> applicationIds,
                                            @Min(value=1L)
                                            @Min(value=1L) int memory)
                                     throws GenieException

Update the job with the various resources used to run the job including the cluster, command and applications.

Specified by:

updateJobWithRuntimeEnvironment in interface JobPersistenceService

Parameters:

jobId - The id of the job to update

clusterId - The id of the cluster the job runs on

commandId - The id of the command the job runs with

applicationIds - The ids of the applications used to run the job

memory - The amount of memory (in MB) to run the job with

Throws:

GenieException - For any problems while updating

updateJobStatus

public void updateJobStatus(@NotBlank(message="No job id entered. Unable to update.")
                            @NotBlank(message="No job id entered. Unable to update.") java.lang.String id,
                            @NotNull(message="Status cannot be null.")
                            @NotNull(message="Status cannot be null.") JobStatus jobStatus,
                            @NotBlank(message="Status message cannot be empty.")
                            @NotBlank(message="Status message cannot be empty.") java.lang.String statusMsg)
                     throws GenieException

Update the status and status message of the job.

Specified by:

updateJobStatus in interface JobPersistenceService

Parameters:

id - The id of the job to update the status for.

jobStatus - The updated status of the job.

statusMsg - The updated status message of the job.

Throws:

GenieException - if there is an error

setJobRunningInformation

public void setJobRunningInformation(@NotBlank
                                     @NotBlank java.lang.String id,
                                     @Min(value=0L,message="Must be no lower than zero")
                                     @Min(value=0L,message="Must be no lower than zero") int processId,
                                     @Min(value=1L,message="Must be at least 1 millisecond, preferably much more")
                                     @Min(value=1L,message="Must be at least 1 millisecond, preferably much more") long checkDelay,
                                     @NotNull
                                     @NotNull java.time.Instant timeout)
                              throws GenieException

Update the job with information for the running job process.

Specified by:

setJobRunningInformation in interface JobPersistenceService

Parameters:

id - the id of the job to update the process id for

processId - The id of the process on the box for this job

checkDelay - The delay to check the process with

timeout - The date at which this job should timeout

Throws:

GenieException - if there is an error

setJobCompletionInformation

public void setJobCompletionInformation(@NotBlank(message="No job id entered. Unable to update.")
                                        @NotBlank(message="No job id entered. Unable to update.") java.lang.String id,
                                        int exitCode,
                                        @NotNull(message="No job status entered. Unable to update")
                                        @NotNull(message="No job status entered. Unable to update") JobStatus status,
                                        @NotBlank(message="Status message can\'t be blank. Unable to update")
                                        @NotBlank(message="Status message can\'t be blank. Unable to update") java.lang.String statusMessage,
                                        @Nullable
                                        java.lang.Long stdOutSize,
                                        @Nullable
                                        java.lang.Long stdErrSize)
                                 throws GenieException

Method to set all job completion information for a job execution.

Specified by:

setJobCompletionInformation in interface JobPersistenceService

Parameters:

id - the id of the job to update the exit code

exitCode - The exit code of the process

status - The job status for the job

statusMessage - The job status message

stdOutSize - The size (in bytes) of the standard out file or null if there isn't one

stdErrSize - The size (in bytes) of the standard error file or null if there isn't one

Throws:

GenieException - if there is an error

deleteBatchOfJobsCreatedBeforeDate

public long deleteBatchOfJobsCreatedBeforeDate(@NotNull
                                               @NotNull java.time.Instant date,
                                               @Min(value=1L)
                                               @Min(value=1L) int maxDeleted,
                                               @Min(value=1L)
                                               @Min(value=1L) int pageSize)

This method will delete a chunk of jobs whose creation time is earlier than the given date.

Specified by:

deleteBatchOfJobsCreatedBeforeDate in interface JobPersistenceService

Parameters:

date - The date before which all jobs should be deleted

maxDeleted - The maximum number of jobs that should be deleted (soft limit, can be rounded up to multiple of page size)

pageSize - Page size used to iterate through jobs

Returns:

the number of deleted jobs

saveJobSubmission

@Nonnull
public java.lang.String saveJobSubmission(@Valid
                                                   @Valid JobSubmission jobSubmission)
                                            throws IdAlreadyExistsException,
                                                   SaveAttachmentException

Save the given job submission information in the underlying data store.

The job record will be created with initial state of JobStatus.RESERVED meaning that the unique id returned by this API has been reserved within Genie and no other job can use it. If JobSubmission contains some attachments these attachments will be persisted to a configured storage system (i.e. local disk, S3, etc) and added to the set of dependencies for the job. The underlying attachment storage system must be accessible by the agent process configured by the system. For example if the server is set up to write attachments to local disk but the agent is not running locally but instead on the remote system it will not be able to access those attachments (as dependencies) and fail. See AttachmentService for more information.

Specified by:

saveJobSubmission in interface JobPersistenceService

Parameters:

jobSubmission - All the information the system has gathered regarding the job submission from the user either via the API or via the agent CLI

Returns:

The unique id of the job within the Genie ecosystem

Throws:

IdAlreadyExistsException - If the id the user requested already exists in the system for another job

SaveAttachmentException - If attachments were sent in with the job submission and they were unable to be persisted to an underlying storage mechanism meant to share the data with an agent process

getJobRequest

@Transactional(readOnly=true)
public java.util.Optional<JobRequest> getJobRequest(@NotBlank(message="Id is missing and is required")
                                                                                  @NotBlank(message="Id is missing and is required") java.lang.String id)

Get the original request for a job.

Specified by:

getJobRequest in interface JobPersistenceService

Parameters:

id - The unique id of the job to get

Returns:

The job request if one was found. Wrapped in Optional so empty optional returned if no job found

saveResolvedJob

public void saveResolvedJob(@NotBlank(message="Id is missing and is required")
                            @NotBlank(message="Id is missing and is required") java.lang.String id,
                            @Valid
                            @Valid ResolvedJob resolvedJob)

Save the given resolved details for a job. Sets the job status to JobStatus.RESOLVED.

Specified by:

saveResolvedJob in interface JobPersistenceService

Parameters:

id - The id of the job

resolvedJob - The resolved information for the job

getJobSpecification

@Transactional(readOnly=true)
public java.util.Optional<JobSpecification> getJobSpecification(@NotBlank(message="Id is missing and is required")
                                                                                              @NotBlank(message="Id is missing and is required") java.lang.String id)

Get the saved job specification for the given job. If the job hasn't had a job specification resolved an empty Optional will be returned.

Specified by:

getJobSpecification in interface JobPersistenceService

Parameters:

id - The id of the job

Returns:

The job specification if one is present else an empty Optional

claimJob

public void claimJob(@NotBlank(message="Job id is missing and is required")
                     @NotBlank(message="Job id is missing and is required") java.lang.String id,
                     @Valid
                     @Valid AgentClientMetadata agentClientMetadata)

Set a job identified by id to be owned by the agent identified by agentClientMetadata. The job status in the system will be set to JobStatus.CLAIMED

Specified by:

claimJob in interface JobPersistenceService

Parameters:

id - The id of the job to claim. Must exist in the system.

agentClientMetadata - The metadata about the client claiming the job

updateJobStatus

public void updateJobStatus(@NotBlank(message="Id is missing and is required")
                            @NotBlank(message="Id is missing and is required") java.lang.String id,
                            @NotNull
                            @NotNull JobStatus currentStatus,
                            @NotNull
                            @NotNull JobStatus newStatus,
                            @Nullable
                            java.lang.String newStatusMessage)

Update the status of the job identified with id to be newStatus provided that the current status of the job matches newStatus. Optionally a status message can be provided to provide more details to users. If the newStatus is JobStatus.RUNNING the start time will be set. If the newStatus is a member of JobStatus.getFinishedStatuses() and the job had a started time set the finished time of the job will be set.

Specified by:

updateJobStatus in interface JobPersistenceService

Parameters:

id - The id of the job to update status for. Must exist in the system.

currentStatus - The status the caller to this API thinks the job currently has

newStatus - The new status the caller would like to update the status to

newStatusMessage - An optional status message to associate with this change

isV4

@Transactional(readOnly=true)
public boolean isV4(@NotBlank(message="Id is missing and is required")
                                                  @NotBlank(message="Id is missing and is required") java.lang.String id)

Get whether the job is a v4 job.

Specified by:

isV4 in interface JobPersistenceService

Parameters:

id - The id of the job

Returns:

true if its a v4 job

getJobStatus

@Transactional(readOnly=true)
public JobStatus getJobStatus(@NotBlank(message="Job id is missing and is required")
                                                            @NotBlank(message="Job id is missing and is required") java.lang.String id)
                                                     throws GenieNotFoundException

Get the status for a job with the given id.

Specified by:

getJobStatus in interface JobPersistenceService

Parameters:

id - The id of the job to get status for

Returns:

The job status

Throws:

GenieNotFoundException - If no job with the given id exists

getJobArchiveLocation

@Transactional(readOnly=true)
public java.util.Optional<java.lang.String> getJobArchiveLocation(@NotBlank(message="Job id is missing and is required")
                                                                                                @NotBlank(message="Job id is missing and is required") java.lang.String id)
                                                                                         throws GenieNotFoundException

Get the location a job directory was archived to if at all.

Specified by:

getJobArchiveLocation in interface JobPersistenceService

Parameters:

id - The id of the job to get the location for

Returns:

The archive location or Optional.empty()

Throws:

GenieNotFoundException - When there is no job with id id

getFinishedJob

@Transactional(readOnly=true)
public FinishedJob getFinishedJob(@NotBlank(message="Job id is missing and is required")
                                                                @NotBlank(message="Job id is missing and is required") java.lang.String id)
                                                         throws GenieNotFoundException,
                                                                GenieInvalidStatusException

Get a DTO representing a finished job.

Specified by:

getFinishedJob in interface JobPersistenceService

Parameters:

id - The id of the job to retrieve

Returns:

the finished job

Throws:

GenieNotFoundException - if no job with the given id exists

GenieInvalidStatusException - if the current status of the job is not final

isApiJob

@Transactional(readOnly=true)
public boolean isApiJob(@NotBlank(message="Job id is missing and is required")
                                                      @NotBlank(message="Job id is missing and is required") java.lang.String id)
                                               throws GenieNotFoundException

Get whether the job with the given ID was submitted via the REST API or other mechanism.

Specified by:

isApiJob in interface JobPersistenceService

Parameters:

id - The id of the job. Not blank.

Returns:

true if the job was submitted via the API. false otherwise

Throws:

GenieNotFoundException - If no job with id exists