Re: [gyrex-dev] Gyrex background Job

[Tomas Tamošaitis <siena@xxxxxxxxx>]

Hi Gunnar

Just to clarify: You want to monitor state changes of jobs on any node, i.e. not necessarily the node that is executing the job. Is that correct?

Yes. At least get callback then job is done and returns any state.

 

It sounds like a good idea to have such an API. I'm interested in building one with you. Thoughts?

At the moment it is designed to for scheduled jobs needs. It will be good to have functionality more orientated into multitasking (multithreading).

First  WokerEngine purpose should be different. It should not manage job execution, but wokers (threads)  pools for job execution. Create them dispose and etc.

Each wokers pool should have this own job queue, max concurrent jobs, filter of cluster node where is should be enabled and other settings.

Next about jobs. From scheduled job perceptive it good to store information about job and his execution context and proprieties. But from multitasking perceptive job are dynamic. At the same moment the same job can have different execution context with properties and executed concurrently.

At the moment you can an queue job with properties, but internally properties will be written into job preferences node.

Then declaring new job we need to pass information about like : job id or name; maybe runtime context; lock (but it will be good to have semaphore). Job properties should be passed with job id/name  in execution queue (like it is now) with out storing them in preferences node. 

And the job execution callback. First of all we need some call back then job finished execution and return any result. I think it possible to implement with ZooKeeper Wathes [http://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#ch_zkWatches]

I have some more ideas but I think it will be enough to start discussion for the best functionality.

Also i need to dig some more knowledge about ZooKeeper and this message queues functionality and limitations (and other stuff).

As for the time delay ... the workers are currently polling the queues. There are a lot system settings to change it. If you are not scared, please have a look at WorkerEngine. 

The properties you might be interested in are:
- gyrex.jobs.workerEngine.idleSleepTimeMs (default is 20 seconds)
- gyrex.jobs.workerEngine.maxConcurrentScheduledJobs (default is Runtime.getRuntime().availableProcessors())

Thanks to show me an directions. It will be good to have some documentation with the list of configuration properties.

  

Those need to be set on the worker nodes.

-Gunnar

> Am 17.02.2015 um 13:45 schrieb Tomas Tamošaitis <siena@xxxxxxxxx>:

>
> Hi,
> I'm trying to use gyrex background jobs.
> I have some questions about this.
>
> First of all is there any job state change callback? Current interface provides no functionality for this. I'm thinking to user PreferenceChangeListener on job preferences node, but class CloudPreferncesJobStorage is not visible and i will need to hardcode path to the node.
>
> About job running worker configuration is there any way to configure how many parallel job can be executed on single cluster node?
>
> Also i found that there is an significant delay between registering job in execution queue ant then worker finally execute the job. is there any way to shorter this delay ?
>
> P.S. May be there is some plans to extend background job functionality. In the future i will need more powerful tool for my application requirements. At the moment i'm thinking about akka inegration in to gyrex.
>
> --
> Pagarbiai,
> Tomas Tamošaitis

> _______________________________________________
> gyrex-dev mailing list
> gyrex-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/gyrex-dev


_______________________________________________
gyrex-dev mailing list
gyrex-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/gyrex-dev

--

Pagarbiai,

Tomas Tamošaitis