Bugzilla – Bug 4545
WS GRAM JSDL Support in GT 4.2
Last modified: 2008-05-06 16:33:12
You need to log in before you can comment on or make changes to this bug.
Title: WS-GRAM JSDL Support in GT 4.2 Communities: - UK NGS - NAREGI - (any others?) Technologies: Globus Resource Allocation Manager (GRAM) Definition: JSDL is an accepted GGF standard specification for describing grid jobs. Since JSDL became a standard, many groups have started to use it. To accommodate this emerging standard, we propose to extend our factory web service interface with an operation that takes a JSDL document, while leaving the original createManagedJob operation the same. This will allow users to better plan when they take on the transition from the current WS GRAM job description schema to JSDL. Then general plan is to add new operations and resource properties required to support JSDL to the current WS GRAM service. Resulting in a single WS GRAM service that is capable of processing a job specified using the current JDD schema as well as a job specified using the JSDL schema. Clients written before these JSDL enhancements will be able to interoperate with the new WS GRAM service that supports both the current job description schema and JSDL. The gram clients: globusrun-ws and gramJob java API will be updated to identify the job as either JDD or JSDL and use the matching WS GRAM service operations. The globusrun-ws job wrapper scripts (globus-job-*-ws) will not be altered and will continue to specify job's in JDD format. Deliverables: 1) Auto-generated classes for the JSDL and JSDL Posix extension schemas. 2) A document describing exactly how we propose to map JSDL to JobDescriptionType and MultiJobDescriptionType. 3) A new createManagedJobFromJSDL operation in the ManagedJobFactoryPortType WSDL that takes a JSDL document instead of a JobDescriptionType or MultiJobDescriptionType document, converts the JSDL data structure to the associated data structures for the latter two types, and calls the original createManagedJob operation. 4) Updated ManagedJobPortType WSDL that includes a "JSDL" resource property which contains the JSDL document. 5) A Java class named JsdlGramJob that extends GramJob but also takes JSDL documents. 6) An updated globusrun-ws client that accepts JSDL in addition to our proprietary job description format. 7) Additional unit tests for the new operation. 8) Additional scheduler tests that use JSDL instead of our prioprietary job description format. 9) Documentation describing the new operation, the new resource property, the mappings from #2, and client examples. Tasks: 1) Create a document that describes all mappings and extensions needed to convert a JSDL document into a JobDescriptionType or MultiJobDescriptionType document. 2) Import the JSDL and JSDL Posix extension schemas into the 4.2 schema CVS module. 3) Add a "createManagedJobFromJSDL" operation in the 4.2 factory schema. 4) Add a "JSDL" resource property to the 4.2 job schema. 5) Update Ant bulid files to generate Java classes for the new factory, job, and JSDL schemas. 6) Update the WS-GRAM C bindings package to generate C code for the new factory, job, and JSDL schemas. 7) Create a new JdslGramJob class that inherits from GramJob with additional methods and functionality to handle JSDL documents. 8) Update globusrun-ws to detect which job description format is being used, modify the document as needed, and call the appropriate job creation operation. 9) Add unit tests that stress the createManagedJobFromJSDL operation. 10) Add a unit test for the new "JSDL" resource property. 11) Add duplicate scheduler tests that use equivalent JSDL documents. 12) Add factory public interface documentation describing the createManagedJobFromJSDL operation. 13) Add job public interface documentation describing the "JSDL" resource property. 14) Add public interface documentation describing the mappings form JSDL to our prioprietary job description format, including any extensions to JDSL that were necessary.
Tasks 2, 3, and 5 are essentially done. I'm still debugging stub generation errors, though. I've added JSDL extension schemas for GRAM and RFT, but have yet to create the document required for task 1.
Actually, #3 is not done. I added the method skeleton but have not implemented it yet.
Created an attachment (id=994) [details] GRAM JDD to JSDL mappings Deliverable #2
I'm changing task/deliverable #4 so that that resource property is simply a reference to jsdl:JobDefinition instead of gram-job:JSDL (of type jsdl:JobDefinition_Type). JSDL afterall is technically the schema rather than a document that conforms to the schema. It also makes it more clear what you're getting back. Along these lines, I'm also changing the name of the new operation (deliverable #3) to CreateManagedJobFromJSDLJobDefinition.
Created an attachment (id=997) [details] GRAM JDD to JSDL mappings Update element mapping document that makes things a little clearer and includes changes based on development experience.
After much C WS Core debugging help from Joe, task #6 has been completed. I'm almost done with the JsdlGramJob class (task #7/deliverable #5) so that I can at least test the service to some extent.
Task #7 (deliverable #5) is done. I've been able to submit the following simple JSDL job using the JsdlGramJob class: <?xml version="1.0" encoding="UTF-8"?> <jsdl:JobDefinition xmlns:jsdl="http://schemas.ggf.org/jsdl/2005/11/jsdl"> <jsdl:JobDescription> <jsdl:Application> <jsdl-posix:POSIXApplication xmlns:jsdl-posix="http://schemas.ggf.org/jsdl/2005/11/jsdl-posix"> <jsdl-posix:Executable>/bin/date</jsdl-posix:Executable> </jsdl-posix:POSIXApplication> </jsdl:Application> </jsdl:JobDescription> </jsdl:JobDefinition>
Created an attachment (id=1017) [details] GRAM JDD to JSDL mappings
I updated globusrun-ws, but I'm having problems debugging seg faults. The JSDL extension schemas might change again, so I may update for those changes first before either getting help from Joe or handing it off to him altogether.
A status update is long overdue... I was on vacation from the end of July through the first two weeks of August. I did some work on rearranging the JSDL extensions and some other implementation details after some feedback for most of the rest of August. I also had a Chicago trip that prevented me from doing anything for almost a week. I was again off for the first week in September. I was at GlobusWorld/OGF/GridWorld for the second week in September. The third week of September I was occupied with preparations for another meeting in the fourth week of September. This means that I got nothing done wrt this campaign in September. I've been playing catch up and doing some GRAM planning for the last few days. There are some serious issues that are blocking MPICH-G development that I need to attend to. Once I'm done with that I will hopefully be able to once again do some work on this campaign.
Created an attachment (id=1128) [details] XSLT jsdl to xrsl transformation, only core jsdl elements without any extensions We are preparing a plugin for gt4 for our stuff at PSNC in Poland. We have web application which generates JSDL job descriptions so we need such thing. I managed to write some xslt transformation from JSDL v1.0 to GT4 job description xml. I used your mappings. I decided to translate only core elements without adding any gt4 extension for jsdl but I think it could be helpful for you anyway.
Now that SC is out of the way I can finish this stuff up. I had to help out with getting some patches to VDT too. Not to mention some bug work. Anyway, I abused the gram_jsdl_branch to work on the BES demo since that's where all the JSDL work was and I had little time to mess with branching. That means I need to clean some stuff up before merging to HEAD. I should be able to finish that on Monday. To avoid dealing with backwards compatibility issues, I changed the service to accept the JSDL document in the extensions section instead of a replacement create operation which is really a hack to get around the inability to have an operation with the same local name but different namespace. All JDD elements are optional, so the clients can simply wrap the JSDL document in an extensions element inside of a jobDescription element and reuse the same submission code.
Things are working pretty good wrapped in the <extensions> element. I'm almost done adding equivalent scheduler tests for JSDL. I'm considering disabling automatic delegation in globusrun-ws for JSDL documents for lack of time. It does work with manual delegation, though.
I finally fixed things for the local ouput, non-multi scheduler tests that I added . The things left to address are automatic delegation with globusrun-ws, multijob functionality, and documentation.
Created an attachment (id=1145) [details] JSDL Delegation Pseudoschema I'm attaching a pseudoschema that illustrates with incomplete examples the differences between how the GRAM JDL and JSDL (via extensions) supports delegated credential references.
We've decided to go ahead and create an extra service deployment with a WSDL that overrides the current createManagedJob operation. I have the service work done. I've been working on a new client API to abstract out all the different versions of the service that we're supporting (currently 4.0, 4.2/JSDL, and BES) since GramJob is an utter nightmare now. I've finished coding on the client API but still need to make it compile and test everything. I'm told Joe Bester will update globusrun-ws once I'm done with the Java work. That means that multijob support and documentation will be outstanding. Documentation is more important at this point, but if I have time in January after writing the documentation I will try to get that working.
I'm closing this campaign. It can reopen or rewrite a new campaign when we are ready to take on jsdl.
