Pages

Sunday, June 22, 2014

XQCodeGen- An automated XQuery code generator


Why automated code generation ?           

- Code generation is often little more than copy and paste programming.

- An automatic code generation tool always helps developers in all programming languages to get the standard template before writing actual code.  

- This standard template will have basic code standards already defined in the generated files.

- It helps developers to follow the already given standards in their future development.
               
*****************************************************************************                                                     

1 Overview


Problem statement:

For XQuery programming language we do not have such kind of tool that produces the standard template for development.

Solution:
For automated code generation I have designed “XQCodeGen” tool.

Features:

  • Reduce the time spent in writing the code which is repetitive. For example: If developer is working in MVC then he has to design  model and controller code every-time a new feature is added. 
  • XQCodeGen is very easy to use, it can be easily configured on developer machine.
  • XQCodeGen can be easily integrated ant script.
  • XQCodeGen takes XML as an input to generate the template so, developers can easily configure the tool as per their need.
  • If developer will not provide the custom configuration, XQCodeGen will generate the code using its built-in rule-sets.
  • Using XQCodeGen developers can generate MVC (Model, View & Controller) code template for XQuery.
  • Generated MVC can be directly deployed into MarkLogic app server without any modification in url-rewriter and front controller.
  • Using XQCodeGen developers can generate XQuery library modules and main modules.
  • Developers can edit the generated template and add their business logics to the generated templates and can deploy to MarkLogic app server.
 \
2- Using this tool:

1- Modify the config xml (codegenmapping.xml) as per your requirement. Note, do not change the element names.

2- If you want to generate the code for a generic MVC then make the "enabled" attribute of MVC element to true.

3- If you want to get url-rewrter with main controller the make the "enabled" attribute of optional element to true.

4- If you want to generate the main module and library module then make the "enabled" attribute of main-module and lib-module element to true.

5- If you want to disable any of the code generation logic for MVC/library module/main module, you can do it by just putting the value of 'enabled' attribute to 'false'.

Console based UI:


















Config XML:


GUI:

1- Open “XQCodeGen.exe”


2- Select the target output directory and configuration XML file.



3- Click on “Generate Code”, to generate the code



4- Open the target directory to view the generated code.




3Important instructions about MVC.

·         All controller xquery files should go inside the controller directory as mapped in mapping xml file.
·        All model xquery files should go inside the model directory as mapped in mapping xml file.
·         All view xquery files should go inside the view directory; you can also keep views inside a separate        directory for specific components (or with name of controller).
 In this case you need to modify the view-resolver code little bit to append directory name in the uri.
            Line of code needs to be modified in the above case is:
            let $view-uri := fn:concat("/view/",$controller,"/",$controller,"-",$view,".xqy")
            This will create an uri as given below.
            For example:
if controller is 'test-controller' then views can be kept inside /view/test/test-default.xqy


Naming convention for namespaces in model, view and controller XQuery files:-

Follow the naming conventions as given below while mapping the namespaces in mapping config xml file, which will be passed to the tool to generate the code.

1.      In model XQuery file namespace should have 'model' followed by the model name and file name should be same as model name given in namespace.

For example:
            ModelName: test.xqy [file name containing the same name as model name in namespace]
            Then Namespace: http://www.example.com/model/test

2.      In view XQuery file namespace should have 'view' followed by the view name and file  name should be same as view name given in namespace.

            For example:
ViewName: test-default.xqy
 [file name containing the same name as view name in namespace]
 Then Namespace: http://www.example.com/view/test/default
   
             ViewName: test-custom.xqy
  [file name containing the same name as view name in namespace]
              Then Namespace: http://www.example.com/view/test/custom

[Note: default and custom in the view namespace denotes the type of view such as tree view, custom view , abstract view etc.]
   
3.      In controller XQuery file namespace should have 'controller' followed by the controller name and file  name should be same as controller name given in namespace.

  For example:
  ControllerName: test-controller.xqy
  [file name containing the same name as controller name in namespace]
  Then Namespace: http://www.example.com/controller/test


4.      Namespace URI before '/model/','/view/' and '/controller/' should be unique

 For example: 'http://www.example.com' is unique for model,view and controller URIs.
  
 model-- http://www.example.com/model/test
   
 view-- http://www.example.com/view/test/xxxx
 [Note: xxxx in the view namespace denotes the type of view i.e tree view, custom view ,    abstract view. Framework will generate view with 'viewname/default']
   
 controller-- http://www.example.com/controller/test

Follow the naming conventions as given below while mapping the module-names in mapping config xml file, which will be passed to the tool to generate the code.


1.      Controller name should always ends with "-controller.xqy"
             For example: test-controller.xqy
  
2.      If want to handle binary data in the request, then controller name should ends with "-binary-controller.xqy"
            For example: test-binary-controller.xqy
  
3.      Model name could be any name. But name should tell that, which module it belongs to.
   
            For example:
            Modelname: test.xqy
[Here, name 'test' saying that it belongs to test related module where controller name is test-controller and view is test-default]
   
4.      All view xqy files should ends with a type of view
For example: test-default.xqy

Here, -default in the view namespace denotes the type of view such as tree view, custom    view , abstract view etc. Framework will generate view as 'viewname-default.xqy'


4- Using the generated MVC:

 
Suppose you have generated the MVC based on the above given example.

1.      Let say your controller name is: test-controller.xqy and having method as get($params as element(uit:params)).
2.      Let say your model name is: test.xqy and having method as get($id as xs:unsignedLong)
3.      Let say your view name is: test-default.xqy

         So, to send the request to above controller follow the given naming convention.
  
         Your REST url should have controller name followed by the method you want to access.
   
         For example:
           
         http://localhost:8080/test/get?id=1254
           
Here, 'test' is the name of controller and 'get' is the name of the controller you want to access.
           
            It will return the default view i.e whatever written in test-default.xqy
If you want to fetch an specific view then for that a view should already be present. You need to pass the parameter as view=viewname
           
            To call a specific view:
            http://localhost:8080/test/get?id=1254&view=custom
            It will return the custom view i.e whatever written in test-custom.xqy
    
       
Method written inside the 'test-controller' will intern call the model. After the processing the request from the model view will be invoked.
       
For example:
       
 (:sample method in controller:)
declare function get($params as element(uit:params)) as element() {
    (:Calls the model:)             
    get($params/uit:id); (:Method declared in model, as given below which accepts Id as param:)
};
   
  (:sample method in model:)
 declare function get($id as xs:unsignedLong) as element() {
     (:Do something:)
 };
   
           
 (:View sample code:)
           
 declare namespace test-default="http://www.example.com/view/test/default";

declare variable $response-to-process as element() external;
declare variable $view as xs:string external;
           
  try{
     (:TODO-Process the view as required:)
    <default-view>{$response-to-process}</default-view>
} catch ($excp){
xdmp:log(fn:concat("VIEW-EXCEPTION: Exception occurred in view : ",$view," , Returning the original response!"),"error"),
            xdmp:log($excp,"error"),
    $response-to-process

}

Sunday, September 22, 2013

XQStyle - An XQuery analysis tool

What is XQuery ?

- XQuery is a query and functional programming language.

- XQuery is designed to query and transform collections of structured and unstructured data,      usually in the form of XML (not just XML files, but anything that can   appear as XML).

- XQuery is a W3C Recommendation.

- XQuery is compatible with several W3C standards, such as XML, Namespaces, XSLT, XPath, and    XML Schema.

- XQuery is built on XPath expressions.

*****************************************************************************

Why Xquery ?

- Now a days XML is most widely used data transportation and data storage medium, XQuery is    an official W3C recommendation to deal with the XML data.

- XML use is widespread across modern information systems in all industry, government, and    academic sectors.

- XQuery is used to extract, create and manipulate the XML data.

- After the evolution of MarkLogic, exists, BaseX, XMLdb etc. XML databases XQuery is becoming the primary language to deal with XML content.
**********************************************************************

Why an analysis tool is needed? 

- A good quality code is a code which is well documented, easy to understand, well structured and well optimized.
- Analyzing code without executing it. Generally used to find bugs or ensure conformance to coding guidelines.

****************************************************************************************

1-  XQStyle Overview

Problem:

XQuery is designed to query XML data - not just XML files, but anything that can appear as XML, including databases. Xquery is a recommendation by W3C.
W3C recommends coding conventions and coding standards for Xquery that should be strictly followed to deliver a good quality code.

At present we don’t have any specific tool/plugin for xquery code analysis which performs automated code analysis or code review.
We have to do it manually and that also requires extra hours and extra effort to ensure and deliver a quality code.

Approach Finalization:

The idea for XQStyle tool came from an existing tool PMD (A java code analysis tool).
After researching and analysing a lot on the working and features of PMD, I decided to develop  the XQStyle tool in Java, as java has very good capability for file handling, IO operations, string handling and xml document parsing.

Solution:
"XQStyle - An XQuery code analysis tool"

Features:

v  XQStyle is very easy to use, it can be easily configured on developer machine.
v  XQStyle supports analysis of XQuery versions such as 0.9, 0.9-ml, 1.0 and 1.0-ml .
v  XQStyle can be easily integrated with build tools such as HUDSON/JENKINS (CI Tools).

v  XQStyle takes the customized rule set configuration in the form of XML same as PMD.

v  If developer will not provide the custom rule set configuration, XQStyle will analyse the code using its built-in rulesets.

v  XQStyle has basic all, design & code optimization rules recommended by W3C for XQuery code analysis. 

v  XQStyle rules sets can be easily customized by developers as per their specific needs.

v  Developers can categorise rules and set the prioritise of the rules as per their specific needs.

v  Developers can also pass external parameters specific to configured rules which requires external inputs. For eaxmple, Code Size or Function Length.

v  Developers can easily enable or disable rules without removing the rule from the rule set xml.

v  XQStyle is capable of analysing the code for a single xquery file as well as on directory input by recursively traversing the file system for the given directory.

v  It generates code analysis reports in two formats XML & HTML based on the configured renderer. XMLRenderer generates the report in xml format and HTMLRenderer generates the report in html format.

v  Violations in the report can be easily filtered based on “High”, “Normal” & “Low” priorities.

v  XQStyle can be easily integrated with XQuery code repository using Ant scripts/Batch files.


2       Using XQStyle 

2.1         System Requirements

v  Should have Jre 5 or later version installed. 
  Dependency library: commons-lang3-3.1.jar or later (Download apache commons library from https://mvnrepository.com/artifact/org.apache.commons/commons-lang3)
v  The path where the tool will be installed must have enough space and must have admin privilege.
v  Apache ant 1.8 (Optional, required only when ant script is used to generate reports).

2.2       Using on developer machine via windows bat file.

         To use this tool on developer machine you need to configure it using following steps.

v  Extract the XQStyle.zip file in C:\XQStyle or C:\XQStyle drive which contains xqstyle.jar,its dependencies and a bat file ("runxqstyle.bat") .
v  Click on the bat file “runxqstyle.bat” to run the tool.
v  Provide the xquery file name with full qualified path or directory path.
For e.g. : “C:\Users\Abhinav\Desktop\XqueryRepository

v  Provide the custom xqstyle ruleset xml file with full qualified path [its Optional, Inbuilt rule sets will be used if not provided].
For e.g. : “C:\Users\Abhinav\Desktop\xqstyleruleset.xml

v  Provide the output directory will full full qualified path, where reports will be saved.
For e.g. : “C:\Users\Abhinav\Desktop\report



v  Go to output directory of reports, you will be able to see reports generated by XQStyle.
v  Click on the “index.html”, it contains report links for every xquery file and also package name where xquery file kept on.



v  Click on each link to view the report specific to each XQuery file. You can filter violations based on High,Low and Normal priorities using check boxes.



v  Unmark “High Priority” checkbox if you want to filter violations on Low and Normal Priority.




v  Unmark “Normal Priority” checkbox if you want to filter violations on Low and High Priority.



v  Unmark “Low Priority” checkbox if you want to filter violations on Normal and High Priority.



2.3         Using on continuous integration tool (Jenkins/Hudson)

        v  Create a job on HUDSON/JENKINS specific to your project if not already created.

        v  Create a ant build script xml and add the below given script in the "xqstyle-ant-script.xml".  
   

<project name="XQStyle" basedir="." default="about">
                               
  <property name="lib.dir" value="${basedir}/xqstyle-lib"/>
  <property name="app.dir" value="${basedir}/src"/>
 
   <target name="about">
                 <echo>XQStyle xquery code analyser script.</echo>
   </target>
                               
    <!--
       =============================================================
                    XQStyle CONFIGURATION START
       =============================================================
    -->
       <target name="xqstyle-init">
              <echo>******Cleaning the existing xqstyle-reports******</echo>
              <delete dir="${basedir}\xqstyle-reports" />
              <echo>******Creating the required directories for xqstyle-reports******</echo>
              <mkdir dir="${basedir}\xqstyle-reports" />
              <touch file="${basedir}\xqstyle-reports\index.html"/>
 <!-- Pass the source directory or file as a parameter. e.g. -Dsource=c:\test\xquery -->
      <property name="source" value=""/>
  <!-- Pass the source directory or file as a parameter. e.g. -Dsource=c:\test\xqstyleruleset.xml. 
       If this parameter is empty then built-in ruleset will be used -->

              <property name="ruleset" value=""/>
       </target>
            
       <target name="run-xqstyle" depends="xqstyle-init">
              <echo>******Processing xquery analysis******</echo>
              <java classname="com.xqstyle.renderers.XQStyleHtmlRenderer">
                     <arg value="${source}"/>
                     <arg value="${ruleset}"/>
                     <arg value="${basedir}\xqstyle-reports"/>
                     <classpath>
                           <pathelement location="${lib.dir}\xqstyle.jar"/>
                           <pathelement location="${lib.dir}\commons-lang3-3.1.jar"/>
                     </classpath>
              </java>
       </target>
   <!--
      =============================================================
                XQStyle CONFIGURATION END
      =============================================================
   -->
</project>


v  Install “HTML Publisher Plugin” in Hudson/Jenkins via manage plugins.




v  Go to “Configure” job for your project.
v  Under “Build” section configure targets as “run-xqstyle” and Build file as “./xqstyle-ant-script.xml”.



v  Go to “Post Build Actions/Taks” section.
v  Select “Publish HTML reports” chekbox and click on “add” button.
v  Provide the location or path of reports directory created by ant acript.

         



v Save the configuration changes.


v  Click on “Build Now” to build the project.
Console output:



v  Go to the workspace i.e. “XQStyle-Poc”.
    
                           


v  Click on HTML Reports to view the generated report by XQStyle.

v  Reports can be downloaded in zip format by clicking on “ZIP” link highlighted in red in the index report.
v  User can directly navigate back to work-space by clicking link highlighted in red in the index report.
v  Reports can be viewed same as described in first section 2.2.

2.4    GUI

o   Open 'XQStyle.exe'


o   Select the Input directory
o   Select the target directory
o   Select ruleset xml (Optional)





o   Click on “Run XQStyle” to start analysis



o   See the log file.


o   See the report generated in “Output Directory” provided.

o   Open the “index.html” file.


o   Reports can be viewed same as described in first section 2.2


Developed By: Abhinav Kumar Mishra

Copy of this tool is available here under Apache 2.0 license.