A hierarchical database model means that my data is arranged into a structure that is similar to tree (organization chart). This resides on the premise of a 1:N relationship where a child can have only one parent, where in a parent can have multiple child records. It has characteristics that differ a lot from a relational database. To list a few:

1. Every node is a record
2. Data is stored as properties on the node
3. Every node can be of a different data type – a hierarchical model does not mandate to have same record types under a same parent
4. A child node can be a child to one and only one parent

Hierarchical databases have their advantages:

Performance: Navigating records in a hierarchical model is faster because the references are basically pointers to the nodes/records directly. I don’t have to search in a index or a set of indexes. This however, is true in a case only when my data model does not have a lot of references. If i am working off with a content-model that includes multi-level references, performance will head south
Easy to understand: It is a simple hierarchy; and it represents something that is “non-technical”. It naturally represents what exists.
And Hierarchical databases have their limitations:

Unable to draw complex relationships between various child nodes – Given the premise that a child node will have only one parent, they are identified only by their parents. We have the capability like XPath to navigate directly to a node, which may be faster. If we do not know the exact path, we will have to navigate the tree (up to a parent, maybe the root) and then down to all nodes before we find what we are looking for. Some questions that I am asking myself:
1. What qualifies as a reference for an object?
2. Should speed at which the data can be fetched a driver to defining a reference?
3. What are the best practices that I should be aware off, when I am modeling my domain?
4. When do i decide I need a network model instead of hierarchical model?
Difficult of maintain – hierarchical models also mean that I do not have a command like ALTER TABLE. This essentially means then if I later decide to add another property to a specific node type I will have to write code to update all the nodes
1. Is there a way where I can update a node-type thus updating all the objects which are of that node type?
2. Is there a way to avoid such situations (apart from saying that lets get it right in Release 1.0 and pray to God client will not ask for a change request )
Lack of Flexibility – In this article, Scott Ambler quotes – “Hierarchical databases fell out of favor with the advent of relational databases due to their lack of flexibility because it wouldn’t easily support data access outside the original design of the data structure. For example, in the customer-order schema you could only access an order through a customer, you couldn’t easily find all the orders that included the sale of a widget because the schema isn’t designed to all that.”. This is a typical case of where reporting is a must and it might be in many systems.
1. Are there other scenarios?
With all the context set of Hierarchical, it is now important we look at CQ54′s content repository – CRX. While CRX is a hierarchical repository it should not be confused with a hierarchical database. CRX provides us with JCR node types which allow us to force structure. We also have the capability of creating custom nodes, but should do it with care. The principle is not to go overboard with structure.

Question remains – “how do I manage content in CQ54″. I do not have a “go-to” answer, but what I have described below is how I am going to think when I start the process.

Content modeling: Look at the requirements i.e. wireframes, creative design assets and identify various content types, structures and relationships between content types. We can take the object-oriented approach and define everything as an object or keep similar content types together. There are several things that should be considered when taking one approach over the other:

What is the business process for crating an object type. Do the content types follow same workflow?
1. Steps that are required to activate a content. An article, a blog, a discussion forum entry may have the same process flow of an author and a reviewer then there is a case of having a single abstract content type
2. However, if an article needs a legal review and can be used in several other business process than just a simple article we may want to bring article out as its own content type
Reuse
1. What kind properties do they share
2. Modeling content for an education system where we have content types like a college or a school where we see a lot of similarities there is a case we can build on creating an abstract content
How does the content author wants to look at the content
1. If we have a set of users who want to manage their content as structured content like books, movies etc we should look to provide those content types very specifically
2. In another scenario if we have authors who do not worry a lot about specific objects i.e. Page-centric content creation then we can decide to club content types together

Managing Relationships: In CQ, given it has a hierarchy based data storage model which complies with JCR specifications, we do not have a way to create strict rule-based relationships. We can create relationships using one of the following ways:

Path based references: We can do this by creating properties on objects that hold a “path” or a “list of path” to which the content has relationships with
1. They are semantic
2. Not bound to an “obscure IDs”
3. Do not enforce integrity constraints which may create troubles in extensibility later
4. Being REST-ful they allow us to navigate directly to the node, thus making navigations very quick
5. Being REST-ful, they allow author to visualize their content relationships well thus providing them a business view of the content
Taxonomy based references: CQ uses tags to represent a taxonomy. However,  we can not extend tags to hold various profile information. So, you will need to have a mapping system that maps a tag to a content in CRX
1. Taxonomy is the foundation on which the IA stands. Taxonomy allows us a classification system and how the users will view the content on the site.
2. Allow us to clearly identify where in the system the content type resides
3. Is a conceptual framework allowing customers and their customer to locate what they need easily
4. It is hierarchical
Relational Database
1. Can be used in case we reach a point where relationships are too complex
2. Transactional Data should be kept out of CMS and placed in a relational database (or similar)
3. If we do not have to manage the lifecycle of the content
4. Please note that this  will make architecture complex, but if this is needed that it is

The language is important because it helps you think about the problem in the right way.

This statement stuck a chord reminding me of an instance not so long ago . She asked me – “why did my project had just a couple of risks?” . She went on to probe us (my PM and me) to understand if we were not thinking of the risk.

At that time, I did not answer her in the way the Author summarizes it. We presumed that we had a functional scope documented and hence the risk of not being able to deliver what was required did not exist.

via The Language of Risk « The IT Risk Manager.

Cache is a component that will magically store data so that future requests of that same data will not be to the Remote Server, and hence it will improve the performance of our application significantly faster

We were working on a website which was to integrate with an existing application through the use of APIs and for purposes of closer integration we had decided to store the data in form of XML as artifacts in this tool as tracker items – no RDBMS. It was like running two applications joint at the hip.

I had gathered enormous experience working with SourceForge platform APIs as I had integrated them with Ms-Excel and now we were going to build an entire application on ALM space using the same APIs. Our biggest challenge was going to be performance because of the use of APIs against a remote server sitting in a different geography. And Cache was going to be instrumental in helping us solve that problem.

We had just decided to implement Write-through cache that helped us build a system which was significantly faster that anyone could have thought about. And since then this is one pattern that I have come to use (if possible) whenever I am working with diversified systems. Surprisingly, the percentage of people who have used cache have never heard of this pattern of cache implementation when the underlying issues with the systems can be solved using this pattern (I am still mystified as to why not?).

Before we dig deep, I want to run through some definitions that we are going to use during the course of this article:

• Cache hit refers to a request to the data and finding it in the cache
• Cache miss refers to a request to the data and not finding it in the cache
• Dirty refers to a cached data if it has not the same as the original data
• Lazy refers to an action if it not performed real time, but only when it is required

In its simplest form, a cache implementation is going to something similar to the image below.

Cache - Workflow

As you can already see that this is just one part of the cache implementation and implementing this workflow alone would mean that once I have a data in the cache it would only always fetch the data from the local cache location and go back to the original data source only if the cache data is Dirty. And there are several ways we have to mark the data as dirty – it can be an action we configure in our system like – “If we are update records in the data source, we try to find the key and mark it dirty”. Another way is to decide a time after which the the cache should be expired automatically.

While, this approach is simple enough, this does presents a unique “problem” (and it may not be a problem for everyone). Lets revisit the reason for which we decided to implement cache.

 Cache is a component that will magically store data so that future requests of that same data will not be to the Remote Server, and hence it will improve the performance of our application significantly faster.

This implies that “Caching is a mechanism that is faster when compared to our data source when it comes to data loading”. I have observed cache implementations against RDBMS sitting next to a Application Server, which means that loading data from the RDBMS is still faster and hence there is really no need to improvise on the cache flow as defined earlier.

In our case, we were dealing with an external system from where we fetched XML over HTTPS and then converted the XML to an object. This entire process was time consuming – 3 seconds for one object and there was nothing we could do about reducing the transportation time. It also meant that the classic workflow for us would not work either, especially if a user would update a specific record, and if the cache would be marked dirty, the next request would mean a significant delay time.

We improvised and it was then we used the write-through cache logic that allowed us to manage the data in the cache in real-time with the data-source. The workflow was changed to the one below:

As simple as it may look it was not so. Lets see first what we did. We added a hook to the code which was required to save the data in the DataSource to do two things:

1. Find the cache entry and mark it dirty if it was a hit and;
2. Update the cache after a successful update to the data store

This allowed us to keep the cache in sync with the DataSource and hence not requiring to spend additional time to load the data back again.

But, as I feel that every solution will bring its challenges, this one had as well especially when we decided to scale and move over to a cluster of application server. This meant that a local cache would simply not work because it was meant that an update to the DataSource meant that the cache on other application servers was out dated and users would not get the most latest data making it impossible to work. We did use version to records to manage the concurrency checks, and not keeping the cache in sync meant that other users will see their updates fail because of that very fail-safe. Eventually, we had to find a cache that can be scaled in a cluster which only made things more complicated.

A pattern that I learn in my development adolescence, this had proved to be a powerful technique to build solutions that would work fine in a given scenario.

It was Adobe Day that introduced me to OSGI and during one of the webinars, I was introduced to OSGI using a simple yet powerful graphic (see below).

OSGI - Intro

And from that moment on, I was simply hooked on. I have been a big fan of Java and its deployment frameworks like Maven – but essentially using those frameworks and tools also meant that sooner or later I would be dealing with various versions of same library (ehCache, Log4j being the most common) and when I wen to deployment on JBoss or Apache Tomcat, more often than not I would have to tweak my project dependencies to or servers to ensure that those “dependencies” are resolved appropriately.

Suddenly, OSGI seems to be the answer to everything (or almost everything). I would be able to create and deploy components and then choose to include them as needed; I did not have to worry about backward compatibility or which service to deploy – opportunities were endless.

So today, I am going to start off my journey to read more about OSGI and explore some common available containers like Equinox (used by Eclipse) and Apache Felix (used by Adobe Day). And I hope that while I learn more I am able to share my thoughts and some of the best practices around the same.

• When we reach out destination – on time or late
• In what condition we reach – as expected or all burnt up

I have done this activity several times in my career, and yet still there is one or the other thing that is left behind. Suddenly, the clients and project managers  want to see results. They want to see what is being built and how things are coming up. Suddenly, all the housekeeping tasks become nuisance and nothing is getting accomplished unless there is something to be seen. So, what goes wrong? Three Things

Planning

It is mandatory to plan for things up ahead. We may not be navigating the ship, but are responsible in many ways that everything that is required for the trip is on board. If the ship is not built to stand the storms, it will crumble; sooner or later. All that is required to be done should be a part of the plan. Once other can visualize the value that the activities will bring, is becomes very easy to negotiate and progress. It becomes simple to justify the load. Once it in the plan, then no body is worried because it is accounted for.

Execution

No matter what project you are on, things are bound to go bad. On any journey, there are bound to be some bumps. How much they affect your ship or its passengers is again decided how well you are prepared. Prepared not just with the tools, but also with the expertise to execute measures. You will seldom have a A-team, and you need to know who performs how and what scenario. Planning will take you so far. If you do not know how to utilize the resources available to you at that time, you are as good as dead meat. It is panic that causes most troubles. You have to find ways to be calm especially when there are fires all around you.

Expectations

Any scenario, we fail to set and manage expectations – our family, friends, shop keepers, be it anyone. However, some (most of those) do not come back to haunt us. We can bail out easily with minimal impact. All of it happens when we start to assume. And, it happens because we think we have understand what the other person is trying to convey. And, the longer we linger the process of validating our assumption, the bigger impact it has on the outcome. We should have only one assumption – “There are no assumptions”.

As an Architect, there are certain things I believe have to do. There are several books and handbooks that talk about all that stuff. However, many don’t have access to them or they have so many things that this topic gets lost somewhere. In my follow-up post, I will publish the activities, with descriptions and their possible exit criterion that all architects should be aware off.

Recently, when I decided to make my projects available to the Java Community under Open Source license, I did a lit of things like Hosting my code on Google Code, using Maven to build and test the code and also making sure that I have a project site up and running. However, even after all of this, there was one thing that was missing which was the most essential – “Hosting the project files on a repository so that users who may find these utilities useful may use it”.

I started to research on what was possible and after little research I found out that sonatype would allow me to deploy my code to Maven Central Repository. However, after 3 days of trial and error, I have finally been able to release a version of my code to the Central Repo. I am writing this post so that if you ever want to do this, you will not have to go through the same pain as I did.

Pre-requisites

• A domain name which will be your groupId – I used kapilvirenahuja.com which is used as “com.kapilvirenahuja”
• A PGP key – this is a requirement for you to sign your artifacts and upload it to Central. You can refer to the document here for help on how PGP keys can be generated.
• Have the username and password of your SVN handy, you will need it while performing builds and releases

Hosting

The easiest way to upload another project is to use the Sonatype OSS Repository, which is an approved repository provided by Sonatype for any OSS Project that want to get their artifacts into Central Repository.

The service is proposed in two variants:

• Maven repository hosting service: You can deploy snapshots, stage releases, and promote your releases so they will be published to Central Repository
• Manual upload of artifacts: The process is mostly automated with turnaround time averaging 1 business day.

Follow the steps in the guide which is self-explanatory. After sign-up, when you are creating the issue for a new project, you will need the domain address. If you do not have a domain registered, you will not be able to proceed.

After your account is created, you will receive email notifications that you are good to go. Follow the steps in the link and update your POM as per Step 6. When you are ready, follow instructions in Step#7 and you should be able to deploy the code to Snapshot directory. If you get 401 error or “Authorization failed”, check for following:

• The groupId in the POM.XML is the same that you registered for
• The Settings.xml has the Nexus username and password. You can run the maven command in -X (debug) mode and see logs to validate the location of the settings.xml file

Release to Central

Once you have the artifacts on Snapshots repository you are past a major step in the process, but the next one is even bigger. The step 7.a.3 seems simple but is not that simple especially if you have only used mvn:compile in most cases. Ensure that you follow standard directory layout anything else and the process will fail at the last step.

Step 1

The First step is to Clean up Release using Maven plugin

mvn release:clean

You will notice that it will be done quickly (apart from the time it takes for maven to download dependencies and plugins).

Step 2

Run the following command. You will have to pass the username and password through mvn. The command should be

mvn release:prepare −Dusername=username −Dpassword=password

It will do the following:

• Ask for the version number for the current release
• Ask for the version of the tag that will be created, should be same as the release number provided above
• Ask for the next version of the release – you can choose to provide a major or minor release.

Please be careful when selecting the version numbers, because this process will update the POM files and also the subversion repository by adding new tag and also updating the POM file in the current working location.

Step 3

mvn release:perform -Dusername=username -Dpassword=password

This is a non-intrusive method and will simply upload everything to the repository. If you miss the username and password, the build will fail without any descriptive errors and you will keep wondering what went wrong.

Now follow the remainder of the documents on the Link and enjoy.

Related articles

• Publishing Project using Maven (scratchpad101.com)

Background

I came across this article on Dzone, which is just right to set the tone for me to release my latest code, which I find myself to be very proud of. The article speaks about the importance of Unit Testing for developers and how it can save the day for you especially when you are moving into Integration phase of the application. My first encounter with Unit Testing was back in 2004 and it was not just any kind of Unit Testing, it was fully automated Unit Testing using JUnit and NUnit. To add to the mix it was Test Driven Development (TDD). When I started with unit testing it was really frustrating and coding just took time. I knew that my logic was right and yet I had to write test cases. I was going through the motions until after 4 months our Architect told us that we need to a major refactoring. It was then that I saw how useful those test cases were. As we went about changing the implementations of Data Access Objects, we knew all was well once we were able to get the Unit Test cases to pass. And, from that day onwards I was addicted.

Objectives

I was working with a client in Agile mode and had a team of 40 developers who cranked 1000s of lines of code (LOC) every day and those LOCs included unit test cases. Given the nature and size of the project it was evident that I wanted TDD model for the project delivery and for the following main reasons:

• We wanted test cases to double up as requirements;
• We wanted our Quality Assurance (QA) team to actively participate in test data formation so that we can reduce the gap between our developers and testers;
• We wanted to externalize test data from Unit Test cases so that new data could be added without changing the code (Test code) and different scenarios could be executed based on the test data

Approach – Where we failed

We started with the following:

• TestNG: TestNG had support for data provides which allowed us to pass data during invocation of the test methods
• Used XML files for dbSetup and initialization. We created custom classes that allowed us to read data from XML files and run queries against database from within the test cases
• JIBX Dataprovider: We used JIBX Data provider to pass the data into TestNG during test execution. This provided expected us to map inputs and output data in XML format

However, after just few months, we realized that these choices were not helping us in achieving our goals. The entire setup was too complex and too developer oriented that we could not take it to client. Also, for the QA team it was too much of an overhead to manage all of those files. However, it was too late in the game to change anything and our development team was now getting experienced with the tools and framework. We just did not have time to replace the whole thing at the cost of refactoring and training the team on the new framework. Another problem was that there was no open source and free tool or framework that existed that allowed us to do that (this was back in 2007).

Journey

In 2008, I got so fed up with the underlying choices that I decided that this is something I would not use. The only piece I loved was TestNG – Eclipse integration, Maven integration and Data providers just made life so much easier. JUnit later added the capability for Parametrized Unit Testing, but it had certain shortfalls (I compared the two a while back, you can find the article here). In 2009, I decided that in long term I would need to have a framework in place that will help me meet my objectives and I started to look for any existing tools. I do not have my research results, but I did not find any framework that allowed me to do what I wanted (at least nothing open source and free). Even today when I search I find the following links. There is not much that they offer. Or maybe I am just not seeing those.

• Approach to use Excel with Parametrised Unit Tesitng with JUnit)
• DDSteps: This comes close to what I wanted to do, but this has been built for JUnit which was not choice of unit testing tool. Since then they have come a long way. Their current version of website was non-existent back then and documentation was missing then and it is missing now. Hence, I did not want to reply on something that could be buggy and a project that may be closed
• The biggest drawback in this framework was that it did not support child objects in Java objects. If my parent Java Object had a child object, I had to load the parent and children java objects separately and then set them manually in my test case

So I started to visualize what I needed and it was then that I conceived this framework or set of utility classes.

Concept

I have tried to keep it very simple and the result is – EAMSteps:

• External data store for test data like excel spreadsheet
• Automated assertions using co-related inputs and outputs
• Minimal lines of code to get started

Current State

• I have the basic structure of the project ready and it supports loading the data from Excel sheets only for certain data types;
• The files are hosted with Google Code and available here: https://eamsteps.googlecode.com/svn/trunk/
• Registration with Maven Central is pending; hoping to do that in a few days;
• The documentation is only around JavaDocs API and is available here;
• Project will be hosted on http://eamsteps.com, which will by end of this month and will have all relevant documentation

If you have any inputs or suggestions, please add your comments and let me know.

Related articles

• TestNG or JUnit (scrtchpad.wordpress.com)

Image via Wikipedia

Most of us use Maven for building our projects and preparing a JAR or a WAR or an EAR to be deployed on server. However, during the development life-cycle we do not realize that Maven can be used for a much higher purpose – “Create a project site to show all of projects’ information”. For many years, that I have been using Maven myself, this is something that I never thought off using to my advantage – I guess I never wanted to do something like that.

Last week, when I launched my project as an Open Source Library, I realized that I need a easy and quick way to make many artifacts available for my project. I have used Google for code hosting and issue tracking, however there are other documents like JavaDocs, list of dependencies, XRef Source that provide a lot of additional information which would be useful to whom so ever decides to use the library.

Necessity is the mother of invention

In my case, it was not an invention, but it sure was a learning experience as I started exploring maven-site-plugin. This plugin generates various reports for a maven project including those that have been configured in the pom.xml file. However, it was not a straight forward implementation because I had some additional requirements like placing Google Analytics Code and placing custom Project information on my site. In this post, I will make an attempt to explain some of the things I did to help me generate my site.

To understand what I did, you should have a quick look at the site that I have deployed. I will now start take one section at a time and explain what changes I made to my pom for that section to work. I have also attached the pom file with this post for you quick reference.

POM - Main

The image on the right explains the main sections of the POM.XML. This section will configure the Project Information and Project Team pages. I have added only a few details that i needed, but you can add a lot more information as you need.

The next section holds the License information. I am using the Apache License 2.0 and hence I am providing the URL for the license from Apache. This section generates the Project License page for site. The next two sections define the Project Repository and Issue Management (Defect Management) settings.

Rest everything is standard POM stuff i.e. dependencies and Build plugins to run test cases etc. I am not going to take a deep dive into those because there are just so many tutorials on the same. However, what is important are the sections on Java Cods and Source Code Reference. To generate these two sections as a part of the Maven Site, you need to configure the Reporting Plugins as follows:


              org.apache.maven.plugins
       maven-javadoc-plugin
      
                    http://download.oracle.com/javase/6/docs/api/                
      2.8
               org.apache.maven.plugins
        maven-jxr-plugin
       2.3
     


And lastly, to setup the left-navigation menu, you need to add a site.xml file into the “src\site” folder of you project and provide the menu details and their linking to the pages.

In this file you can also add code for Google Analytics, you will notice in the image below that I have added the Google Analytic code to the header section which then copies the code over to each of the static HTML pages and allows me to track the same.

Site.XML

I hope you find this useful and are able to use it to publish your own site too.

I just released a few more framework/utility classes for this Framework. The three new features that have been added are

Exception Handling

This simple yet powerful implementation provides some of the most desirable features in any Exception Handling framework:

• Unchecked Exceptions for Business and System errors
• Message Externalization
• Message Localization
• Configurable Message Data Sources

Message Readers

This package provides factory to fetch various types of Message Readers. The current implementation only supports Resource Bundle based message stores, but it can be easily extended to other data stores.

Concurrent Processing

This package provides base classes that allow you to run your functions in a producer and consumer fashion. By using this framework you have to worry about writing your business logic for a Producer and a Consumer and not about multi-threading and how these two entities interact. This framework currently has only one implementation of a Broker i.e AsyncQueueBroker but more will be added in coming months.

Related articles

• A new OSS project: iFramework (scrtchpad.wordpress.com)

Introduction

This is a set of utilities and classes that I have found useful over the years. In my career spanning over a decade, I have time and again written the same code or some part of the code over and over again. I never found the time to collate the details in a reusable library. This project will be a collection of such files. The work that I have been doing is more than 5 years old, however the project has been conceived in 2011.

Features

• Improved Loggers: This is built on top of the Log4j and Apache Commons Logging and provides some basic and additional capabilities like
• Logging String
• Logging Objects
• Logging only if a specific type of logger is enabled, thus improving performance
• Database Setup: During unit testing, there are times when we have to setup test data. In those cases, we are dependent on external tools like MySQL scripts or Maven builds to setup data. And, then in many cases after running a set of tests, we find the need of resetting the data to its original state. These helper classes allow us to manage such data appropriately.
• Exception Handling: This simple yet powerful implementation provides some of the most desirable features in any Exception Handling framework:
• Unchecked Exceptions for Business and System errors
• Message Externalization
• Message Localization
• Configurable Message Data Sources
• Message Readers: This package provides factory to fetch various types of Message Readers. The current implementation supports
• Resource Bundle based message stores – Messages can be read based on various locales
• File based XML files – messages are read only for a specific locale
• Concurrent Handling: This package provides base classes that allow you to run your functions in a producer and consumer fashion. By using this framework you have to worry about writing your business logic for a Producer and a Consumer and not about multi-threading and how these two entities interact. This framework currently has only one implementation of a Broker i.e AsyncQueueBroker but more will be added in coming months.
• Cryptography/Secure: This package provides utility classes to secure Strings that you may want to use as Passwords or Unique Tokens. This package provides three variants of the secure digest that you can generate:
• Secure Hash: Computes a text digest using the Secure Hash Algorithm (SHA-1) which produces a 160-bit one-way digest of the text. The digest is said to be one-way since the original text cannot be recovered from it. This is ideal to use for securing passwords.
• Salted Secure Hash: Computes a text digest using the Salted Secure Hash Algorithm (SSHA) which adds extra security to the digest produced by SHA-1 by adding a random string salt to it. Since the salt is generated using another sufficiently random algorithm, the resultant digest is more secure than the original SHA-1 digest.
• Unique Token Generator: Generates text strings that can be used as unique tokens in an application. It uses the randomness of the SSHA  algorithm to generate the tokens.

Project Details

This project is developed as an Open Source technologies using Google Hosting and other other open source frameworks like Spring, Maven, Apache, MySQL so that anyone can make use of it as they find fit. Project information has been hosted here (http://projects.scratchpad101.com/iframework/1.1.x/index.html).

Use / Download

The project is hosted on Maven Central Repository. You can use it in your project if built via Maven by adding the following depdendency:


com.kapilvirenahuja
iframework
1.1.0
jar