Month: August 2015

Make Sense out of your Stack Traces

I recently came across this cool tool http://www.stackifier.com/ which allows you to make so much sense out of your stack traces. Once I have locked on a stack I want to look at, this tool makes it so much easier.

You can also read about it on their blog post – http://blog.takipi.com/stackifier-make-sense-of-your-stack-trace/

This part of my life is called “Chasing Quality”

Chasing QualityAs far as I can recall, one thing that always stuck with me is the OCD around doing things the right way. No matter what I did – could have played a game of cricket when I was a kid, my studies and needing to score a perfect 100, the time when I was fixing computer monitors, writing scripts in fox, writing code in Java/.Net and even now when I am defining architectures.

Morale compass pointing north around Quality means nothing if we do it at the root. If a developer, a lead, an architect and me decide to drop focus on quality at the first instance of delivery delays.

I won’t go back all the while, but I will cover what quality meant to me when I was a developer. While my quality compass is still pointing north, it has been a while since I have sat down and defined what I want to do by the time I deliver this project. Of course there are expectations like “Deliver on time”, “Deliver against scope/requirements”, and then there is that “Deliver as per quality”. Of these 3 the first 2 are fairly easy to measure and clients do it well. It’s the Quality one that has become that beast – and as I have transitioned to working in an industry when we have to deliver websites (.com) where the user base is just unknown the expectations that clients now have are very different from what they were in a behind the locked doors of “enterprise CRUD application”.

I have had The Quality Conversations several times in past but I wanted to do something different

Today in a few hours I walk in a room with 6 of my architects and talk about Quality. I know my vision of quality, but that means nothing if in a few weeks, we will make decisions to drop quality measures because we wont be able to meet the Go Live Deadline. That Vision means nothing if my team has different viewpoints of quality.

Quality meant, once my work is done it’s done.

As a developer

This goes back to the time when I was just coding and there was no team leading, no program management responsibilities on my shoulders:

  1. Quality used to mean to send a story to my QA track and knowing that they won’t find even a single defect;
  2. Quality used to mean that when I would get time, I can open the code and refactor knowing that my unit test cases will tell me what I broke as I improved my code for quality;
  3. Quality used to mean that when I will commit the code, and when CruiseControl fires up it will not break the build;
  4. Quality used to mean that when a colleagues opens my code to enhance with a functionality (s)he won’t have to come back to me asking questions; (s)he won’t have to refer to an external documentation to know what this code does
  5. Quality meant, once my work is done it’s done. There is nothing more that I had to do.

As an Architect

My goals mean nothing if the team I am working with don’t share the same goals.

I am asking myself – if those 5 simple rules that I followed 10 years back worked well for me – should they change?. Does my role today means I need to do anything else. Yes, I produce some design documents now, I write some code, I prepare presentations and yes I own all of quality including whatever my team produces, but why should those simple 5 rules change one bit.

We are as strong as the weakest link.

So the question is not really – what quality means for me, but what it means for those 50 people on my team. Do they want to meet these 5 goals or do they want to do something else. Yes, I will share what my goals are, but I also now need to know what their goals are. Then we have to make a journey and do it fast is to find the least common denominator and sign up for those (as a start). Next, we need to work together to find what herculean effort we have to fill those gaps.

Chasing Quality

My Goal today – Never spend a minute extra in office. Deliver on time, Deliver on scope and Deliver with Quality – something that makes everyone smile.

My bigger task is – make sure no one on my team is spending an extra minute in office.

So I continue to find the perfect formula. I haven’t seen that happen and the several articles and books I read tell me there is no secret recipe and I have almost all forces of nature working against me, yet it’s my task to make it happen for my team. Let’s see how it goes. Now start yet another journey to meet the almost impossible.

What does Quality Mean for me

We will see how this goes. But, for now the 5 rules for quality for me stand even today are:

  1. Deliver an artifact and make sure no one can find a defect;
  2. I can refactor the code knowing it won’t break anything;
  3. Code commits work like a charm and i can deliver code in environments quickly and with certainly;
  4. When I send this application to my dear friends in operation, they don’t hate and curse me;
  5. Once my work is done it’s done

Stories and artificial intelligence

Ravi Pal aside from being a dear friend is someone I respect a lot; he is a technologist who wows me every time I talk about something. I recently came to know he has been writing a blog.

I managed to read only 1 post till now and the very first post is amazing. I’d to share with all of my readers and I quote him to

As a young soul I was always fascinated by stories, stories of Rama and Arjuna, stories of freedom fighters and stories of superheroes (guess we all have been fans of larger than life heroes). It was always fascinating to hear about the superpowers that these heroes possessed, how they were able to overcome the odds and how they had a personal story – a wonderful journeys of their own ups and downs. As a kid it was almost as if I wanted to live their lives, always felt connected to their emotions.

stories and artificial intelligence.

 

Don’t Like Throttling?

You don’t have a choice – the underlying system (The JVM here will do it for you).

I still recall the summer of 2013 when I was running a project and it was 1 URL in my whole of application that brought the servers down. The problem was simple – a bot decided to index our site at a very high rate and the bot was creating a millions of URL combinations which bypassed all of my caching layer and they were all hitting my application servers. Well we had a very high cache rate in the application (95%) or so and the application server layer was not designed for a high load (it was Adobe AEM 5.6 and the logic to do searches and make pages was very computational heavy). Earlier that year we wanted to handle the case of Dog-Pile effect and we had spoken about having some sort of throttling in place. At the start of the conversation every one frowned about the idea of throttling the same (except 2 people).

In fall of 2012 Ravi Pal had suggested to have error handling in place such that a system should not just fall on it’s head but degrade gracefully. I only realized the gravity of his suggesting when we hit this problem in 2013.

Now I am here working on yet another platform and the minute I bring up the idea of throttling, it’s being frowned upon again. One guy actually laughed at me in a meeting. One other person suggested that we want to handle the scenario by “Auto-scale” instead of throttling the same. We have our infrastructure on AWS Cloud and I am not expert but the experts tell me a server can be replicated as-is in around 10 minutes (we will be proving benchmarking this out very soon).

I was an ambitious architect who though I controlled the traffic coming to my site. I no longer live in that illusion.

This may be a series of posts, but today here I start off with showcasing that you do not have a choice and whether you like it not, the system will throttle your traffic for you.

The Benchmark Overview

  • A simple Web application built using Spring Boot
  • A Spring MVC REST controller that will accept some HTTP Requests and send back a OK response after a induced delay
  • jMeter to simulate a load
  • custom plugin (a big shoutout to these guys for the plugin) to generate stepped load and capture custom enhanced graphs
  • Tomcat 8.x to host the web site – launched in memory using Spring Boot. No customizations done

First Groups – The Good One

Test Plan

This Thread Group is going to simulate a consistent stream of requests to our application server. A typical scenario that happens very often.

Throttling - thread group - the good one

Server Performance

As Expected? Yes.

As you see below, the chart shows that the application server is behaving normally. All the requests over a time period of 15 minutes is consistent with a “single user model” aka 1 second request response time.

throttle - the good one - TPS - Scenario 1

Second Group – The Sudden High Traffic

Test Plan

This test plan is a stepped approach and it’s trying to simulate a scenario where a campaign will start hitting a certain page (or set of pages) for a short duration. This is a use case we see most often in the industry where our websites are open to the whole world to hit.

this thread group is not OOTB and I downloaded a plugin

throttle - the high traffic one - test plan

Server Performance

So what do we expect to happen? Depending on how much juice my server has (threads, cpu cycles, etc.), my server may or may not be able to handle the requests. Given I am running everything on my local laptop, it will be interesting if my local box can handle 600 threads on .

throttle - the high traffic one - TPS - Scenario 1

And we see that my laptop cant really handle 600 thread. So what does tomcat do?

It Throttles

How the Good One changes behave

Test Plan

I run the 1st test plan and follow it up with the high traffic plan (introducing a 30 second delay).

Impact

The following image shows how the Good One has been impacted. While the traffic for The Good One has not changed a bit, it has still been impacted because something else introduced a spike.

Please go and tell the JVM that you do not like throttling

throttle - the good one - TPS - Scenario 2

So What’s Next

You have really 3 choices (we will look into details of each of the following in separate posts)

  1. Auto-scale the application servers and hope that the new servers are ready in time to handle the load or;
  2. Do something about throttling and control your destiny – what if the high traffic is not a revenue generation resource and the Good One was?
  3. Continue to frown upon Throttling

Are annotations bad?

I eased off into this topic with my principles on my post about Spring XML vs. Annotations that other day. This easy inlet was also my way of not complicating things too much for my team who is currently involved in writing this new app that will probably have a production life-span for 3-5 years (if we do it right and hope world of technology does not changes over it’s head).

I have been working with Spring Days since 1.1 so yes I have a level of comfort working with very large and complex XMLs. But, I know how to write them and more importantly I know how to read them. Since then Spring has made it easy for developers to understand them – Spring STS with Beans Explorer /Graph. Developers now do not have the need to worry about looking at multiple XML – those tools do the job for them even writing and managing beans for them.

We sacrifice the art of writing good and performant code for the short term gains of improving developer productivity

Since I saw Spring 3.x introduce this notion of Annotation based configurations, and the hype train of using these annotations instead of using XML has been huge for at-least 7 years (if i remember correctly). I have not been able to make peace with this change in direction. Not saying it’s bad, but the point that this feature has been anything but abused by the community to it’s core and Spring has been guilty of promoting the abuse. Any Spring Documentation today, talks about annotation-style’d coding only to follow with the “classic XML way” of doing things.

While people say – it’s easier to read the Code, it’s easier to debug the code with annotations in the mix, they forget what’s it’s not code in code anymore – they have embedded configuration in code. And as far as I remember Configurations were supposed to be externalized. The problem is more severe in cases where we use ORM frameworks like Hibernate and JPA.

Even in original Spring Design, even with XML I feel that how we setup spring applications are not what spring was design for. It’s time for me to go find what Rod Johnson had in his mind when he designed Spring (I know a bit but I need to find some details and get into depth). But thats for another day.

So let’s look at this blog post that explains using JPA with Spring or read this StackOverFlow thread. Well, they both explain how to use, but very soon we realize that but using these so called Rich Annotation based configurations in Code we have diluted the overall meaning of What code/design is supposed to be. This style of programming is great when I have to try something new as a personal pet project to get off the ground quickly – i can just write a class, type a few annotations and boom i am ready to do CRUD, but does this really works in enterprise level applications especially how do we manage this in production.

These articles are nothing but a bunch of marketing/sales pitches that want us to go use these frameworks and new features, but they hardly put in context the complex situations we have to deal with in big production systems

In 2007, we used hibernate extensively on our project (with Spring 2.x with XML based configurations) and we realized very soon that we had taken the ORM framework beyond it’s limits. we had complex queries which we were trying to retrofit into Hibernate and something that was possible to write in MS-SQL as optimized procedures and fire away those queries were now becoming major bottleneck. I was new to the framework but more importantly I had a push from my technical leadership to use Hibernate to it’s fullest. Those people had access to article like I shared earlier and this looked like the way to go but they were nothing but marketing material to sell a feature that Hibernate and ORM brought onto the table. When rubber hits the road is when i had to go back and refactor the code and follow good old ways of writing queries.

90% of the times these frameworks that use annotations work well, but those 10% where you need your system to perform under stress is EXACTLY when these fail

Back tracking to Spring and Annotations now – why i do not like them? Simply because they make me write code like I am a college student who is learning something. They force me away from what used to be good practices in golden old days. Yes it used to take time to setup a few bunch of classes and it used to take time to write the SQL queries but I had right stuff in right places. And Yes it took time before we gathered momentum, but once we had those basics setup tight not only we could development speed, we also had done the things the right ways.

And yes no one can force us, but the average Joe Developer or the average Jim architect do not have the time and inclination and make these POVs, they do a google search and when they see 5 articles saying the same thing, they presume it’s the right thing to do and they proceed happily. And many of our Senior Technologists who also read these articles support the designs and many a times challenge the POV of what I am trying to put here.

TLDR;

Think about it and please do not use annotations to configure your applications. Configurations were never meant to be part of code – the reason they are called configurations. So let’s let those be. A small gain in short term wont go the long way especially when a client asks for a change in a table or a value and you tell him that will beed 5 days of development, testing and deployment.

Making Thread Dumps Intelligent

Long back I had learnt about something called Log MDC, and I was a big fan of it. I was suddenly able to make sense of anything that happens in log files and pin-point to a specific log entry and find what’s right or wrong with it especially when it was about debugging a bug in production.

In 2013 I was commissioned to work on a project that was running through some troubled waters (combination of several things) and almost every week I had to go through several Java Thread Dumps trying to make sense what’s happening in the application to make it stop. Also, there were times when I had to have profilers like AppDynamic, jProfiler, jConsole all hooked up to the application trying to find what’s the issue, and more importantly what’s triggering the issue. jStack was one of the most helpful tools that I had worked with but the thread dumps being bumps had no contextual information that I could work with. I was stuck with seeing 10(s) of dumps with stack traces of what classes are causing the block but there was no information of what’s call and what inputs were causing the issues and it got frustrating very fast. Eventually we found the issues but they were mostly after several rounds of deep debugging the code with variety of data sets.

Once I was done with that project I swore that I will never find myself in that situation again. I Explored ways in which I can use something similar to Log4j’s NDC but have that in threads so that my dumps mean something. And i was able to find that I can change the ThreadName. And my next project I did use that very effectively. I recently came across an article that explains that concept very well. I am not going to rewrite everything they said, so here is a link to their blog post.

So last week I am starting a new project and as I get into coding the framework (using Spring 4.1 and Spring Boot), this is the first class I am writing for the application and ensuring that the filter gets into the code ASAP which not only helps us in post-production but also makes my development logs meaningful.


A copy of the code for both Log4j NDC, and setting up a ThreadName is below.

Spring Framework – XML vs. Annotations

This question has been around for many years since Spring started to move heavily towards Annotation based configurations (if i recall right it’s called configuration by convention). Annotations based configurations was like a jungle fire which spread across the industry and very soon it was a norm. But, this question “XML vs. Annotation” always existed.

I for one have been around in Spring world since it’s version 1.1 when annotations weren’t a thing and I know what’s it about to write those XML and the power to configure an application to suit my needs. Since then whenever I have went about writing an application in Spring I have asked myself this question and I never really had a good answer until recently. While you will find tons of post around in google,  when you search for this, only a few really give you an unbiased opinion.

I started to work on an application that needs some very flexible configuration options and before I dive into that I had to yet again make this decision and this time I wanted to keep things simple and my rationale was…

Use Annotations to anything is is core to the application and defines the core structure of the application. Anything that would need a code change is okay to sit as an annotation.

Use XML based configurations when you know you may have a need to alter the behavior of an application without the need of compiling and deploying the code all over again.

This is how simple i kept it for my team. Once this principle is defined, job is only halve done. But we will get there soon.