Agile Fluency

Nice Article from Martin Flower (or more specifically by Diana Larsen and James Shore) on Agile Fluency.

Good read on how to agile teams should evolve

http://martinfowler.com/articles/agileFluency.html

HBase schema design

The HBase reference documentation dosen't really say much about schema design and most of us coming from a relational back ground will need to change our way of thinking in order to swtich to a more column based kind of designs.

You need to look more at the types of queries that you will be needing to access your data rather that just nomarlise every thing to 3rd normal form as in traditional relational databases. And that itself might not be easy. I will be writing more as i try to design my current application's data base into a Hbase kind of design, and what are the challenges i find.

Below are some of the useful resources that helped me in understanding column based kind of designs.

http://ianvarley.com/coding/HBaseSchema_HBaseCon2012.pdf

http://traackit.blogspot.sg/2010/05/20090713-hbase-schema-design-case.html

Schema Design Guidelines and Case Studies
https://docs.google.com/viewer?a=v&q=cache:Oe0Kf9oQhO0J:india.paxcel.net:6060/LargeDataMatters/wp-content/uploads/2010/08/Schema-Design-Guidelines.doc+&hl=en&pid=bl&srcid=ADGEEShyf8WIXTdsWbYRTWWLHUgvYbWDa2AilTnm6xUWnaHDvpX6gARtzDNBx6GlsbE1hjdZoR3hYBYjk03zlwq8LLkA9VRYlLwihD4S75pxlMtitX640uwrQsOa0cC5K5JPGfhAfQIK&sig=AHIEtbR8TyqVgcc3WFGtpKWX7eZOZBpnFQ&pli=1

Varley Masters report (as referenced by HBase documentation)
http://ianvarley.com/UT/MR/Varley_MastersReport_Full_2009-08-07.pdf

Big List of 20 common bottles necks

All developers and implementors should take a look at this list of bottles necks from HighScalability.com

http://highscalability.com/blog/2012/5/16/big-list-of-20-common-bottlenecks.html.

Great stuff wish i have this list before i started work on any big systems.

Common Lisp the untold story

Nice history about the Common Lisp standardization process by Kent Pitman who wrote the Common Lisp Hyperspec.

http://www.nhplace.com/kent/Papers/cl-untold-story.html

Opening Slides from Oracle vs Google

The opening slides from Oracle
http://www.oracle.com/us/corporate/features/opening-slides-1592541.pdf

The opening slides from Google
http://www.groklaw.net/images/GoogleOpeningStatementSlides.pdf

It will be interesting to see how all this will come out, since it involves whether the API it self is copyrighted, not the implementation but the API signature itself, it the implmentation is the same then all the "worse" in Oracle's eyes.

Dr Dobbs has an interesting article @ http://www.drdobbs.com/jvm/232901227?itc=edit_stub talking about how all this has an impact on all programming languages and even OS's. Like JRuby is an implementation of Ruby and not forgetting the python implementations (CPython, Jython, IronPython).

Let's hope all this comes out in Google's favour though, i think it would be a major step backwards if no one can come up with a better implementation or an alternative implementation of the APIs.

Better Coding: Dealing with Exceptions

During code reviews or debugging sessions, i always find new programmers having a hard time trying to deal with exceptions especially when the compiler forces you to do something with them. And the thing with exceptions is that if you don't handle them properly, you might have a hard time trying to debug your programs.

What are exceptions anyway?
Exceptions are conditions that change the normal flow of the program, they are usually in reponse to some part of the code having an unexpected error that the programmer needs to handle. Things like when you are reading a file you might get an IOException when something went wrong with IO operations, or when you are converting from a String to a number, you will get a NumberFormatException when the string is not a number.

So what are some wrong ways to handle them,

Don't just print the stacktrace.
IDE's tries to be helpful and will help you to automatically write the "try catch" block statements, and they will usually put in a e.printStackTrace() for you.

public void readAFile(){
    try{
        //Some code that tries to open and read a file
    }catch(IOException e){
        e.printStackTrace();
    }
}

What most people do after that is to forget about it. Printing out stacktraces usually is not advised, hackers can find out what your application does in the back end, and sometimes if you are not careful, SQL errors are printed out and the hackers can devise an attack based on SQL Injection.

Do Nothing!

public void readAFile(){
    try{
        //Some code that tries to open and read a file
    }catch(IOException e){
       //Do nothing;
    }
}

And then we are left scratching our heads what went wrong with the program because nothing was print out when there was an error! These are they type of errors that can make you stay up late in the office, because you simply can't see what went wrong from the UI, console or the logs.

So what should you do?

Rethrow the exception if the class can't handle it.

It is likely that your method might not be the best place to handle the exception, so you rethrow to the calling method to handle it.

Example: The FileLoader class tries to readAFile but things do go wrong when reading a file, but the FileLoader is probably not the best place to handle the error. The calling class and method would like to know when there is an error so that they can either do some remedy action, inform the user or log it. In this case SomeUIClass is trying to read the file and when FileLoader throws the exception, a messagebox comes out and logs the error to the log file.

  public class FileLoader{  
      public void readAFile(File f) throws IOException{  
           //file reading code  
      }  
 }  
 public class SomeUIClass{  
      public void readFile(File f){  
           try{  
                FileLoader.readAFile(f);  
           }catch(IOExecption e){  
                //Print some message to the UI to inform the UI that it failed to read a file  
                //Maybe log something to the log file  
           }  
      }  
 }  
>

Rethrow a more generic exception.


We all like to "program to an interface", but sometimes programming to an interface can be made challenging if we rethrow a checked exception. An example.

Let say we have an interface called BooksDao. It is an Data Access Object used to retrieve books. So we have an implementation called JDBCBookDao that retrive books that is stored in a relational database using SQL. So there is a chance that it will throw a SQLException when trying to retrive some data.

public interface BooksDao{  
      public List getBooks(String query);
 }  
 public class JDBCBooksDao implements BooksDao{  
      public List getBooks(String query){  
           try{  
                //Some JDBC code
           }catch(SQLException e){  

           }  
      }  
 }  

But you see what happens if there is an error, it would be quite hard to tell the call method that an error has occured. But if we rethrow the SQLException like the previous rule then we are saying that all implementations of BooksDao.getBooks needs to throw an SQLException.

public interface BooksDao{  
      public List getBooks(String query) throws SQLException;
 }  

What if we change the implementation of BooksDao to a MongoDBBooksDao ? and the MongoDB api dosen't throw an SQLException but throw a MongoDBException, we will be force to convert whatever exception that MongoDB has to an SQLException.

public class MongoDBBooksDao implments BooksDao{
 public List getBooks(String query)throws SQLException{
  try{
   //Some MongoDB api
  }catch(MondoDBException e){
   throw new SQLException(..);
  }
 }
}

The trick here is to create your own exceptions for your classes, this makes it easy to handle the different exceptions that the various underlying implementations can throw. All you need to do if just to re-throw the specific exception like SQLException to your custom exception.

public interface BooksDao{  
      public List getBooks(String query) throws DataException;
}  

 public class JDBCBooksDao implements BooksDao{  
      public List getBooks(String query) throws DataException{  
           try{  
                //Some JDBC code
           }catch(SQLException e){  
    throw new DataException(..);
           }  
      }  
 } 

public class MongoDBBooksDao implments BooksDao{
 public List getBooks(String query)throws DataException{
  try{
   //Some MongoDB api
  }catch(MondoDBException e){
   throw new DataException(..);
  }
 }
}

See it now looked much neater, what you should also consider is whether to re-throw all checked exceptions into unchecked exceptions. One limitation of Java is that if you throw a checked exception you need to declare it in the method, for unchecked you need not and it makes your interface cleaner because it does not need to depend on the exception. However you will need to make a conscious effort to try and catch the exception in your code. There are lots of debate on whether checked exceptions are better or not, i leave that up to you. My preference is that if you can throw an unchecked exception.

Hadoop World videos

Hadoop World videos can be found here  http://www.cloudera.com/resources/Hadoop+World/. Seems like I will be spending some time watching these for the next few weeks.

Links:
Hadoop project