Why use standards and which ones?

                An entity without standards is chaotic.  I’ve seen, experienced and even generated my own projects that had no standards (back in the day).   From all these, I’ve seen some common traits:

1)  If the coding takes long, like more than 16 hours, it starts to get complicated to work through and keep things in the bit of architecture you originally intended.

2)  You find that as you move along, you need/want to rebuild parts of it to simplify them or just because that part doesn’t make sense to you anymore. 

3)  After leaving the project and coming back, or entering a new project from someone else built the same way, you tend to think it would be easier to rebuild the whole thing, than work with the existing code. 

4)  You lose track of the meaning behind certain methods, classes, variables and processes, and when you look at it, you don’t understand how  it seemed so clear at the time. 
5) Bugs start showing up all over the place. 

6)  You start applying little hacks here and there that keeps things working in very specific conditions.

7)  Fixing a bug usually produces two more, 1 you get to see right away, and hides out for a little while. 

8)  If any timeline was generated, it has been exceeded.

9)  If the project isn’t very important, the developer will drop it.

10)  Developers get sick more often due to stress.

11)  It is fun to start, but after a short time, the fun is gone.

The list goes on…

                If you find your code, team or other’s suffering from the symptoms, it’s a pretty good sign there is a lack of a standard (at least a good one).  I’d also like to point out that a programmer without standards does not make them a bad programmer, but it does make their code more susceptible to the symptoms above.  Just like OOP is a step over the basics, having solid standards is a step above that. 

                So I’ve listed some of the bad things that happen.  Now on to the solution. 

                When it comes to standards, I’ve got a saying.  There are many right ways, and many wrong ways, but only one right way.  :D  (it makes sense in a moment)  The point is that while there are many good standards to follow, you have to choose one and stick to it.  Standards are no good if they are not standard.  2 weeks ago, I posted a way to document standard processes.  (here)  Now I’m going to give you some ways to find and keep your standards in specific. 

                The first thing is to know why you follow something.  There are many outdated standards, which are outdated for good reason.  The second is to be open that your own style might not be the best.  (This is something I consider all the time.)  Hungarian Notation was pretty big for a while, until they realized (not everyone knows this) that by following Hungarian Notation, you are actually reducing the implications that should be used through vocabulary.  Lets take a look at 2 variables “_adAmount” and “prices”.  Looking at the Hungarian Notation on the right, which is using an “_” to show it is a private variable, “a” to indicate it is an array, “d” to indicate it is of doubles, and “Amount” to give it a name.  Now, following some more modern standards, we see “prices” which is lacking all the symbols and details of “_adAmount”.  It is lowercase, indicating it is private.  Price indicates a decimal is needed and also implies the accuracy needed of a double.  The fact that it is plural implies an array of some sort.  Both of these followed a standard.  Hungarian got away with “Amount” though.  Amount is questionable, amount of what?  Amount of time since the last loop?  Amount of apples?  There is nothing in Hungarian Notation standards that sets up the implications of its meaning. 

                Sure, you could have chosen “_adPrices” as your variable name, but what standard are you pushing that requires that?  Another developer may have entered this and just used “I” or “x”.  The point is that the word is often chosen because the developer needs a “notch” to refer to for a variable.  If it’s only for a single method, they often use smaller less obvious terms.  While these work great for the quick entering of data, in order to figure out what it is, you have to take a look through the code to see how it is used.  I recommend keeping a standard where things make more sense, and are easier to read with native English. 

                Now to the key point, 1 true standard.  The point is for you to write down your standards that you will follow.  But, you also need to record the defenses with it.  (Why this is best)  And finally keep it open for critics.  Anyone who works with you, should also follow this standard, or successfully argue why to use something different.  You are not allowed to use a “because I said so” or “because I’m above you”.  The whole point of this is to make better code, not stick to some obscure ideology you grew up with, because you aren’t open to change.  Be practical.  When you write down defense, also include challenges to it, even if you keep that one.  Later on if you find a better way that solves more, then upgrade and keep it.  Don’t hold to practices that are not defendable.  You’ll find yourself hurting in the long run, and may not even realize it.

                One Standard:  All standards are defendable, or they are not standards.

                One Standard:  All standards are open to change from anyone who has to use them.

                One Standard:  Have a standard. 

                One Standard:  Keep it practical.

                One Standard:  Keep it Easy to find, Easy to read and Easy to use.

                One Standard:  KISS: Keep It Simple Stupid.

                One Standard:  Keep it flexible.

                One Standard:  Keep it living.

               

Open-Book Management

                I just finished reading a book called Open-Book Management, by John Case, which I originally figured I would read the first chapter and decide I already had a lot of the ideas that the book would have to promote.  I was wrong.  When I first selected it, I saw it as keeping open with employees, and keeping a clear goal and vision.  While that is important, this book is actually about Money. 

                It taught me several things about how to keep employees driven at all areas.  I’m about to describe part of the book, but by all means, not even close to covering it.  If you find any of this helpful, GO BUY THE BOOK.  You won’t regret it.  If you are trying to exist as part of a company, run one or are starting your own, you have a responsibility to make that company successful.   Especially if you are running it. 

                While there are many who want their companies to be successful, there are few who actually make companies that want to be successful.  The difference is small, but very clear.  Think of any random gas station, or fast food place.  At the corporate level, you have lots of people who are running around trying to make the company succeed, and exceed in its industry.  Now imagine the cashier, or the person who made your sandwich.  What do they care if the company is thriving or not.  If they only get 25 people in that day, instead of 500, they’ll be happy, because its less work for them.  They get paid the same whether they serve 10 or 100 people an hour. 

                Now think of the customer.  Does the customer care one bit about how much the corporate layer’s are doing to try to keep the company afloat?  No.  They care about the service they get.  They care about the quality the sandwich was made.  They care about the speedy handling, and the accuracy of their order.  Corporate can do all they want to “streamline” and provide the fastest method to deliver the service, but it really is the employees that give the customer their clear feelings regarding the quality of the company.

                So what can you do?  How can you make all the different people, EVERYONE that is part of your company, interested in making the company succeed?  Open-Book Management starts with opening the books, the financial books.  Exposing the numbers for all to see.  Then training the different areas to be able to clearly understand how certain parts of it affects them.  As well as starting some kind of profit sharing. 

                There is no part of any company, that is there without having some effect on the bottom line of the finances.  If you can show them how they have a direct impact on the success of the company, and give them goals to improve their area, and if their area improves, then they can get bonuses from it, suddenly it is not just a job, but a goal.  The average employee will see that they can have a real effect.

                If the person doesn’t care how the company does, no more than guaranteeing their paycheck anyway, it is only a job.  If they want more jobs, there is a section for that in the news paper, online services dedicated to finding new ones and networking to find more.  But if your company provides them with a strong feeling of connection with the success and failure of the company, then you’ll be able to retain employees longer, there will be more open communication, more areas will be found that can improve.

                There is another key to this Open Book Management though, empowerment.  You need to give people the power to actually make change, at least in their own areas.  You need to keep open communication between all layers of management about ideas that could improve the company.  Honestly, who is going to know better about how to save money at the customer service level, the employee who is working it every day, or the VP sitting 3, or more, layers up? 

                The higher a person is on the management ladder, the more power they should have at shaping the company as a whole, and primarily deal with keeping communication up with those that report to them.  It is more often that management thinks that the lower levels of the company need to be directed, told what to do, when more often than not, they just need the initial training, and someone to help keep their obstacles cleared. 

                If you get the chance, I recommend reading it, as the author explains through many stories, how this can be applied, and what issues people went through to get there.

Documenting Processes...

To my readers, I have spent a long time refining my own processes, and now I’m trying to come up with a standard way to document it.  I’m going to suggest a standard documentation practice, and ask for insight and considerations in the comments.  Like any process, or method I follow, I’m open to criticism and suggestions.  Things work best when more than 1 mind is involved.  Too many things in this world are of sheep.  Once something is created, all the sheep follow.  Don’t be a sheep, comment on this.  :D

I recommend using a Wiki, and that each part of the process be its own document.  Now, I’ve worked for lots of different companies as a contractor, and have seen many good and bad practices.  Ultimately it comes down to usability.  If the people can’t or don’t use it, it won’t matter.  All documents should have 3 directives, if and when possible.

1)      Keep things clear. 

a.       Do you get the point of it?

b.      Is it confusing? 

2)      KISS (Keep It Simple Stupid)

a.       The more specific something is, the less it can be applied, and therefore, the less it will be applied/retained.

3)      Keep it short.

a.       I’ve worked at a company that there entire process was clearly defined, with around 8000 documents.  Each document had an average of 5 pages of explanation, and it often took 10 minutes of reading to understand the basics of it.

b.      Take your most vital and basic explanations and put that at the top.  Then, if more space is needed, Re-iterate at the bottom.  Most people don’t need to know the details.

c.       This does not mean to cut out necessary information, but simply to allow the fastest use of it, and more info when needed.

My general structure would follow this means:

1)      Include the documenting style

a.       If the style changes over time, this will let you know how this was intended to be displayed, for past editions.

b.      This is a reference to an explanation document that follows the same format that it describes.

2)      Give it a short/clear sentence for a title.  It should include key words.  Less than 80 characters if possible.

a.       This is the document title for links, or the file system, as well as a headline/title for the page.

3)      List the exact steps that are the same as often as possible.

a.       This should be bulleted and short, anywhere from 1 sentence to 8.  If it is more, consider whether it should be split into more process documents.

4)      List Prerequisites

a.       This should either be 1 line explanation of what was needed

b.      Or it should be a link to another process, that always needs to precede this.

5)      List Follow Ups, which is just like the Prereqs, except that these are things to do next.

6)      Optional references. 

a.       This is where links go that you do sometimes.  List the most obvious condition(s)

7)      Detailed statement of why this is being followed.

a.       This should be a bulleted list,

b.      Each Item should include sub items that either defend or challenge it. 

c.       This should be open to being challenged by anyone,

d.      Challenges should not be removed.

                                                                           i.      They should include the reason they are not following this.

e.      If need be, this section should become its own document, if there is too much explanation.

I like the idea that all reasons are defendable, and if we *have* to support something that we challenge, we should know why.  All politics should be stripped from this.  Only fact, and purpose.

Another thing about this is that for some projects, a area of the general process documents might not apply, and may cause undue stresses and difficulties.  In those cases, the project should include a document outlining the needs for a change, including the process (in the same format) of what they are changing and following the same model for giving supporting reason’s and challenges with it. 

Finally, This needs to be flexible, and ANYONE who works with this process NEEDS to have the right to post challenges.  Things change, better ways might be found, and this allows the processes to grow over time.

To recap: This is an initial draft for how to document.  It follows many standards that I have seen succeed, particularly in the areas it was best suited.  However, I am one man, and one island.  No general process should be made by me, or anyone, alone.  

By all means, post any ideas or concerns.  :D

Data Sources

                When storing data, there are typically 3 major choices, Databases, XML and custom Binary Files.  I’ll be discussing several pros and cons to each method.  I will also be discussing a 4^th option that has been recently emerging, involving living network data.

                Before emerging into any as the end all be all solution for your project, you should look into the values and strengths of each option.  Weigh them carefully, once you select a route, it is often very difficult to change it later.

                Databases are very good at storing lots of information, and being able to pull it out very quickly as well.  It is very common and simple for most database system to hold on to millions of records of data and search, sort and update very quickly.  Most databases follow a simple language called SQL for retrieving, inserting and editing data, which (when taught well) is not difficult to learn.  Databases can also be expanded to additional machines to help improve overall performance in larger systems.

                Databases tend to lock the format of your data down.  It becomes difficult and time consuming to continually change and support databases as they grow.  Typically this produces a decent amount of administrative work.  Data storage be wasteful. (if 5% of your users use 3k for a description field, but the other 95% only use 0.5k,  Every row still requires 3K, whether used or not)

                XML is very flexible to data formats.  If you are constantly updating and changing data, XML can provide a solid system to work with it.  If you format your changes well, saved files can be forward and backward compatible with different versions of your application, without needing to write additional code to support it.  XML is humanly readable, which means that a notepad can easily edit and read the data for you.  Not only is it human readable, but it is descriptive, so just by looking at the XML file, you can determine what and how it is used, making it easy to edit files from programs you didn’t write.  XML is also easily transferred over the Internet, saved locally on any machine and is supported quite well on Every reasonable OS and programming language.

                XML is not intended for large amounts of data.  When reading it, it all has to be parsed and stored in memory.  Because of this constraint, it is often not easy or fast to conduct searches of XML.  XML is stored as plain text, which also means it takes up more space.  Since Lengths can be anything, your program has to be written to expect any length.  Also, data types like numbers and Boolean values are stored as strings, so they have to be specifically converted into numbers.  XML is also a poor choice when storing and loading binary data, like images, sound files or movies.

                Custom Binary files have been used for a long time.  This means storing the data (typically) in the smallest amount of space it possibly needs.   It also means that it is harder to use outside of your own applications, which is often helpful for protecting local data.  You control entirely control the format, and can decide how best to shape it, making portions of it similarly searchable like a database, and other portions flexible like XML.  When storing large amounts of data that will only be used by your application, this is a common practice to use.

                Custom Binary files lock you into a format.  Any changes you make have tend to require a decent amount of code to transfer from one version to another.  They are difficult to read, and if bugs are caused by misplaced data, or some accident in writing the data, the entire file can be ruined.  It does not allow your data to be easily read by other systems, which could be negative depending on your user’s needs and interests.   They also require more design time because you are not only deciding the format of the data, but also writing all the tools and code for handling the data.

                A fourth option I referred to as “Living Networks” is demonstrated quite well by a system called “Terracotta”.  The reason I selected a particular brand name for this, is because it is new (and old, as similar concepts have been around for a while) system, that I have not seen a valid competitor for yet.  In this system, it manages memory over multiple networked servers.  It has proven faster than the best databases when the data is designed for it.  It is also flexible like XML, and also provides (by its nature) redundancy and load balanced data access.  It is a system that I look forward to in the near future becoming a new standard for data storage and retrieval.

                Terracotta only supports Java, leaving C++, C#, VB, PHP and other languages out.  Because of that lack of interoperability, when selecting this, you are forcing your data to be accessed through JAVA/J2EE alone, meaning your data has now limited your application choices, not very sustainable in mixed environments.    It is not intended for home computing and should not be considered a solution for local data. 

                Each of these 4 data solutions target different audiences.  Even within these systems, we are not very technically mature.  It should be noted that every one of these areas is coming out with new and better ways of storing information, (XML being an exception) and will continue to do so for a while.  While one solution might be the most optimal at the moment, improvements in other environments may make them your optimal choice a year later.