Whether to build software or to buy?

 Many organizations struggle with the question of when to build a software in-house and when to buy off the shelf commercial product. There is no easy answer to it. 

A number of criteria's come into play and need careful evaluation of each of them. From in house software product management and development capabilities to long term sustenance to the perceived complexity of the Software. 

I have compiled various levers in the video linked below. 

Intuition and System Design

What is intuition? We often hear that "This is very intuitive." Intuition in simple terms is the ability to comprehend in a natural way. Intuition is also a function of one's background and experience. Intuitive for one may not be the same for others. For example, a painter knows how with a couple of strokes and adjustments of colours he or she can change the mood of the painting. But that capability has been built through years of practice. That is why I consider intuition as a relative term and not an absolute one. Intuitive for one may not be so intuitive for others. That also leads to another interesting thing that people interpret things based on their experience and background. Confirmation bias has a role to play here but it also is a function of the tendency to put lesser mental load while comprehending a thing. We are driven by our intuition and our intuition is a function of our experience and background. 

Back of the envelop calculation

Most of you would have heard about "Back of the envelope calculation", which stands for what is a rough estimate of a certain thing. It can be the time a certain task will take or how many people or other resources will be needed to accomplish something. We do many kinds of estimation in our daily life both in professional and personal life. For example, how many years I should work and at what level so that I can get my retirement.  Unfortunately, that never happens. If you really want to retire it's today or you will be made to retire for some other reason.

Git Branching

Red dots denote commits. The model can be tuned based on the size of the team. For example, a single developer might just use one dev branch from the master and work on it and keep merging back in the master. However, it's suggested to still cut release branches.

Larger teams would want to use a more elaborate mechanism of tagging and cutting release branches. Versioning of the build can be done as per the blog.

The basic theme that should always be followed is to integrate the code as much as possible to the parent branches and maintain the stability of every code base across all branches. Pull frequently, push early and run integrations as much as needed. Have integration environments where any commit can trigger integration tests.

Migrating from DynamoDB to Postgres

The mass popularity of NoSQL databases has also resulted in it being used in all use cases without understanding the suitability for the use case. One fundamental rule that is usually forgotten is that the NoSQL databases are designed around queries. Initially, the schema is evolved based on the need for initial business use cases. However, the business use cases evolve and change in multiple ways and soon the new needs of interacting with the database become unwieldy. This is the fundamental problem that people usually hit wiht NoSQL databases.

Where DynamoDB gets into trouble:

• As the business use cases evolve and change the need to query the database in multiple ways arise. DynamoDB is not easy to query if it is not queried based on the partition key. One can build indexes but there is a cost associated with it. Filters can be used to query the data but that involves scan and as the data grows it starts becoming costly both in terms of money and time.
• DynamoDB is schemaless so with time the data evolve in multiple ways. In older data records, it is quite possible that the fields might be missing or might have a different interpretation. Developers keep handling them in the deep layers of code to keep the world moving. However, soon it results in too many if-else statements. Migration is a pain to handle such cases, however, one has to be ready for missing fields and handle them with suitable defaults.
• There is no relationship integrity so it's easy to put wrong data in relationships and it's very difficult to figure out even if something like that has happened. In SQL also it's possible to put a wrong key with a valid foreign relationship but still in terms of integrity SQL provides much better primitives.
• This will be a repetition of the above points but a different perspective. As it's fine to add any kind of data in the table, people start putting all kind of data in it. Imagine that everyone in the world is given all kind of freedom. Sounds romantic. However, soon ti will be chaos as everyone is living in all different ways. 

More discussion is available at https://tech.lalitbhatt.net/2018/09/sql-vs-nosql.html

Sample code for migration

import boto3

import psycopg2

# Create a connection to DynamoDB. Please fill the required keys. A better way to do is 

# to put it in config file and pass it through. In AWS environment it's better to use Roles

dynamo = boto3.client('dynamodb',aws_access_key_id='<access-key>', \

                      aws_secret_access_key='<access-secrte>', \

                      region_name='<region>')

# Create the database connection

db = psycopg2.connect(host="<db_host>,database="<db>", user="<user>", password="<pass>")

dbCurr = db.cursor()

#Assume we have a user table in dynamoDB and insert it into the postgres Users table

user = dynamo.query(TableName="User", KeyConditionExpression ="Email = :email", \

                               ExpressionAttributeValues = {":email": { 'S': email }})  

email = user.Items[0]['Email']

#Note the returning id so that we can use the id of the newly persisted record.

#This can be used to create foreign key relationships for further table

userSql = 'INSERT INTO users(email) VALUES(%s) returning id'

userValues=(email)

dbCurr.execute(userSql,userValues)

dbUserId = int(dbCurr.fetchone()[0]) 

db.commit()

dbCurr.close()

Some more stories from the web:

• https://dzone.com/articles/why-we-moved-from-nosql-mongodb-to-postgresql
• https://blog.codebarrel.io/why-we-switched-from-dynamodb-back-to-rds-before-we-even-released-3c2ee092120c

Learning a programming language

There are tons of programming language and new ones keep popping and old ones going out. There are some who withstand the test of time and continue to flourish. Being in the information technology industry, it's imperative to keep learning new languages and keep sharpening the skills. Learning a new language is not always easy as many times there are paradigm shifts and it takes time to grasp them. 

In this blog, I will delve on the art of learning a language and what has helped me while learning nuances of Java, Python, Javascript and couple of more. I still would not claim myself as an expert in any of this but have done a fair amount of coding in those languages and couple of more. Whenever I had to pick a new language, I try to understand the language on the following dimensions to get a good understanding of what the language offers. And be careful, the languages are not sometimes plain programs, they are ecosystems in themselves. 

AWS pricing model

Many organizations have adopted cloud services as an inherent part of their IT infrastructure.  With the cloud infrastructure, the upfront capital cost is not needed. One can hire services on a need basis. However, one still has to be careful about the cost that gets incurred as part of hosting the IT infrastructure on the cloud. Every second service is up and running, it makes the dollar meter to rotate.