numpy ndarray

numpy ndarray is a n dimensional array. The array is a table of elements of same type. Let's go through some basic operations that we can do with ndarray

Pandas dataframe data manipulation

Python dataframe is a tabular structure and can be think like a spreadsheet. It provides a lot of functionality to play with data at column, row or individual data point level. 

Let's look into a jupyter python notebook about how the different data can be manipulated 

Pandas Dataframe data selection

Python dataframe is a tabular structure and can be think like a spreadsheet. It provides a lot of functionality to play with data at column, row or individual data point level. 

Let's look into a jupyter python notebook about how the different data can be displayed 

Reading csv and excel files in pandas dataframe

Dataframe can be used to load the data from various files. Let's look into how we can load the data from excel and csv.

Make sure the data file that you are loading is present in the given location. In my case, I will be assuming that the data file is present at the same location where python code is.

Code to load different files

Changing Jupyter configuration

If one wants to open Jupyter notebook on a different browser than it can be done by couple of ways outlined below

The first step is to generate the config file

jupyter notebook --generate-config

This generates the config file in home directory at home directory

Initializing Pandas Dataframe

Dataframe can be initialized in multiple ways. Let's see some of the approaches below

#createing dataframe with
#list of countries, GDP and per capita
countries_df = pd.DataFrame([["Germany",4.150,50206],

                             ["India", 2.454, 7153], 

                             ["USA",18.558,57467]])
print(countries_df)

Logging in Python using dynamic log file

Sometimes we need to send logs to a log file supplied dynamically. The standard mechanism of the python is to set the file as part of configuration. I will put down a way to do this.

Let's write a python file which captures the initialization of logger.

Let the file name be init-log.py

Virtualenv in python

A decent python code needs multiple libraries to run. The usual approach is to install those libs and dependencies using pip install.  However when you ship the code to someone else, the environment details need to be provided including the dependencies with the version numbers. It becomes very cumbersome in that way to manage it.

A better way is to create a virtual environment. The virtual environment contains all the required dependencies in a local folder and the whole environment can be shipped. It can now run in any foreign environment without worrying about what is available or not. However this does makes the size of the artifacts quite large.

To create a virtual environment, follow the following command. (You can wrap it in a script)

Dictionary in Python

A dictionary is a dictionary. It's a lookup and contains key value pair. Dictionaries are mutable.

#create an empty dictionary
>>> capital = {}

@asssign elements to the dictionary
>>> capital['India'] = 'Delhi'
>>> capital['Germany'] = 'Berlin'

#print the dictionary
>>> capital
{'Germany': 'Berlin', 'India': 'Delhi'}

Tuple in Python

Tuple is like a list. Only difference is it's immutable whereas list is mutable. What it means is that one can change the elements of a list after initial assignment, however the elements of the tuple can't be changed. Tuple in that sense provides a write protection guarantee.

#Create a tuple of Country and Capital and we don't want to change it
#Note that tuple use parenthesis for definition

>>>tuple = (['India','Delhi'],['USA','Washington'],['Germany','Berlin'])

Listifying String and Strigifying List in Python

At times we want to convert a text into individual elements and to convert a list of words into texts

Let's say we want to convert the following text into a list of words. Let's have the famous Lorem ipsum text. This text is a scrambled version of De finibus bonorum et malorum, a 1st century BC Latin text by Cicero. 

Lists in Python

Python Lists are implemented as arrays. The elements of the list can have any types. Let's work with a list example to see the various ways of interacting with it.

Let's create a list first. A list can be created by putting elements in the square brackets:

# A list of famous indian comics and their debut year. List can hold mix data types
>>> comics = ["Lot Pot", 1969, "Tinkle", 1980,  "Nagraj", 1986, "Doga", 1992, "Chacha Chaudhary", 1971]

String handling in Python

A string in python can be declared simply as:

>>> first_string="Pythonidae"
>>> print first_string
Pythonidae

Pandas : Plotting a data series

A common need is to plot the data using pandas dataframe having different series data. For example, consider the following data:

Student, Class, Marks

A,       1,     56

A,       2,     67

A,       3,     89

A,       4,     76

A,       5,     76

B,       1,     78

B,       2,     99

B,       3,     75

B,       4,     44

B,       5,     77

Python finding the encoding of a file

chardet is a universal character encoding detector in python. It can find the encoding of a file also provides a confidence score of the encoding. To have chardet in your environment, install chardet with your package manager. With pip it would look like

pip install chardet

Airflow - Beginners Tutorial

Airflow is a workflow engine from Airbnb. Airbnb developed it for its internal use and had recently open sourced it. In Airflow, the workflow is defined programmatically. Airflow document says that it's more maintainable to build workflows in this way, however I would leave it to the judgement of everyone. A developer, anyway would love anything programmatic. :) Airflow comes with a UI also and I can say that the UI is very clean and impressive. 

The main concept of airflow is a DAG (Directed Acyclic Graph). Yes, it's the same graph that you have seen in Maths, if you have seen it. A DAG contains vertices and directed edges. In a DAG, you can never reach to the same vertex, at which you have started, following the directed edges. Otherwise your workflow can get into an infinite loop. In workflow context, tasks can be defined as vertex and the sequence is represented with the directed edge. The sequence decides the order in which the tasks will be performed.

Python - Taking input from user

Taking input from user is a very common task and all programming languages support this. In this post, we will look into how we can do this in Python. A very simple program

#!/usr/bin/python

# Take the input entered by user in inputString

inputString = raw_input('Please Enter your name: ')

#print the string back

print('Welcome ' + inputString)

Simple is beautiful

Python reading from config file

Many times we need to read properties from config file. In python, you can define properties with sections. Each section can handle a related group of properties. Let's define a sample properties file and we will read it in Python.

Let's say we have a file which contains detail of countries and details of it's game. Let's call it game.properties file

game.properties

[COUNTRY]

name=India

continent=Asia

[GAME]

national=Hockey

popular=Cricket

Python Boto - Creating EC2 Instance

Python Boto is a powerful python module to interact with AWS environment. In this we will look into a simple way to create an instance.

#!/usr/bin/python

import boto.ec2

# Create a connection to a region. For example us-west-1

# Check in your AWS console or with your AWS administrator

ec2conn = boto.ec2.connect_to_region(<aws-region>)

Spur SSH Client in Python

Spur is a SSH client which can be used to invoke commands on a remote machine. This is similar to Paramiko.

For spur, install the module first

pip install spur

A simple spur client looks as follows:

#!/usr/bin/python

import sput

# Password can be provided in place of ssh key

shell=spur.SshShell(<address of remote machine,

                    <valid user in remote machine>, 

                    private_key_file=<Private SSH key file>,

                    missing_host_key=spur.ssh.MissingHostKey.accept)

# Make a dir in remote machine

with shell:

        shell.run(["mkdir","temp"])