Pandas(a Python package)

Pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. It is already well on its way toward this goal.

pandas is well suited for many different kinds of data:

• Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet
• Ordered and unordered (not necessarily fixed-frequency) time series data.
• Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels
• Any other form of observational / statistical data sets. The data actually need not be labeled at all to be placed into a pandas data structure

The two primary data structures of pandas, Series (1-dimensional) and DataFrame (2-dimensional), handle the vast majority of typical use cases in finance, statistics, social science, and many areas of engineering. For R users,DataFrame provides everything that R’s data.frame provides and much more. pandas is built on top of NumPy and is intended to integrate well within a scientific computing environment with many other 3rd party libraries.

Here are just a few of the things that pandas does well:

• Easy handling of missing data (represented as NaN) in floating point as well as non-floating point data
• Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects
• Automatic and explicit data alignment: objects can be explicitly aligned to a set of labels, or the user can simply ignore the labels and let Series, DataFrame, etc. automatically align the data for you in computations
• Powerful, flexible group by functionality to perform split-apply-combine operations on data sets, for both aggregating and transforming data
• Make it easy to convert ragged, differently-indexed data in other Python and NumPy data structures into DataFrame objects
• Intelligent label-based slicing, fancy indexing, and subsetting of large data sets
• Intuitive merging and joining data sets
• Flexible reshaping and pivoting of data sets
• Hierarchical labeling of axes (possible to have multiple labels per tick)
• Robust IO tools for loading data from flat files (CSV and delimited), Excel files, databases, and saving / loading data from the ultrafast HDF5 format
• Time series-specific functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging, etc.

2018 - shortage of up to 190,000 workers with analytical skills !!!!

“By 2018, the US could face a shortage of up to 190,000 workers with analytical skills” McKinsey Global Institute

Who is a Data Scientist ????

“A data scientist is someone who knows more statistics than a computer scientist and more computer science than a statistician.” - Josh Blumenstock 


“Data Scientist = statistician + programmer + coach + storyteller + artist” - Shlomo Aragmon

Biswajit Das Definition of a Data Scientist is as follows >>>

Data Scientist = Statistician + Analytical Programmer + Storyteller + Some Domain Expertise + Creative ”

This MIT course was one of the many Machine Learning materials I went thru. to know in and out of ML ....

Machine Learning Course Description

This introductory course gives an overview of many concepts, techniques, and algorithms in machine learning, beginning with topics such as classification and linear regression and ending up with more recent topics such as boosting, support vector machines, hidden Markov models, and Bayesian networks. The course will give the student the basic ideas and intuition behind modern machine learning methods as well as a bit more formal understanding of how, why, and when they work. The underlying theme in the course is statistical inference as it provides the foundation for most of the methods covered.

http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-867-machine-learning-fall-2006/lecture-notes/

LEC #	TOPICS
1	Introduction, linear classification, perceptron update rule (PDF)
2	Perceptron convergence, generalization (PDF)
3	Maximum margin classification (PDF)
4	Classification errors, regularization, logistic regression (PDF)
5	Linear regression, estimator bias and variance, active learning (PDF)
6	Active learning (cont.), non-linear predictions, kernals (PDF)
7	Kernal regression, kernels (PDF)
8	Support vector machine (SVM) and kernels, kernel optimization (PDF)
9	Model selection (PDF)
10	Model selection criteria (PDF)
11	Description length, feature selection (PDF)
12	Combining classifiers, boosting (PDF)
13	Boosting, margin, and complexity (PDF)
14	Margin and generalization, mixture models (PDF)
15	Mixtures and the expectation maximization (EM) algorithm (PDF)
16	EM, regularization, clustering (PDF)
17	Clustering (PDF)
18	Spectral clustering, Markov models (PDF)
19	Hidden Markov models (HMMs) (PDF)
20	HMMs (cont.) (PDF)
21	Bayesian networks (PDF)
22	Learning Bayesian networks (PDF)
23	Probabilistic inference Guest lecture on collaborative filtering (PDF)

Big Data Processing with Spark

Big Data Processing with Apache Spark – Part 1: Introduction

https://www.infoq.com/articles/apache-spark-introduction

Big Data Processing with Apache Spark - Part 2: Spark SQL

https://www.infoq.com/articles/apache-spark-sql

Big Data Processing with Apache Spark - Part 3: Spark Streaming

https://www.infoq.com/articles/apache-spark-streaming

Big Data Processing with Apache Spark - Part 4: Spark Machine Learning

https://www.infoq.com/articles/apache-spark-streaming

Spark and Hadoop’s combination for improving storage and processing scalability

Spark and Hadoop’s combination is a key solution to address many organizational challenges. Firstly, improving storage and processing scalability which can help to cut costs by 20-40% while simultaneously adding high volumes of data.

Secondly, unifying separate clusters into one that supports both Spark and Hadoop. Finally, with only a retrospective view of data, companies have limited predictive capabilities, hampering Big Data’s strategic value to anticipate emerging market trends and customer needs. Spark helps to process billions of events per day at a blistering analytical pace of 40 milliseconds per event. Through tackling these issues with Spark and Hadoop there is a huge potential of benefits for companies!

See the below article to find out why you still need Hadoop with Spark :- 

http://www.forbes.com/sites/bernardmarr/2015/06/22/spark-or-hadoop-which-is-the-best-big-data-framework/#6a17e623532c

Scala - Sacalable Language

A Scalable language

Scala is an acronym for “Scalable Language”. This means that Scala grows with you. You can play with it by typing one-line expressions and observing the results. But you can also rely on it for large mission critical systems, as many companies, including Twitter, LinkedIn, or Intel do.

To some, Scala feels like a scripting language. Its syntax is concise and low ceremony; its types get out of the way because the compiler can infer them. There’s a REPL and IDE worksheets for quick feedback. Developers like it so much that Scala won the ScriptBowl contest at the 2012 JavaOne conference.

At the same time, Scala is the preferred workhorse language for many mission critical server systems. The generated code is on a par with Java’s and its precise typing means that many problems are caught at compile-time rather than after deployment.

At the root, the language’s scalability is the result of a careful integration of object-oriented and functional language concepts.

Object-Oriented

Scala is a pure-bred object-oriented language. Conceptually, every value is an object and every operation is a method-call. The language supports advanced component architectures through classes and traits.

Many traditional design patterns in other languages are already naively supported. For instance, singletons are supported through object definitions and visitors are supported through pattern matching. Using implicit classes, Scala even allows you to add new operations to existing classes, no matter whether they come from Scala or Java!

Functional

Even though its syntax is fairly conventional, Scala is also a full-blown functional language. It has everything you would expect, including first-class functions, a library with efficient immutable data structures, and a general preference of immutability over mutation.

Unlike with many traditional functional languages, Scala allows a gradual, easy migration to a more functional style. You can start to use it as a “Java without semicolons”. Over time, you can progress to gradually eliminate mutable state in your applications, phasing in safe functional composition patterns instead. As Scala programmers we believe that this progression is often a good idea. At the same time, Scala is not opinionated; you can use it with any style you prefer.

Seamless Java Interop

Scala runs on the JVM. Java and Scala classes can be freely mixed, no matter whether they reside in different projects or in the same. They can even mutually refer to each other, the Scala compiler contains a subset of a Java compiler to make sense of such recursive dependencies.

Java libraries, frameworks and tools are all available. Build tools like ant or maven, IDEs like Eclipse, IntelliJ, or Netbeans, frameworks like Spring or Hibernate all work seamlessly with Scala. Scala runs on all common JVMs and also on Android.

The Scala community is an important part of the Java ecosystem. Popular Scala frameworks, including Akka, Finagle, and the Play web framework include dual APIs for Java and Scala.

Functions are Objects

Scala’s approach is to develop a small set of core constructs that can be combined in flexible ways. This applies also to its object-oriented and functional natures. Features from both sides are unified to a degree where Functional and Object-oriented can be seen as two sides of the same coin.

Some examples: Functions in Scala are objects. The function type is just a regular class. The algebraic data types found in languages such as Haskell, F# or ML are modelled in Scala as class hierarchies. Pattern matching is possible over arbitrary classes.

Future-Proof

Scala particularly shines when it comes to scalable server software that makes use of concurrent and synchronous processing, parallel utilization of multiple cores, and distributed processing in the cloud.

Its functional nature makes it easier to write safe and performant multi-threaded code. There’s typically less reliance on mutable state and Scala’s futures and actors provide powerful tools for organizing concurrent system at a high-level of abstraction.

Fun

Maybe most important is that programming in Scala tends to be very enjoyable. No boilerplate, rapid iteration, but at the same time the safety of a strong static type system.