Benefits and capabilities of Data Lake

We know that data is the business asset for any organisation which always keeps secure and accessible to business users whenever it required. 

Data Lake is a storage repository that holds a vast amount of raw data in its native format, including structured, semi-structured, and unstructured data. The data structure and requirements are not defined until the data is needed. We can say that Data Lake is a more organic store of data without regard for the perceived value or structure of the data.

The data lake is essential for any organization who wants to take full advantage of its data. The data lake arose because new types of data needed to be captured and exploited by the enterprise. As this data became increasingly available, early adopters discovered that they could extract insight through new applications built to serve the business.

Benefits and capabilities of Data Lake - It supports the following capabilities:

• To capture and store raw data at scale for a low cost – i.e. The Hadoop-based data lake
• To store many types of data in the same repository – data lake store the data as-is and support structured, semi-structured, and unstructured data
• To perform transformations on the data
• To define the structure of the data at the time, it is used, referred to as schema on reading
• To perform new types of data processing
• To perform single subject analytics based on particular use cases
• To catch all phrase for anything that does not fit into the traditional data warehouse architecture
• To be accessed by users without technical and/or data analytics skills is ludicrous

Silent Points of Data Lake - It is containing some of the salient points as given below:

1. A Data Lake stores data in 'near exact original format' and by itself does not provide data integration
2. Data Lakes need to bring ALL data (including the relevant relational data)
3. A Data Lake becomes meaningful ONLY when it comes a Data Integration Platform

4. A Data Integration Platform (Meaningful Data Lake) requires the following 4 major components: An ingestion layer, A multi-modal NoSQL database for data persistence, Transformation Code (Cleanse, Massage & Harmonize data) and A Hadoop Cluster for (generate batch and real-time analytics)
5. The goal of this architecture is to use 'the right technology solution for the right problem'
6. This architecture utilizes the foundation data management principle of ELT not ETL. In fact T is continuous (T to the power of n). Transformation (change) is continuous in every aspect of any thriving business and Data Integration Platforms (Meaningful Data Lakes) need to support that.
7. So the process is as follows:

• 1) Ingest ALL data
• 2) Persist in a scalable multi-model NoSQL database  - RawDB
• 3) Transform the data continuously - CleanDB
• 4) Transport 'clean data' to Hadoop to generate Analytics
• 5) Persist the 'Analytics' back in the NoSQL database - AnalyticsDB
• 6) Expose the databases using REST endpoints
• 7) Consume the data via applications

Azure - How to create Azure SQL Database

SSIS - How to Call Multiple Child Packages by Master Package

In our day to day activities, we can build a data extraction module that can be called from different packages. 

For example, we have to load the data into a star schema, and we can build a separate package to populate each dimension and the fact table and these packages are located in some folder. They should be executed in certain order one by one.

Now, we want to create one master SSIS package that will go to that folder and grab those child packages and execute or run them one by one (no repeat) in a predefined order.

To accomplished this, You could build a Foreach Loop Container with a Package Execute Task in it to execute them, then we would need to name them so that they were retrieved in the order we wanted. 

With the help of variables, we can set the package name property in Package Execute Task and this variable will get the next value from the Foreach loop Container.

Foreach Loop Container can get these package names from a data table that contained the order we wanted and then we populate an SSIS object type variable with the record-set and use that to feed the order and the list to a Foreach loop just like we mentioned above. The only difference would be the source of the list.

The below video is capable to explain that How can we call multiple child packages by parent or master package?

SSIS - Validate Processing Flat File Format || Check Columns Name and C...

SSIS - Script Task Check if Sub-folder || Directory Exists or Not

In this tutorial, we are going to explain the functionalities of Script Task to Check if Subfolder Exists in SQL Server Integration Services. The Script task provides code to perform functions that are not available in the built-in tasks and transformations that SQL Server Integration Services provides. The Script task can also combine functions in one script instead of using multiple tasks and transformations.
For more information, please visit us at http://www.sql-datatools.com/2016/04/ssis-object-types-variable-in-execute.html

SSIS - Tracking Object Type Variables || Foreach Loop Container

In this tutorial, we are going to explain the functionalities of Object Type Variable or Custom Variables in Foreach Loop Container in SQL Server Integration Services. Variables are extremely important and are widely used in an SSIS package. A variable is a named object that stores one or more values and can be referenced by various SSIS components throughout the package’s execution.
For more information, please visit us at http://www.sql-datatools.com/2016/04/ssis-object-types-variable-in-execute.html

SSIS - How to work with Foreach Loop Container || Basics of Foreach Loop...

In this tutorial, we are going to explain the functionalities of Object Type Variable or Custom Variables in Foreach Loop Container in SQL Server Integration Services. Variables are extremely important and are widely used in an SSIS package. A variable is a named object that stores one or more values and can be referenced by various SSIS components throughout the package’s execution.
For more information, please visit us at http://www.sql-datatools.com/2016/04/ssis-object-types-variable-in-execute.html

SSRS - How to add a Polar or Radar Chart

SSRS - How to Add a Pie Chart

SSRS - How to add Spark-lines in your report

In this session, we are going to show you how to add sparklines in your report. Sparklines are small, simple charts that convey a lot of information in a little space, often inline with text. Their impact comes from viewing many of them together and being able to quickly compare them one above the other, rather than viewing them singly.
Because sparklines display aggregated data, they must go in a cell associated with a group. Sparklines and data bars have the same basic chart elements of categories, series, and values, but they have no legend, axis lines, labels, or tick marks.

SSRS - How to Create Line Chart Report

A line chart displays a series as a set of points connected by a single line. Line charts are used to representing large amounts of data that occur over a continuous period of time.

Life is about taking chances, risks and always moving forward

No one can predict where a person will be in the near or distant future. For life is about taking chances, risks and always moving forward. The plan we may have now could change the next day or  in the next few months or may be just a bad one after all. Don't get fooled, people entering or exiting other people's lives have a huge impact. Especially in a working environment. Asking someone about the next five years, to me, screams loudly like the five year communist idealistic phoney perspective for the next financial political congress. Funny right? How things resonate in other people's minds, huh?

This so important topic should be discussed more widely, All everyone needs to do is to adjust to the conditions of their own expectations and lived experiences and to envision a life/career destination. Taking and assuming all the necessary steps through the process is the key to all achievements. Being true to their own personal values and agenda will set them out of the crowd. 

A high culture atmosphere, perfect planning, positive climate. During employment, assessors looked how we communicate with each other and do we care of all in the hall during breaks. No typical questions, everything orientated to customer service and personality. Attitude is maybe not everything, but % really is. The rest is personality.

Job interview questions - Prepare Yourself

Talent and integrity is what needed in people you hire. Steve jobs said it best, we don't want to hire smart people to tell them what to do,we want smart people to tell us what to do? They’re too ambiguous and open ended, their answers won’t reveal too much about the candidates as they are the kind of questions in which a candidate can prepare and create answers that you’d want to hear.

Questions should be:

1. What have you done in the last 5 years?
2. What did you make?
3. Who were the parties involved? 
4. What was the product?
5. How long did it take?
6. How did you measure the success of the project or product?
7. Did you fail and what did you learn?
8. Would you do it all over again?

All standard interview questions should be replaced. Everyone can buy a sexy resume and can pay a career coach to ace the interview. Anyone can pay to find info and prepare for standard interview questions but you cant pay to answer questions that are designed to reveal ones talent, integrity, intelligence, dedication commitment and self worth.

The worst part is that we need to prepare the answers ahead to beat people that look up the answers on the internet to show a quality that they do not have. But at the end of the day, if we try to escape them, it will somehow imply that we "do not care" and came "unprepared" for the interview, regardless how great we are on our jobs. We need to prove with ineffective questions how good we are in words not in actions.

The Most Annoying Interview Questions

If we want to hire smart people, we should stop asking stupid questions. They ask the questions typically because officially once you walk in the door then they have decided if they like you or not or if they wish to pursue your candidate status. 

In case you have not noticed, good people hate these following questions in their interview scanning process- 

1. Why are you leaving your current company?

2. Where do you see yourself in 5 years ?

3. Why should we hire you ?

4. What are your weaknesses ?

People leave their employer due to poor management and lack of involvement in the growth of the employees. How many people leave in 1 year time because of this? People are mislead with broken promises 50% of the time. Take your applicant and turn the tables. Bring in 3 employees and CEO,VP higher management. Let the applicant do the  interview. If the applicant is well educated will know to run it like a strategic meeting.

A company will hopefully realise there is a problem and correct it. Turn around of employees will slow down, and they will be doing interviews and hiring due to growth. I mean, a candidate's smartness and his hard working nature is measured by such stupid questions on his first visit to the company, how would he happily join that company or join with full satisfaction and then later on do his job and how could he enjoy the job he is doing?

Even, I don't like these kind of questions. 

Q1. There are 101 reasons why we leave a company. And the common reason for everyone is for better pay check. I am sure what recruiters will make out of this question. Also, nobody will say a genuine reason. 

Q2. It all depends of the organization structure and the open positions in a company if you would like to grow and many other factors. 

Q3. Why did you open a position to recruit people or fill a position. That is a reason to hire me

Q4. Why do you want to know the weakness. Somebody's weakness maybe someone's strength. N I am 100 % sure no one will tell you their genuine weakness even if they have one.

Ask relevant questions related to the open position and questions which get's most out of the candidate whom you are interviewing. Like his attitude, how best he can get along with the colleagues, His Ideas, His Future Goals.

Nowadays people round very quickly and companies are also not too keen in retaining talents like they used too. Also people used to stay put for a long term benefit packages which has diapered , so the question is totally irrelevant !! 

It's not irrelevant if you're a company deciding whether or not to invest thousands of dollars and who knows how much time training someone. You'd be surprised by how many people excitedly tell you about the business that they're less than a year away from launching or that they plan on moving to Korea to teach English.  You wouldn't be a good steward of your organization's resources if you didn't at least attempt to inquire about a candidate's short term plans.

Trendiness needs a reality check.  Anyway, good to see some thought is being centred on interview questions and their relevance. How unjust it is for your future, your career, your ambitions, your dreams and aspirations and chance to success depend one that person who possibly woke up on the wrong side or feels threatened or is not even best qualified to assess your candidacy.

SSRS - How to add Grouping and Totals in SSRS

.NET: How to check if a SQL Server Agent job is running

.NET: How to check if a SQL Server Agent job is running: I was getting an error that "SQLServerAgent Error: Request to run job <Job name>  from User <User name>  refused because t...

SQL - Basic Interview Questions

What is Stored Procedure?

Stored procedures are nothing but a stored procedure is a named group of SQL statements that have been previously created and stored in the server database. Stored procedures accept input parameters so that a single procedure can be used over the network by several clients using different input data. And when the procedure is modified, all clients automatically get the new version. Stored procedures reduce network traffic and improve performance. Stored procedures can be used to help ensure the integrity of the database.

e.g.  sp_helpdb, sp_renamedb, sp_depends etc.

What is Trigger?

A trigger is a SQL procedure that initiates an action when an event (INSERT, DELETE or UPDATE) occurs. Triggers are stored in and managed by the DBMS. Triggers are used to maintain the referential integrity of data by changing the data in a systematic fashion. A trigger cannot be called or executed; DBMS automatically fires the trigger as a result of a data modification to the associated table. Triggers can be viewed as similar to stored procedures in that both consist of procedural logic that is stored at the database level. Stored procedures, however, are not event-drive and are not attached to a specific table as triggers are. Stored procedures are explicitly executed by invoking a CALL to the procedure while triggers are implicitly executed. In addition, triggers can also execute stored procedures.

Nested Trigger: A trigger can also contain INSERT, UPDATE and DELETE logic within itself, so when the trigger is fired because of data modification it can also cause another data modification, thereby firing another trigger. A trigger that contains data modification logic within itself is called a nested trigger.

What is View?

A simple view can be thought of as a subset of a table. It can be used for retrieving data, as well as updating or deleting rows. Rows updated or deleted in the view are updated or deleted in the table the view was created with. It should also be noted that as data in the original table changes, so does data in the view, as views are the way to look at part of the original table. The results of using a view are not permanently stored in the database. The data accessed through a view is actually constructed using standard T-SQL select command and can come from one to many different base tables or even other views.

What is Index?

An index is a physical structure containing pointers to the data. Indices are created in an existing table to locate rows more quickly and efficiently. It is possible to create an index on one or more columns of a table, and each index is given a name. The users cannot see the indexes; they are just used to speed up queries. Effective indexes are one of the best ways to improve performance in a database application. A table scan happens when there is no index available to help a query. In a table scan SQL Server examines every row in the table to satisfy the query results. Table scans are sometimes unavoidable, but on large tables, scans have a terrific impact on performance.

What is a Linked Server?

Linked Servers is a concept in SQL Server by which we can add other SQL Server to a Group and query both the SQL Server dbs using T-SQL Statements. With a linked server, you can create very clean, easy to follow, SQL statements that allow remote data to be retrieved, joined and combined with local data. Stored Procedure sp_addlinkedserver, sp_addlinkedsrvlogin will be used add new Linked Server.

What is Cursor?

Cursor is a database object used by applications to manipulate data in a set on a row-by-row basis, instead of the typical SQL commands that operate on all the rows in the set at one time. In order to work with a cursor we need to perform some steps in the following order:

Declare cursor

Open cursor

Fetch row from the cursor

Process fetched row

Close cursor

Deallocate cursor

VBA Vs Power BI, PowerPivot, PowerQuery, PowerView and PowerMap

VBA and the Power tools target different pieces of the pie and different use scenarios. For an analyst I would focus on the Power stuff and using VBA for production needs. 

VBA have been around for almost as long as MS Excel and in all this time VBA has created a space for itself and all its related software over many years, Software like Power BI, Power Pivot etc are all relatively new and can be seen as a high level problem solver where as with VBA, the level goes a lot lower and will most definitely involve role players at that lower level

VBA falls short in that many users are scared of it - the interface is too busy, too many moving parts and the object model takes a long time to get familiar with - it is ot particulary easy to break into VBA coding.

PowerPivot falls short in that the types of operations that it performs are somewhat more limited - the output is always a reconfigured version of what was put into it, with a few fancy calculations and metrics.

Where PowerQuery really shines is that you can take a small amount of data and make it big. Running a full fledged simulation is easy, contorting and expanding and blowing up your data is easy. A user can create reusable and global abstract functions and apply across a workbook. 

Power Query is the new swiss army knife for accountants and controllers in my eyes. Although "only meant" as a self-service ETL-tool, it's so versatile that you can extend it's use cases in so many areas.

Its main drawback compared to VBA - lack of runtime parameters, I/O, user prompts etc. is easily worked around by good design and trial and error. The programming language is relatively easy to pick up and more intuitive than VBA IMHO.

PowerQuery also easily interfaces with PowerPivot directly - it is not a either-or proposition, although I find that given how easy it s to mash-up tables in PowerQuery, I can do most of the heavy lifting that I would have used PowerPivot for directly in PowerQuery. I will concede that if I were better with PowerPivot, I might change my position, but The financial data that I work with and the projects that I have do not require the sort of fancy analysis that PowerPivot measures are better suited for. 

Powerquery is very underrated and under appreciated - much more powerful and useful in everyday tasks than VBA and even basic excel formulas.

The Power BI stuff appears to be on an unannounced track to replace SSRS. It is much better and modern. The new announcements about open source visualizations, mobile, and more will make Power BI even more competitive in that space. Take an xls combining Power BI using Sharepoint 2013+ data source and a few VBA supported easy buttons and you have a powerful tool.

If data cleansing is a big part of your work, PowerQuery should always come before PowerPivot. As a convert from VBA to PowerQuery, A huge drawback to VBA is that VBA is scary to outsiders - if you are not a programmer, you think it is like Pandora's Box; and if you are a programmer, you have likely seen to many excel "experts" that have macro-recorded themselves incredibly non-extendable and difficult to follow scripts that they cant even read their own code for.

PowerQuery is fresh and intuitive. If you are an outsider reading a query that a novice developed exclusively from the UI, you can pick it up and read it and make it your own with minimal effort.

Event Notifications vs. SQL Trace

Event Notifications convey the very same data as DDL triggers and occur on the very same events, but they are asynchronous and loosely coupled as SQL Trace. 

To know more on click on Event Notifications vs. Triggers.

SSRS - KPI Reports (Indicators and Gauges)