How to tame machine data monster?

What is Machine data?

Machine data are the data that are constantly flowing from the device that we interact. In our everyday life we received data from numerous devices and applications i.e. from the car that use to go to work, electronic coffee blending machine, Wifi connection in office and home, Building proximity sensor, web server that we connect and phone data etc.

This overall data making up the 90% data today’s in accumulation. This machine generated data does not have any specific format. In naked eyes, it would seem gobbledygook to us. This humongous amount of data may look like machine data monster to us.

Let’s assume a scenario to understand our discussion:

Jacob works in PopCandyCup games in IT operation. They have branch all over the world. They have their own machine data monster (huge amount of data generated in every hour). These data are generated from various servers across the globe of that company [web server, point of sale server, Email server, voice mail server, Security appliance server, security badge server]. These data source could give Jacob valuable information regarding company operation. But fetching the accurate one information from this machine data take hours by Jacob.machine_data_monster

Scenario 1: Without operation intelligence software:

In example, Last week, a customer calls PopCandyCup games, IT operation about having issue in purchasing in-game update for stickman vs. monster games. To fix this problem,

  • TOM who is customer support team member received the customer call. He checked and found everything is okay from their side. He forwarded the call to application support group.
  • Susan who is application support team, checked to find any application issue. But no error found. She escalates the call to developer team.
  • Eric who is a developer to find any issue regarding this. But still no error found.
  • Scott, escalate the error to administrator to find any issue regarding this. However, no error found.
  • Mark, Escalate the error to higher team to find any issue regarding this they checked [web server, point of sale server, Email server still no error.
  • Taylor, Escalate the error to Database admin team, She finally found out that slow query making it hard for some user to do the transaction. In the mean time they spent almost 11 hours and still tried to resolve the issue.

Scenario 2: With operation intelligence software: Using machine data information

  • TOM received the call, checked the users log and forwards the call to application support team.
  • Susan who is application support team member, she searched the data in Search head for events like previous purchase events and searched for error. She easily pinpoints user error and found out slow query log from database server and directly contact with database team, Taylor about slow query. And they resolved the query within 20 minutes.

Using Search head for search events

 Jacob wants to correlate, pinpoint and alert specific error before getting call from customer about any specific event from any data sources. This is where operation intelligence software or SIEM base software comes in. It simply put any data and grabs it and makes it to searchable event or index. Make unstructured data to structure format.

We can use it for security, user behavior, sales monitoring, hardware monitoring. It’s like having a translator for machine data monster.

Here I will discuss about one of the operation intelligence software or SIEM base software called Splunk enterprise. I want to discuss about it because I had the opportunity to play with it. It’s not product marketing. I am just sharing information so other can get benefited from similar kind of software.

I will discuss how to use splunk in subsequent post.