cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

Data Governance and Automatic Data Lineage: Surviving Regulatory Requirements

Image Interview MEGA x MANTA_Blog.jpg
2970
0

Trace the complex journey of data with automatic data lineage

Ernie Ostic: Data lineage is above all a flow of transformation. It makes it possible to trace the technical genealogy of data by giving a precise overview of the path traveled in computer systems. This approach provides a complete view of the life cycle of the data, from its collection to its use, to its destruction. Although automated lineage is necessary for complex technology portfolios, it is also about connecting with users (businesses) beyond the technical aspects, especially during the data discovery phase and the modeling of underlying processes.

 

Frédéric Fourquet: The challenge of the lineage process is to understand what is happening in the journey of the data: where it comes from, where it goes, who collects it, who uses it, reuses it, etc. Data is not static; it integrates processes and requires an in-depth and dynamic view. The technical lineage of the data makes it possible to know what precisely happened in the system, and what treatment the data have been through since it's set up. This work allows more in-depth data knowledge through retracing its history.

 

Data governance: proving the origin and destination of data

Ernie: With the increasing number of data regulations - notably GDPR in Europe - the challenge is to prove to the regulator how each piece of data was obtained. Failure to comply with this requirement can put organizations at risk in any industry. Since data management is a long-term transformation process, it is crucial to be able to trace the history of each piece of data to track its origin, processing, etc. It is no longer just a question of providing the processed data to the legislator, it is also necessary to be able to demonstrate its lineage, i.e., the data’s genealogy in the system. Given the exponential growth in the volume of big data, automation is essential.

 

Frédéric: Modeling the data life cycle in an automated way makes it possible to avoid enormous manual efforts - efforts which might even be impossible with a certain volume of data. Automation is essential, for example, when a company has several hundred critical data items to process and only 10 to 15 data items per year can be processed manually by the Data Office. Automating the lineage phase frees up more time to focus on the data governance aspect to ensure regulatory compliance work.

Ensure compliance: the truth is in the code

Ernie: Automation ensures dynamic adaptation over time, depending on the different versions, processing periods, etc. The truth of the data is in the code. It is written somewhere in the technical process and the lineage is there to probe it, to have a precise view of it. This is the case with the COBOL programs, for example, whose secrets must be revealed by describing the lineage through a meticulous scan of the systems under the prism of their evolution over time. Thus, finding the path of the data can help analyze what COBOL programs are doing in a system.

 

Frédéric: The regulator needs to understand the data at the business level, but also at the technical level. Stakeholders are aware of the risk, particularly in the banking industry, and know their exposure to non-compliance fines. With data lineage and data governance, the processes are objectively described and everyone who has processed the data is identified. It is no longer necessary to investigate who made the code and who has the process in mind; everything is immediately available to the Chief Data Officer and the regulator.

 

Artificial Intelligence: leveraging data insights

Frédéric: While the Data Steward collects and models data for the data catalog, the role of the Data Scientist is to design algorithms that make recommendations to create or improve a service or product. This is possible, for example, thanks to the modeling of customer behavior based on data provided upstream. Therefore, it is interesting to have knowledge of the data life cycle (lineage) from the design phase of artificial intelligence to enhance selection of the best data sources and get the best possible results. It is also essential for the production phase to unlock the full potential of the A.I.

 

Ernie: To have reliable artificial intelligence, stable and good quality data are mandatory. The ability to detect changes over time and set up alerts by topic would be of great interest. The most important thing is to help Data Scientists by working on the technical life cycle of data and by bringing value through smart tags, reminders on quality, or other data-related issues. This notification feature is an additional step to be able to detect new data through progressive lineages.

 

Meet the experts

Frédéric Fourquet: Data Governance Product Marketing Manager, Frédéric started his career in 1997. Prior to joining MEGA as Product Marketing Manager for Data Intelligence, Frédéric was previously Product Director in Artificial Intelligence & Data Intelligence for Banking & Insurance - Strategy, Marketing, Business Development, Consulting & Partnerships (4 years), Consulting Director in Regulations, Compliance, Data Governance & Data Innovation (16 years), and Sales/Presales Executive and Consultant in US/EMEA Software Companies (4 years).

 

Ernie Ostic: Ernie is SVP of Products at MANTA, focusing on solutions for lineage and metadata integration. He has over forty years of experience in the data integration space, including twenty-plus years at IBM, working in a variety of roles with responsibilities in product management and technical sales support. For most of the past decade, Ernie has been providing guidance in information governance and helping architect custom lineage solutions. Earlier in his career, Ernie was building decision support systems with fourth generation languages and data access middleware. Ernie maintains a blog on open metadata, data lineage and overall metadata management and governance. He is a graduate of Boston College.

 

Recently, MEGA International and MANTA formed an important business and technological partnership to combine their respective expertise. This innovative collaboration enables companies to achieve regulatory compliance and provide them with access to highly accurate business data insights.

2970
0
Comment
OliviaO
MEGA

Trace the complex journey of data with automatic data lineage

Ernie Ostic: Data lineage is above all a flow of transformation. It makes it possible to trace the technical genealogy of data by giving a precise overview of the path traveled in computer systems. This approach provides a complete view of the life cycle of the data, from its collection to its use, to its destruction. Although automated lineage is necessary for complex technology portfolios, it is also about connecting with users (businesses) beyond the technical aspects, especially during the data discovery phase and the modeling of underlying processes.

 

Frédéric Fourquet: The challenge of the lineage process is to understand what is happening in the journey of the data: where it comes from, where it goes, who collects it, who uses it, reuses it, etc. Data is not static; it integrates processes and requires an in-depth and dynamic view. The technical lineage of the data makes it possible to know what precisely happened in the system, and what treatment the data have been through since it's set up. This work allows more in-depth data knowledge through retracing its history.

 

Data governance: proving the origin and destination of data

Ernie: With the increasing number of data regulations - notably GDPR in Europe - the challenge is to prove to the regulator how each piece of data was obtained. Failure to comply with this requirement can put organizations at risk in any industry. Since data management is a long-term transformation process, it is crucial to be able to trace the history of each piece of data to track its origin, processing, etc. It is no longer just a question of providing the processed data to the legislator, it is also necessary to be able to demonstrate its lineage, i.e., the data’s genealogy in the system. Given the exponential growth in the volume of big data, automation is essential.

 

Frédéric: Modeling the data life cycle in an automated way makes it possible to avoid enormous manual efforts - efforts which might even be impossible with a certain volume of data. Automation is essential, for example, when a company has several hundred critical data items to process and only 10 to 15 data items per year can be processed manually by the Data Office. Automating the lineage phase frees up more time to focus on the data governance aspect to ensure regulatory compliance work.

Ensure compliance: the truth is in the code

Ernie: Automation ensures dynamic adaptation over time, depending on the different versions, processing periods, etc. The truth of the data is in the code. It is written somewhere in the technical process and the lineage is there to probe it, to have a precise view of it. This is the case with the COBOL programs, for example, whose secrets must be revealed by describing the lineage through a meticulous scan of the systems under the prism of their evolution over time. Thus, finding the path of the data can help analyze what COBOL programs are doing in a system.

 

Frédéric: The regulator needs to understand the data at the business level, but also at the technical level. Stakeholders are aware of the risk, particularly in the banking industry, and know their exposure to non-compliance fines. With data lineage and data governance, the processes are objectively described and everyone who has processed the data is identified. It is no longer necessary to investigate who made the code and who has the process in mind; everything is immediately available to the Chief Data Officer and the regulator.

 

Artificial Intelligence: leveraging data insights

Frédéric: While the Data Steward collects and models data for the data catalog, the role of the Data Scientist is to design algorithms that make recommendations to create or improve a service or product. This is possible, for example, thanks to the modeling of customer behavior based on data provided upstream. Therefore, it is interesting to have knowledge of the data life cycle (lineage) from the design phase of artificial intelligence to enhance selection of the best data sources and get the best possible results. It is also essential for the production phase to unlock the full potential of the A.I.

 

Ernie: To have reliable artificial intelligence, stable and good quality data are mandatory. The ability to detect changes over time and set up alerts by topic would be of great interest. The most important thing is to help Data Scientists by working on the technical life cycle of data and by bringing value through smart tags, reminders on quality, or other data-related issues. This notification feature is an additional step to be able to detect new data through progressive lineages.

 

Meet the experts

Frédéric Fourquet: Data Governance Product Marketing Manager, Frédéric started his career in 1997. Prior to joining MEGA as Product Marketing Manager for Data Intelligence, Frédéric was previously Product Director in Artificial Intelligence & Data Intelligence for Banking & Insurance - Strategy, Marketing, Business Development, Consulting & Partnerships (4 years), Consulting Director in Regulations, Compliance, Data Governance & Data Innovation (16 years), and Sales/Presales Executive and Consultant in US/EMEA Software Companies (4 years).

 

Ernie Ostic: Ernie is SVP of Products at MANTA, focusing on solutions for lineage and metadata integration. He has over forty years of experience in the data integration space, including twenty-plus years at IBM, working in a variety of roles with responsibilities in product management and technical sales support. For most of the past decade, Ernie has been providing guidance in information governance and helping architect custom lineage solutions. Earlier in his career, Ernie was building decision support systems with fourth generation languages and data access middleware. Ernie maintains a blog on open metadata, data lineage and overall metadata management and governance. He is a graduate of Boston College.

 

Recently, MEGA International and MANTA formed an important business and technological partnership to combine their respective expertise. This innovative collaboration enables companies to achieve regulatory compliance and provide them with access to highly accurate business data insights.