cancel
Showing results for 
Search instead for 
Did you mean: 

Model Risk Management – How Reliable Is Your Data?

0
0
Model Risk Management Reliable Data

Ernst & Young published this white paper in 2014 where they warn that “Financial models introduce risks at all insurance organizations and should be addressed as part of a comprehensive risk management program to protect an organization’s financial strength and reputation.” Although this paper is directed at the insurance industry, the same is true for all organizations that rely on models to drive their business.

Michael Rasmussen of GRC 20/20 has stated “Models are used across industries to analyze, predict, and represent performance and outcomes that impact operations and business strategy.” Many organizations leverage financial models where information comes in, it’s processed through an equation/algorithm, and information goes out. The resulting data then cascades to the next model, and the next, and so on. What if information going in was incorrect and/or some part of the equation was incorrect? Of course the data coming out would be incorrect. That cascades to the next model, which may or may not also be incorrect – the result being even more flawed data now coming out of that model into the next, leading to a buildup of incorrect information. How do you trace it back?

In June of 2014, Chartis Research published a paper called “The Risk Enabled Enterprise – Model Risk Management”. In it, there are survey results that demonstrate that “Poor quality data and insufficient data are viewed as the two most significant sources of model risk. 72% of respondents said they view poor quality data as an important or very important source of risk. Firms need to set up a dedicated model inventory system and develop dedicated model risk management technology tools and systems to assess model risk.”

Now that potential problems have been identified, making sure there is a clear understanding of the nuances associated with model risk management is important. Sean Keenan, senior managing director of model risk at AIG, recently wrote a book titled “Financial Institution Advantage and the Optimization of Information Processing”. Keenan discusses many concepts related to model risk management, including the differences between model risk management and model governance. Many organizations, specifically financial Institutions, sometimes equate model risk management with model governance – but the two are very different, and the relevance of this difference is growing:

  • Model governance seeks to control risk across diverse processes (model development, deployment, and validations) and stakeholders (modelers, model users, validators) using granular rules and analysis of the model inventory. The focus is on minimizing downside risk.
  • Model Risk Management seeks to support the maintenance and evolution of analytic infrastructure, enhancing the effectiveness of that infrastructure to achieve evolving commercial objectives over time. Model risk management focuses on systems that include models, data sources, modeling platforms, and model delivery capabilities.


Understanding these differences is crucial, but now there is a need to take action. Mature organizations are beginning to understand they can use enterprise architecture to capture business and enterprise knowledge, to capture model logic, and above all, put the financial models in their actual use case and business context. If the organization is innovative, they can also incorporate standard GRC capabilities such as model testing, model control and business value assessment.

Sean Keenan notes that models are typically embedded in systems/streams, in which they interact to process data for downstream consumption. Analyzing models at stream level is key to:

  • Validating models: Model validation requires some consideration of this broader context/infrastructure.
  • Planning: Business strategic planning is also typically done at the stream level, to enhance analytical capabilities and to appropriately sequence work.
  • Assessing risks of models: At stream level, model risk & data risk can more easily be quantified and communicated. One of the biggest challenges to measuring model risk is establishing an appropriate “exposure” concept. The complexity of this problem is reduced by elevating the unit of observation to the stream level.

If managed from an enterprise architecture perspective, an organization can leverage data from a single repository to manage model related objects, including the models themselves and their inter-relations, their data sources, their delivery & monitoring mechanisms, and the specific reports and other end-uses that they serve. As Keenan points out, model relationships can be graphically represented in streams, which show how models interact to produce output for downstream consumption. This approach facilitates the management of the mass of complex information needed to preserve an accurate representation of the complete analytic infrastructure.

Understanding the problems associated with financial models, the confusion between model governance and model risk management, and the steps that can be taken to create visibility into the relationships between the data will help organizations to more effectively manage model risk. Model governance platforms - especially in the GRC domain - are progressively evolving to become major influencers of the way models are managed and how model risk management is handled internally. Given the influence of these platforms, platform features and platform architecture may either be enablers or constraints to future capabilities. Systems that are based on an object-oriented architecture are more likely to be an enabler on future capabilities. 

0
Comment
MEGA

Ernst & Young published this white paper in 2014 where they warn that “Financial models introduce risks at all insurance organizations and should be addressed as part of a comprehensive risk management program to protect an organization’s financial strength and reputation.” Although this paper is directed at the insurance industry, the same is true for all organizations that rely on models to drive their business.

Michael Rasmussen of GRC 20/20 has stated “Models are used across industries to analyze, predict, and represent performance and outcomes that impact operations and business strategy.” Many organizations leverage financial models where information comes in, it’s processed through an equation/algorithm, and information goes out. The resulting data then cascades to the next model, and the next, and so on. What if information going in was incorrect and/or some part of the equation was incorrect? Of course the data coming out would be incorrect. That cascades to the next model, which may or may not also be incorrect – the result being even more flawed data now coming out of that model into the next, leading to a buildup of incorrect information. How do you trace it back?

In June of 2014, Chartis Research published a paper called “The Risk Enabled Enterprise – Model Risk Management”. In it, there are survey results that demonstrate that “Poor quality data and insufficient data are viewed as the two most significant sources of model risk. 72% of respondents said they view poor quality data as an important or very important source of risk. Firms need to set up a dedicated model inventory system and develop dedicated model risk management technology tools and systems to assess model risk.”

Now that potential problems have been identified, making sure there is a clear understanding of the nuances associated with model risk management is important. Sean Keenan, senior managing director of model risk at AIG, recently wrote a book titled “Financial Institution Advantage and the Optimization of Information Processing”. Keenan discusses many concepts related to model risk management, including the differences between model risk management and model governance. Many organizations, specifically financial Institutions, sometimes equate model risk management with model governance – but the two are very different, and the relevance of this difference is growing:

  • Model governance seeks to control risk across diverse processes (model development, deployment, and validations) and stakeholders (modelers, model users, validators) using granular rules and analysis of the model inventory. The focus is on minimizing downside risk.
  • Model Risk Management seeks to support the maintenance and evolution of analytic infrastructure, enhancing the effectiveness of that infrastructure to achieve evolving commercial objectives over time. Model risk management focuses on systems that include models, data sources, modeling platforms, and model delivery capabilities.


Understanding these differences is crucial, but now there is a need to take action. Mature organizations are beginning to understand they can use enterprise architecture to capture business and enterprise knowledge, to capture model logic, and above all, put the financial models in their actual use case and business context. If the organization is innovative, they can also incorporate standard GRC capabilities such as model testing, model control and business value assessment.

Sean Keenan notes that models are typically embedded in systems/streams, in which they interact to process data for downstream consumption. Analyzing models at stream level is key to:

  • Validating models: Model validation requires some consideration of this broader context/infrastructure.
  • Planning: Business strategic planning is also typically done at the stream level, to enhance analytical capabilities and to appropriately sequence work.
  • Assessing risks of models: At stream level, model risk & data risk can more easily be quantified and communicated. One of the biggest challenges to measuring model risk is establishing an appropriate “exposure” concept. The complexity of this problem is reduced by elevating the unit of observation to the stream level.

If managed from an enterprise architecture perspective, an organization can leverage data from a single repository to manage model related objects, including the models themselves and their inter-relations, their data sources, their delivery & monitoring mechanisms, and the specific reports and other end-uses that they serve. As Keenan points out, model relationships can be graphically represented in streams, which show how models interact to produce output for downstream consumption. This approach facilitates the management of the mass of complex information needed to preserve an accurate representation of the complete analytic infrastructure.

Understanding the problems associated with financial models, the confusion between model governance and model risk management, and the steps that can be taken to create visibility into the relationships between the data will help organizations to more effectively manage model risk. Model governance platforms - especially in the GRC domain - are progressively evolving to become major influencers of the way models are managed and how model risk management is handled internally. Given the influence of these platforms, platform features and platform architecture may either be enablers or constraints to future capabilities. Systems that are based on an object-oriented architecture are more likely to be an enabler on future capabilities.