Is anyone modelling Kafka Topics with HOPEX?

[BenAvdicevic]

Hi HOPEX Community,

Is anyone modelling Kafka topics with HOPEX?

If so - what metaclass are you using to represent a topic?

Also - on which diagram types are you showing your kafka topics?

• Application Structure?
• Application scenario of flows?
• other....

[ATomlin]

Hi,

Don't know if this will help, but we started studying this subject recently, and the best solution we came up with (HOPEX V2R1 and HOPEX V5)  was to use the class Evenement EAI (in English I suppose it's something like EAI Event). It can be used directly between applications (one publishing the event and others subscribing to it). However I haven't seen where you can store the bus used (KAFKA or other). 

If you've found a bette idea, I'm interested 🙂

[BenAvdicevic]

@PBessodes Thank you for this.

However, putting all this into a blog or forum post with a realistic modeling example would be the best for the community.   I know what I'm asking is hard.  But, this is the real modeling people are trying to do with HOPEX.   Official documentation and even ITA training does NOT give enough details.

[PBessodes]

Hello,

Here are the principles Mega has set and is currently following:

• All systems (regardless of granularity or systemic layer) can be modeled from 3 points of view: data, processing and communications.
• First systemic layer ("Application" layer) must contains components and dependencies (data exchange, service call...) without information regarding the way it is done (technology used). Separation of concerns ensures stability.
• Drilling in the technology layer of communication is made in Hopex with Communication System modelling (Help on Communication System - Hopex v5 ).
  It is based on the Enterprise Integration Pattern notation (Enterprise Integration Patterns ) and they provide advice for all kind of communication technology tool/context/pattern/framework/etc.
  You can find here (Competing Consumers - Enterprise Integration Patterns ) advice on Kafka moddeling.

Lastly, we plan to improve and complete the communication layer in Hopex in the future and the links with other layers.

[BenAvdicevic]

@PBessodes Does MEGA have a perspective on this?  

Like from your metamodel - what objects make sense to use - especially from ITA module perspective.

A blog on this topic would be most welcome.

[BenAvdicevic]

@serm Thank you very much!  

This is extremely helpful!   

I will need some time to digest in detail and provide my feedback.

Quick comments and observations.

Key statement that I 100% agree with is this:  

"i beéieve that what is important is to consider Kafka in the technology layer and consider business applications in the business layer, whatever the modeling techniques are."

I am pushing for a method that "hides" Kafka from Application interaction layer.     When we model Application A interacts with Application B via events (i.e. Kafka topics)   we are trying not to show Kafka "Application" in-between the Applications A and B.  Important to note is that we do have something like  "Enterprise Event Streaming Platform" in our Application catalog.   But again, we don't try to show that in our Application layer diagrams in HOPEX.

So - what we're documenting right now for Scenario of Flows diagrams is something like this:

We're still working on how to model similar thing in the "Structure" diagrams in HOPEX - using the "Interactions" and Exchange Contracts.     But, again - I think our approach will be to show the Topic as the interaction - and not as an Application object in between the applications.   Finally - we are thinking if - in Application structure diagram of Application A (above)  we should model the "topic" as a component IT Service  - or simply as a "Service Point" tied to an Exchange Contract representing the Topic.     The way I see it - "some" application must "own" the Topics.    It doesn't make sense that "Enterprise Event Streaming Platform" is the "owner" of all the defined topics.  

[SergeThorn]

Hi,

Yes I did that but with the ArchiMate module and in french but I will try to explain this in english explaining what I did and how that could be done with Hopex modelling.

In blue this is the Application layer. I have an Application called Data Pipeline and streaming in real time.

Two application component has been included, one related to real-time alerting and the second one a widget for account balances. These would be your IT Services.

These application components have application functionalities which are in your case functionalities.

Then in green the Technical Layer. There is a node which is some sort of grouping for Events.

Apache Kafka is the Software Technology. It also has two sub-software technologies which are Kafka Broker and Kafka Connect. This could your technical functionalities.

Replication and Streaming are technical functionalities in ArchiMate but could be your technical services.

Obviously there are relationships between the 2 architecture layers and would be different with Hopex.


i beéieve that what is important is to consider Kafka in the technology layer and consider business applications in the business layer, whatever the modeling techniques are.

I hope this helps a bit...

Kind regards