What is EDI Translator ?
Part 1 introduced the basic building blocks of EDI documents — envelope, functional group, transaction, EDI separators and EDI version. Part 2 will delve into the structure of EDI transactions.
All segments in between ST and SE for X12 or between UNH and UNT for EDIFACT, constitute an EDI transaction. They represent the business data carried out by EDI documents and adhere to the rules of the standard. Each EDI transaction is defined by what is commonly known as the EDI rule (also EDI guideline, EDI spec or EDI implementation guide).
The rules are usually distributed either in a tabular form (X12 & HIPAA):
or as a tree (EDIFACT & EANCOM):
EDI rules stipulate the exact format for each transaction such as invoice or medical claim. From a programmer’s perspective they govern the following:
- The blocks of segments grouped together and known as loops (X12) or groups (EDIFACT).
- The sequence and positions at which segments or loops can appear.
- The number of times a segment or a loop can repeat at the same position.
- Whether a segment or a loop is mandatory or optional.
EDI parsers transform the linear structure of EDI documents (EDI document is the collective for the physical representation of EDI data such as EDI file or EDI stream) into the hierarchical structure of EDI rules. They are like functions that take EDI transaction as the input parameter and then output a parse tree which conforms to the grammar of its corresponding EDI rule.
The history of this format dates back to the days of the Berlin Airlift in 1948, hence it was one of the first attempts to automate processing of large volumes of data. It was a way to compress business data at the source and to provide the instructions (the EDI rules) to unambiguously decompress it at the destination. It’s like the data is encrypted according to the EDI rules and one can only decrypt it if he follows those rules. They always go hand in hand — the EDI document (the encrypted data) and the EDI rule (the decryption key).
EDI segments are identified with their segment tags or segment IDs. The tag of the ST segment is ST. The tag of the UNH segment is UNH. The tag is defined as all characters from the beginning of the segment (assuming that we’ve already split the EDI document to segments by using the segment separator, the escape indicator and cleared any post fixes) until the first occurrence of the data element separator. Segment tags are usually comprised of 2 to 3 characters.
EDI loops (or groups or blocks of segments) are identified by their first segment, also known as the trigger segment. This segment is always mandatory for the loop and can repeat only once. A repetition of the trigger segment means that the whole loop is repeated. Usually loops in X12 are named after the trigger segment, e.g. ‘NM1 Loop’. Loops in EDIFACT are enumerated as GR1, GR2, etc.
The designer of the EDI rule must ensure that it describes the transaction unequivocally. For example the following EDI rule has an ambiguous sequence:
It is unclear to which BEG from the two should the BEG segment from the transaction below be mapped to:
The same logic applies if there was an optional segment in between the BEGs etc. The point is that EDI rules must be properly designed.
Parsing EDI means that each segment in an EDI document can be uniquely mapped to a segment in its corresponding EDI rule. To find a match the parser compares segments by their tags. The search path is important because segments with the same tag can be nested or appear on multiple levels. For the purpose of this article we’ll assume that tags are sufficient to identify a match. In reality however, it all depends on the representation of the EDI rule and of the parser itself. HIPAA transactions for example can have adjacent same-tag segments on the same level. To determine which one is a match by tag only is not enough. The parser must peek into the data elements and work out the match by the EDI codes of the first few data elements.
EDI Transactions are forward only — there are no jumps back and no GOTOs. Once a segment had been matched, all previous segments on the same level will never be used for matching again.
If one can represent the EDI rule as a tree where segments are the nodes and children stem only from trigger segments then matching an EDI transaction against that tree would be to apply a Depth First Search (DFS) algorithm. DFS traverses the tree (EDI rule) by starting from the root (ST or UNH) and then exploring along the branches e.g. if a match was found to a trigger segment, then the search continues a level down before backtracking.
DFS means that if we have the following EDI rule with two segments with the same tag NM1 but at different levels:
In the following EDI transaction:
NM1 will be parsed to the green NM1, hence DFS — it will search for a match in depth first. This is because there is a REF segment which is the trigger segment for the REF Loop. Trigger segments play the role of switches to alternate the search path.
If there was no REF segment NM1 would have gone into the grey NM1, e.g. in the following EDI transaction:
NM1 will be parsed to the grey NM1.
The parsing algorithm can be roughly explained with the following statements:
- Iterate through the segments in the EDI transaction.
- For each segment find a match in the EDI rule tree.
- Maintain the position in the tree (both the position in the level and the position of the level).
- Search forward only.
- Match by segment tags.
- Match the first segment with the same tag in the same level.
- If the last matched segment was a trigger segment then continue searching in the level down (or the sequence of segments within the loop).
- If no match is found backtrack to the level up or the original trigger segment.
- Account for repetitions, e.g. search from the last found segment again rather then the next in level.
- Finish when all segments were matched or reached the trailer segment.
I’ll leave out the details of Depth First Search implementation because it is not the premise of this article. The important thing is the DFS fits nicely into parsing EDI and is what we use in EdiFabric to translate EDI files.
In Part 1 I briefly touched on the problem of describing EDI rules programmatically. There is no unified way to do so. All EDI products I know of (commercial or open source) rely on proprietary mechanisms to do it — they’ve created custom script languages, meta templates, XSD, SEF, class hierarchies, etc. It is a very similar story for the actual output or what to parse EDI to.
Most parsers would produce either XML or object of some kind (which is often based on a deserialized XML or some other wrapper around XML). The thing about XML is that it is language agnostic, is well supported in all languages and is widely used in SOAP. Oh, and it represents hierarchical data structures. Sounds like the perfect match for EDI, no wonder they’ve been like conjoined twins for so long.
Let’s also assume that we have the following use case:
As a user I want to be able to parse EDI files and then to save the data from the files to an existing database.
Let’s also assume that the following diagram describes the general flow:
The main points are:
1. Maintain the EDI Rules.
EDI Rules are actual files or documents that require a separate application for editing them. In the case when XSD was used then any XSD editor would be sufficient. In the case of custom format then there would be a custom tool such as SEF editor.
2. Performance — parsing speed and memory usage.
The parser will read the data from the EDI file, search the EDI Rule and build the output instance. That’s at least 3 in-memory instances if the output is not instance of the EDI Rule, e.g. if the EDI rule is XSD and the output is XML.
3. Manipulating the output.
Depending on the type of the output it would be XML DOM if it’s XML or language specific if it’s not XML.
Remember that EDI versions are not backward compatible ? And that companies have extended the standard rules according to their liking ? All this leads to having to support a number of maps for the same transaction. These maps will translate all the partner-specific or version-specific variants of the business transaction to the internal domain representation.
It’s clear that parsing EDI is not an easy task. There are too many moving parts and external dependencies. Looking at the diagram however, highlights something rather important — integrating with a partner using EDI is no different than integrating with a partner not using EDI. A non-EDI related process, say pure XML or UBL, would have resulted in a similar diagram. It’s business data in the end of the day, why would it matter how it is structured.
Companies want to build value networks. They have applications to onboard other trading partners to be able to communicate with them and to exchange documents. They have strategies to do so. There is absolutely no point to duplicate that for EDI. What we see today is that companies still maintain separate integration approaches to communicate with trading partners depending on whether they use EDI or not. Because EDI was considered too expensive, murky and a privilege for the global top players only.
Let’s go back to the diagram. It is a generalized schema of the components involved in parsing EDI and how they are intertwined. It is the most popular architecture today but it is not the most efficient. It clings on XML and its agnostic nature.
My first EDI parser was like that. It spitted out XML at the end and provided XML DOM interface for manipulation. The thing is that EDI is complex already and companies have been marking this complexity for years. What if we scrapped XML altogether. What if it were possible to manipulate EDI directly, as in EDI DOM ?
This is what we did in EdiFabric. We excluded XML. This is how our new parser looked like:
This is the same diagram as above but without the grey boxes. It improves the following:
1. Describe EDI Rules with OOP classes.
This is not agnostic I agree. However, it allows you to manipulate EDI in your native programming language. No more XML DOM, please welcome EDI DOM. In .NET you can serialize any object to XML or JSON natively. So it becomes a matter of choice really. The parser output is an instance of the rule class.
Here is what I’m talking about (samples in C#):
X12 810 (Invoice)
X12 850 (purchase order)
HIPAA 837 P (claim)
EDIFACT INVOIC (invoice)
EDIFACT ORDERS (purchase order)
2. No need for extra editors to maintain the rules.
Being OOP classes you can use your favorite IDE such as Visual Studio or Eclipse to edit the rules.
3. No need for explicit mapping.
The parser output can also be an instance of the destination domain class. You can turn any class hierarchy into EDI Rule thus eliminating the need to maintain multiple external maps. You’ll still need to maintain multiple domain classes for the different EDI rules (per partner and per version) but it’s all in your favorite programming language.
We’ve achieved this with a set of custom attributes which can be used to annotate classes and properties to enable them with EDI capabilities.
The parser will read the data from the EDI file, search the EDI Rule and populate that same EDI Rule. There are only two instances in-memory. There is no extra code to transpose the EDI Rule. There is no XML serialization\deserialization. There is no XML DOM to inflate the original file size tenfold in memory. In reality we managed to increase the parsing speed by 3 times and to reduce the memory consumption by 12 times. The actual code was also much simpler and clearer. We were impressed.
The benefits of EDI DOM
It’s a no brainer when it comes to implementation. Soon enough companies will be able to exchange EDI documents via RESTful services and JSON. They will be able to manipulate EDI business transactions in the same way as their internal objects, let’s say invoices or purchase orders. EDI is nothing more than a format. It doesn’t have to be dealt with differently, it doesn’t have to be tied to XML and it doesn’t have to be the short stick.
EDI DOM is the approach that I, as a programmer, would prefer to any other. It gives me the best model to represent and maintain EDI. It allows me to export it to XML or CSV or anything I want. It saves me the pain of having to create, maintain and deploy multiple maps. It aligns well with all other applications that we build in-house. And it’s the most efficient way to translate EDI.
This is the second part of this article. It described the algorithm and the data structures that can be used to parse and represent EDI documents.