Get raw EDI of current line

• 

  Admin

  • January 25, 2019 12:32

  Hello Brandon,

  Could I please ask why would you want to write the raw segment to a separate file ?

  To read EDI files as raw segments you don't really need a separate framework, you can just read the segment separator at position 106 and then split by it. This way you get a list of all the segments as raw strings.

  EdiItem is either a control segment (ISA, GS, GE, IEA) or a transaction (a derived class from EdiMessage). The purpose of the EDI reader is to transpose the flat EDI file into its hierarchical representation as prescribed by the implementation guide. 

  There are also the following methods available:

  1. EdiMessage.ToEdi() - this generates the transaction as a raw EDI string.

  2. EdiMessage.FlattenSegments() - this produces a List<object> of all parsed segments in the original order.

  3. EdiMessage.FlattenRawSegments() (just released as part of version 9.8.2) - this produces a List<string> of all raw segments in the original order.

  If you are splitting files this may be of interest:

  Split files with multiple interchanges

  How to split multiple transactions into separate files and interchanges

  static void Split()
  {
  	var output = Directory.GetCurrentDirectory() + @"\..\..\..\Files\Out837\";
  	var di = new DirectoryInfo(output);
  	foreach (FileInfo file in di.GetFiles())
  		file.Delete();
  
  	Stream ediStream = File.OpenRead(Directory.GetCurrentDirectory() + @"\..\..\..\Files\In837\837P.txt");
  
  	using (var ediReader = new X12Reader(ediStream, TemplateFactory))
  	{
  		int i = 0;
  		while (ediReader.Read())
  		{
  			var msg = ediReader.Item as TS837P;
  			if (msg != null)
  			{
  				i++;
  				using (var stream = new MemoryStream())
  				{
  					//  Set postfix to CRLF just to inspect the files, remove when not testing
  					using (var writer = new X12Writer(stream, new X12WriterSettings { Postfix = Environment.NewLine }))
  					{
  						writer.Write(ediReader.CurrentInterchangeHeader);
  						writer.Write(ediReader.CurrentGroupHeader);
  						writer.Write(msg);
  					}
  
  					var path = string.Format(output + "837P_{0}.txt", i.ToString());
  					File.WriteAllBytes(path, stream.ToArray());
  				}
  			}
  		}
  	}
  }

  0
• 

  Brandon Stalte

  • January 25, 2019 15:29

  Here's what I'm trying to do. I have EDI files ranging anywhere between 1MB to 200MB in size. I need to iterate over each transaction and if its a certain transaction type, capture it into a POCO to easily aggregate information, then build some new aggregated data into a List<string> and at the end of each transaction, write out the List<string> object before the SE segment.

  Is there an efficient way to do this on large files without putting all of the transactions into POCO first? So we'd only have to deal with one transaction as POCO in memory at one time, write it back out along with the custom data and dispose of it and move to the next?

  0
• 

  Admin

  • Edited January 25, 2019 18:22

  Hi,

  Doing 

  while (ediReader.Read())

  means that there is only one edi item in memory. It doesn't really matter how big is the edi file, memory-wise, the reader will stream through it. If the transactions themselves are too large, then splitting can be applied.

  You already deal with a single transaction at any moment and you can modify it by adding\updating or deleting segments. You can then write it to a string\stream\file. You can build your own template to support whatever custom transaction you require to handle any extra information that you want to aggregate into an existing standard transaction.

  For anything that has to do with segments as strings or lists of strings, you don't really need a framework. Just write the strings with File.WriteallText()

  Please share some pseudo code of your flow, especially around:

  "capture it into a POCO to easily aggregate information, then build some new aggregated data into a List<string> and at the end of each transaction, write out the List<string> object before the SE segment."

  - what is this aggregated data and why should it be in a list of strings if it's part of the transaction ?

   

  0
• 

  Brandon Stalte

  • January 26, 2019 15:51

  Sorry for the confusion. Let's start over.

  I would like to read a large EDI transactions (ST/SE over 70k lines) into an EdiMessage, gather some information from the  EdiMessage and build a new List<T> of data that is then tied to each respective EdiMessage. Then I would like to write the EdiMessage out to a new file with the List<T> data appended to the EDI before each SE control segment. The files can contain several transactions amounting to over 80MB per file.

  So think of my process like this:

   

  Read EDI into a TS867 (X12Reader.Read())

  1. Parse TS867 and create a new List<SomeDataClass> of data
  2. Write the TS867 back out to a new file like:

  ST…

  BPT…

  PTD…1

  QTY…

  PTD…2

  QTY…

  CustomDataHeader~A~B~ 1 from List<SomeDataClass>

  CustomerDataDetail~1~3 from List<SomeDataClass>

  CustomDataHeader~A~B~ 2 from List<SomeDataClass>

  CustomerDataDetail~2~3 from List<SomeDataClass>

  SE…

  The CustomDataHeader and detail objects would not be EDI, they would be essentially delimited string data. I need to preserve the raw EDI but the new lines need to be injected before the SE so they can be related to each transaction.

  I'm testing out just the basic method of reading large 867s into a TS867, and then writing it back out either with X12Writer.Write or with File.AppendAllText of the TS867.FlattenRawSegments but both writing actions then take about double the time it took to build the transaction into the TS867.

  Looking at the frameworks custom template ability, I think its possible to add the List<T> class data to a custom template and have the X12Writer write it out if I made the data's properties seudo-EDI string properties.

  This is basically a pre processor that is meant to inject some aggregated data at the end of each transaction. Then the file is loaded at a later stage.

  I can't simply iterate over each line without building the groups into the TS transactions because I want the TS object looping structure to gather aggregated data. So the X12Reader works well for this task. The downside is that reading the EDI transactions into an EdiMessage and then having the X12Writer write them back out to a new file takes long when dealing with large transactions with over 70k lines.

  0
• 

  Admin

  • Edited January 27, 2019 08:49

  So all the data is batched into a single transaction instead of multiple transactions.

  In this case you'll need to implement splitting by a repeating loop, in your case it's the PTDLoop. To do that simply add [Splitter] attribute to the loop property like this:

  /// <summary>
  /// Loop for Product Transfer and Resale Detail
  /// </summary>
  [Splitter]
  [DataMember]
  [Required]
  [Pos(11)]
  public List<Loop_PTD_867> PTDLoop { get; set; }

  This way, when you do

  ediReader.Read()

  the edi item will contain the loop data only + all the segments positioned before the loop.

  You can then generate the custom headers incrementally and write the current loop out (appending) to a file or stream. To do so do:

  1. For the first edi item - write everything out to a file. If you use X12Writer, set Autotrailers to false to avoid writing the SE segment.

  2. Call FlattenRawSegments() on the subsequent ediMessages and remove all segments before PTD (to only write them once, step 1.). Then use File.appendAllText().

  For optimal speed to read and write in the same time, use the async reader and async writer . 

  This way only small chunks of data will be process at any moment in time, the only constant bit is your aggregated CustomerData objects. This use of EDI is unconventional and that would render the EDI file invalid (if you call IsValid on the EdiMessage).

  If it's possible to implement the custom headers as part of the EDI template (which is straightforward, follow the pattern with group and segment attributes, and the [Pos] attribute applied to properties), that would be the preferred option from EDI point of view. You'll just be dealing with a custom template for this transaction set (867) and your data would be a valid EDI message.

   

  0
• 

  Brandon Stalte

  • February 03, 2019 20:23

  Thank you for the information. Splitting the transactions has really helped break down the data for processing.

  0
• 

  Admin

  • February 04, 2019 09:54

  Cheers Brandon, I'm glad it helped.

   

  0

Please sign in to leave a comment.