The following is an extract from the EDI Fundamentals & Best Practices training course offered by The EDI Academy (https://ediacademy.com/). This small extract is just one element of the whole course and should not be considered as an EDI tutorial.

Common Scheduling Pitfalls/Solutions:

The consequences of the following pitfalls are hours wasted for un-necessary troubleshooting and most importantly the cost of business-to-business transaction delay. Also, in the retail EDI industry this could be a major root cause of chargebacks that end up costing thousands of dollars to due to a scheduling oversight that could have been prevented.

• Cut-off times - are not taken into consideration. For example, banks have specific cut-off times by the time an EDI transaction should be received. If not received by the cut-off time, the bank may keep the transaction in the queue for as much as 24 hours. The cut-off times should be agreed upon in each trading-partner relationship.

• Time zones – Sometimes when trading partners agree on a certain job schedule, they make a careless mistake of not taking the time zone into consideration. For example, an 820 Remittance Advice/Payment Order transaction set must arrive by 2:00 PM eastern time in order for the payment order to process. Also, this is sometimes missed in operational manuals and run-books. Time-zones should be clearly agreed upon with trading partners before going into production.

• Weekends/Holidays – Most companies do their server and networking equipment maintenance on weekends. Since the EDI infrastructure is very much dependent on the network and server being up, jobs will fail if the FTP server or AS2 is down because of network/server maintenance downtime.

• Dependency jobs – Usually B2B Transactions are dependent on some internal application job completing first. For example, a retailer must complete its replenishment job before Purchase Orders can be transmitted to the suppliers. Some organizations make the mistake of not creating an actual dependency on these job completions and instead chose to schedule the EDI job several hours out after the estimated completion time.

• File Arrival/Time-Stamp Dependency– Very frequently jobs simply die because there is no file available for processing. Files are a powerful way to serve as a dependency for a job or a job-step to start/continue. For example, an outbound EDI translation job should not kick-off until the application file is in the directory and ready for processing. In addition to a file actually being available, it is a good idea to put a check in to place to make sure that the file has not been modified for more than a certain time period like 1 minute.  

• File Clean-Up to Prevent Duplicates - This is an oversight that happens too often. A script is written, a job is put into production, but archiving or renaming the file after processing has been completed is overlooked. The next time the job runs it picks up that same file and transmits over and the trading partner receives duplicate transactions. Hopefully the trading partner has duplicate-check processing in place. 

• Lack of deadlines – Some schedulers often have a deadline feature. For example, if a job starts and does not complete in an X amount of time then alert should be transmitted or the job should exit. A deadline should be set for all jobs in the scheduler if possible. 

• Year-End-Processing – Accounting usually asks for a freeze period for incoming and outgoing transactions for year-end processing. Careful planning for suspending jobs and reaching out to trading partners to give them heads up can prevent a lot of unnecessary troubleshooting.  

• Lack of end-to-end testing in the scheduler - Very often scheduling issues come up during the first production run. Usually EDI transaction testing occurs in an adhoc method and the scheduler is only turned on for production. One of the last milestones during testing should be turning on the scheduler and letting the process run end-to-end using an automated scheduled method vs adhoc.  

PUSH vs. PULL Transmissions:

Data transmission method PULL vs. PUSH should be carefully taken into consideration for protocol design. The decision will have implications on security and usability.

PULL: The pull method is when an organization decides for example to initiate an FTP connection to the trading partner’s server or a VAN to download the transaction and process it.

• PULL Method PROS:
  • The organization doing the pull does not have to host and maintain a server with user-names and passwords.
  • The receiver chooses when to receive the data and uses that as a solution for scheduling constraints.
  • Start up costs are inexpensive an FTP/SFTP client software is usually enough to get going.

• PULL Method CONS:
  • Jobs have to be scheduled to pull the transactions and the transaction availability time is unpredictable therefore forcing the receiving trading partner to set up schedules. For example, the file may be available to download anytime from 9:00 AM to 5:00 PM and the receiving party decides to run the job that pulls the data every hour on the hour. That would be 8 scheduled jobs that have to be monitored, tracked and audited.
  • This creates unnecessary traffic for the host.
  • The hosting party is burdened with having to worry about storing the files until they are pulled.

PUSH: The push method is when an organization decides for example to initiate an FTP connection to the trading partner’s server or the VAN and push the data instead of pulling it. The most successful method of implementing this is when both trading partners use the PUSH-PUSH method.

• PUSH Method PROS:
  • On the receiving side the trading partner does not need to worry about scheduling a job that logs on to the host server and pulls the data. The data is pushed by the sending party and the arrival of the data triggers the translation job to start.
  • On the sending side the trading partner does not have to worry about waiting for the receiver to retrieve the data before archiving the file. After the push has been completed the sending trading partner simply archives the file. 
  • There is no unnecessary traffic. The files are pushed only when the data becomes available. 

• PUSH Method CONS:
  • Start-up costs are more expense to host and maintain a server.
  • Sometimes the data will arrive at an un-wanted time when the receiver is not ready for it. However this can usually be remedied by using a dependency constraint in the scheduler tool.

Event-Driven Scheduling:

Event-driven scheduling happens when jobs are triggered by an event, for example on data arrival or even a record update in a database.

One of the easiest ways to accomplish event-driven scheduling is by using the push-push method between by both the sending and receiving parties. In this method trading partners do not schedule jobs for the transmission of the data. They set up host servers and push data do each other as soon as the data becomes available. There are many communication gateway tools that specialize in what’s called Managed File Transfer (MFT) that can make this process easier to implement.

For example, consider the following scenarios to easily tell apart Event-Driven scheduling vs Batched Scheduled Jobs:

• Batched-Scheduled-Method Scenario: A retailer’s procurement system completes a nightly replenishment job of inventories and is now ready to cut thousands of purchase orders to hundreds of trading partners. When the job is completed the application file with all the purchase orders is extracted to a directory waiting for the EDI translator to pick it up. The replenishment job may complete any time between 3:00 AM and 6:00 AM. Then a batched-scheduled scenario occurs twice: (1) The EDI translator has a scheduled job that sweeps the directory with the PO files every 30 minutes between 3:00 AM and 6:00 am. Once the file is picked up and EDI translation occurs, the 850 Purchase Order is now available for the receiving trading partner to download from the retailer’s SFTP server. (2) The receiving trading partner has a scheduled job that sweeps retailer’s SFTP server from 4:00 AM to 9:00 AM for an 850.

This method creates for unnecessary sweeping of data and burdens the operations team that monitors the jobs. Also in the examples above the system doing the extraction has to worry about waiting for the receiver party to retrieve the data before archiving it.

• Event-Driven-Method Scenario: A retailer’s procurement system completes a nightly replenishment job of inventories and is now ready to cut thousands of purchase orders to hundreds of trading partners. When the job is completed the application file with all the purchase orders is extracted to a directory waiting for the EDI translator to pick it up. The replenishment job may complete any time between 3:00 AM and 6:00 AM. Then an event-driven scenario occurs twice: (1) The arrival of the file in the directory triggers the EDI translation job to start immediately without a scheduled batch job that sweeps the server. (2) Once EDI translation completes the file is pushed via SFTP to the trading partner’s host system. The trading partner receives the 850 PO and that triggers an EDI translation job to start on their end. 

This method has a clear advantage over the scheduled batch process and can be accomplished using the PUSH-PUSH method. Also, data is available and processed immediately instead of having to wait. Complicated archiving timing, unnecessary traffic, unnecessary monitoring of multiple scheduled jobs is eliminated and provides for an overall more efficient transmission and scheduling process.

AS2: The most natural solution for Event Driven Processing:

A separate article can be done AS2 alone, so this section does not attempt to define what AS2 is.

If a company has ever chosen to implement to AS2 (or AS1/AS3) as the protocol of choice for data transmission they have automatically signed up for a PUSH-PUSH method. In order for AS2 to work both parties must have AS2 software installed and installing AS2 automatically makes the machine that it is installed on a web-server.

When trading partners are ready to exchange data they do AS2 HTTPS posts of the data. The data is pushed to the trading partner’s AS2 web-server for processing. AS2 is usually tightly integrated with the EDI translation software so the arrival of the data immediately triggers the translation jobs.

AS2 also makes the operations and the monitoring process simpler with MDNs. MDN stands for Message Disposition Notification. Typically trading partners will want some kind of confirmation that the data transmitted was indeed successfully sent. Upon the receipt of the data and successful signature or decryption validation an MDN message is sent to the sender for confirmation. This MDN message can be sent back via HTTP/HTTPS immediately via AS2 w/ "Sync" MDNs or at a later time via AS2 w/ "ASync" MDNs. The MDN option maybe turned off if trading partners agree that no such confirmation is required. Also, the MDN maybe sent back via EMAIL which is very rare.

EDI Academy 2011: Find out more

To learn more and sign up for a seminar, see the organization’s Training Academy course schedule page; or find out more about the EDI Academy at https://ediacademy.com/