Event Subscription Service FAQs
Calling the ESS Dequeue Webservice
How Often Can I Call Dequeue (DQ)?
As often as necessary, but not too often. Yes, that's rather vague, isn't it?
Each Dequeue call can return up to 64k worth of data, and each High Volume Dequeue call can return many times more than that. Ideally, you want to return as much data in each call as possible. Generally:
- If calls result in transaction blocks containing only a handful of transactions, the interval between calls needs to be increased.
- If calls result in completely full transaction block, the interval between calls can be decreased.
So if you call Dequeue every 5 minutes, and many of your calls result in no transactions being returned, then you should call Dequeue less frequently. If you call Dequeue every 15 minutes, and you get a full buffer each time, then you should consider calling Dequeue more frequently in order to keep up with your company's transaction volume.
Calling DQ too often without having any transactions returned is not an efficient use of resources.
How Big Should the Block Size Be?
Note that the block size is only present in Dequeue calls, not in High Volume Dequeue calls.
We recommend you set your block size to 500.
You can tell DQ to give you a certain number of transactions (block size). Ideally you want as many transactions per call as possible. You want that 32k response buffer to be full, every call. There are very few situations where you want less than a full buffer (debugging?), so it's best to set the block size to something big, like 500.
Currently, significantly fewer than 500 transactions can be packed into the return transaction block. However, we still recommend this block size to ensure that the transaction block is filled (if possible) in the event considerably smaller transaction types (lengths) are supported in the future.
How Do I Acknowledge Receipt of Transactions?
Every transaction returned by DQ has a unique transaction ID:
|
|
Every call to DQ lets you "acknowledge" a transaction ID.
|
|
You want to "acknowledge" the highest transaction ID you successfully received.
You're basically telling ESS, "Okay, I got transaction #4021 and every one before it, you can remove everything up to and including #4021 from my queue, thank you."
This helps you in case you "drop" transactions or your program crashes while processing.
If you don't acknowledge transactions (ie: you send zero), you'll get the same block of transactions you got last time. We won't remove them from your queue until you tell us you got them successfully, via acknowledgment of a transaction ID.
ESS XML Schema Definition (XSD)
What IS an XML Schema Definition (XSD) File?
XML Schema Definition (XSD) Files:
- Are human readable!
- Are written in XML
- Define the datatypes and formats that you can expect to receive in an XML document
The XSD basically is a specification telling you about some XML you're going to receive. Without the roadmap you could read the XML, but the roadmap tells you whether the XML you received was valid, whether it was well formatted. It also lets you prepare for receiving the XML document, because you know what kind of datatypes and formats you are going to receive.
It also serves as an integration specification between parties. An XSD defines the only kind of XML that can be exchanged. If some XML exchanged doesn't obey the XSD, it's a broken contract.
What's the Purpose of the ESS Transaction XSD File?
Its purpose is manifold:
- It defines the general format of the XML that ESS will give you.
- It defines the contents of the XML that ESS will give you.
- It defines the datatypes used within the XML that ESS will give you.
- It documents the purpose, meaning and in some cases formats, of the XML elements and attributes and their data values.
It's a specification telling you all about the ESS XML transaction format. It defines the transaction elements and attributes, how they relate, how they're nested (parent-child relationships), what data they contain, what parts are required or optional.
It's our integration specification, telling you the rules we will obey when giving you XML data.
What Can I DO with the ESS XSD File?
You can:
- Validate that the ESS transactions you are given obey the schema
- Use it as a guideline to write your integration code in anticipation of the XML you will be given
- Auto-generate a set of class definitions which represent the data structures defined in the schema
Alternatively, you can do absolutely nothing with the XSD if you choose. It's not required that you actually put it to use. Remember, it's human readable. If all you do is read it to get an understanding of the XML we'll be giving you, that's fine. But many tools exist which can use XSD's to help you do your job depending on the platform you're developing on.
All of the above can be done with the help of tools that you can find easily online. Many platforms (J2EE, .NET, etc) have built-in tools which know how to manipulate XSD's.
Can I Generate Client-Side Integration Code from the ESS XSD File?
Yes, you can!
For example, .NET has a tool called Xsd.exe which takes in an XSD file and can generate C#.NET or VB.NET classes which are equivalents of the structures defined in the XSD. This eases the process of deserializing the XML into platform-specific datatypes that you can use within your code.
Other platforms have different tools available. For example, J2EE uses a tool called JAXB.
You don't have to do this, you can still write integration without importing the XSD file into your application. This is just one method you can choose to help you write your integration code.
How Do I Know When You Are Going to Change or Extend the ESS Schema?
We notify all customers via the main integration documentation page on our public wiki. The notice includes details about the planned changes.
We also contact customers directly by email and/or phone.
XML Validation
Should I Be Validating the XML Returned by ESS to Ensure it Adheres to its Schema (XSD)?
You can if you choose to, but it's probably not necessary. It can provide a level of validation for your application, guarding it against the potential of invalid XML causing it to fail.
However, keep these important points in mind:
- The process of validating XML against a schema is time consuming, it is a CPU-intensive process that might have a noticeable impact on your throughput and performance.
- In theory, the XML given to you by ESS will always validate against the schema. The schema is authored by Omnitracs and we ensure our XML validates against the schema.
What Should I Do if the ESS XML Appears to be Invalid or Does Not Adhere to the Schema?
This circumstance should, in theory, never happen, but no one is completely immune to bugs.
If you encounter XML that does not appear valid or does not validate against the schema, we suggest you log the offending XML and contact Omnitracs customer support with detailed information.
Why Have You Implemented the Transactional Interface as XML Anyway?
XML has numerous advantages which we wanted to benefit from:
- XML is platform-independent, language-independent, vendor-independent.
- XML is flexible and extensible.
- XML is human-readable.
- XML contains both data and meta-data. The data it contains is self-describing and self-documented.
- XML contains the structure along with the data.
- XML can be manipulated and processed by many tools which are readily available for most platforms.
Choosing an XML Parser
What IS an XML Parser?
By this point, you already know what XML is. An XML parser is really just an API (application programmer's interface) that lets you get at the data contained in the XML. It allows you to traverse the XML document and procces its content by giving you functions and methods that let you:
- Navigate the XML tree (parent-child element nodes and their attributes)
- Jump to spots in the XML tree
- Find nodes in the tree by name
- Extract data from elements, attributes
Without the XML parser API, all you have is a big string that contains XML. You'd then have to scan it manually using string manipulation functions. An XML parser will "understand" XML and give you easy access to it at an API level without you having to worry about the beginning and end of elements and attributes and all the required XML markup (i.e. open and close angle brakets, etc).
What's the Best Way to Parse the XML Returned by ESS?
The best approach, with few exceptions, is to utilize an XML parser API that's available on your platform. In virtually all cases, this will simplify your development and allow your integration code to concentrate on the transaction data rather than the XML markup and transaction structure. We use the term "parser" generically here - the XML parsers available on most platforms are typically wrapped by higher-level APIs which simplify their use.
Should I Be Using a SAX or DOM Based Parser to Parse the XML Returned by ESS?
Yes.
You should be using a well-tested, industry-standard XML parser API to work with the XML. SAX and DOM parsers both fit the bill here. Both are well known, well documented, freely available, and highly reliable.
Based on your needs, one variant may be a better fit than the other. Based on your platform/environment, one may be easier to install/configure/use than the other. But both are more than acceptable.
What's the Basic Difference Between a SAX and DOM Based Parser?
The most prominent difference between the two is that SAX (Simplified Api for Xml) is more lightweight (and typically higher performing), while DOM (Document Object Model) is more feature-rich (and typically more resource intensive).
DOM uses a hierarchical object model to represent the data contained within the XML. That typically means loading the XML document into memory by creating linked objects corresponding to the nodes in the XML document.
SAX instead streams through the XML from beginning to end and passes control to your custom callback procedures when it encounters XML objects you're interested in processing (ex: start of element, end of element, element data, etc). Certain platforms expose additional API layers over these parsers, which does away with the complexity of registering and implemeting callback procedures.
It's important to note that the XML parsing APIs available on certain platforms often do not specifically mention or focus on the type of underlying XML parser they're based on. In certain cases the question of whether to choose between a SAX-based parser or DOM-based parser or some other type of parser is moot and the decision is based on the optimal XML API recommended for your platform.
The advantages and disadvantages listed below may not matter to your specific needs.
SAX Advantages
- Typically uses much less memory than DOM since it basically streams through the XML from start to finish like a string
- Runs much faster than DOM, several times faster.
SAX Disadvantages
- Forward-only, can't navigate to any location of the XML document, must process the document sequentially, from the start, through the end.
- Read-only, can't insert XML tags into the document, can't alter the XML document. This may be necessary if you wish to enhance the XML that ESS gives you before sending it elsewhere.
- May be more challenging to a less experienced programmers depending on the platform and the API available.
- You must provide your own data container for the data that comes out of the XML document.
DOM Advantages
- Full random access to the entire XML document, you can skip to any section, move backward and forward, search for specific locations.
- Ability to update the XML document if you wish, add new XML tags, modify existing tags.
- May be less challenging to a less experienced programmers depending on the platform and API available.
- Container for the data is built-in to the DOM API.
DOM Disadvantages
- Uses several times more memory than SAX.
- Several times slower than SAX.
Working with XML Parsers
Do I Still Have to Write Custom Integration Code If I Use an XML Parser?
Yes. Any XML parser will only go as far as exposing the XML document as a logical tree to you, and give you access to nodes and to their contents. You still need to write logic to:
- Navigate through the node tree.
- Tell the parser to skip nodes you don't care about.
- Tell the parser to move on to the nodes you do care about.
- Maintain stateful awareness of your location in the document.
- Maintain stateful awareness of the parent-child relationships.
- Extract data from nodes that matter to you.
- Perform business logic on the data you extract from nodes.
- Make decisions based on the contents of nodes
The parser, even the best of them, will only serve as a tool that you use to get to the relevant data. You still need to write code to use the tool properly, and perform business logic once you've put the tool to use.
Should I Write a Custom Parser to Handle the XML Returned by ESS?
Depending on your platform and the tools available, your needs may require you to write a custom parser. Most of the mature development environments and languages have native support for XML handling, but you may find yourself in a unique situation.
You'll need to ensure your parser embodies all the qualities that off-the-shelf DOM and SAX parsers strive to maintain:
- Flexibility
- Extensibility
- Loose coupling
- Able to differentiate data from markup
- Ignore unexpected attributes/elements
- Retain generic approach
- Assume no element/attribute ordering or positional relationships
If at all possible, it's better to use an off-the-shelf XML parser, meant for your platform, language, or environment. Only as an absolute last resort should you investigate writing your own custom parser.
Can You Provide Some Guidelines on How to Write a Custom Parser for ESS XML Transactions?
Your parser will have to account for many things, including (but not limited to) :
- Stateful awareness - You will need to be aware of your relative location within the XML document. For example, you may encounter an attribute named "ID", but without knowing which element you're currently inside of (or how deep within a set of nested elements), you won't know what the "ID" attribute refers to. Stateful awareness also means you always know what kind of transaction you're dealing with, and how much of it you've already processed.
- Markup vs Data - There may be an attribute named "vehicle", and also an element whose data contains the word "vehicle". You will need to know the difference between the two. This also classifies as "stateful awareness". If you're treating the XML as a string, markup looks the same as data to you, so you have to be aware of "where you are".
- Node Types - You will have to be able to handle all the different node types that XML can throw at you. The most common (and the only ones currently used by ESS) are elements and attributes, but that's not guaranteed to remain unchanged.
- Hierarchy, Parent-Child Locations - Elements can be nested many levels deep. You must be able to account for multiple levels of nesting. At any point you must know not only what element you're inside, but what its parent is, etc.
- Decoupled from Schema - Don't hardcode logic based on the current schema. New things will be introduced, new tags, new attributes, new combinations. Flexibility on your part is crucial. If you are bound tightly to the current version of the schema, you will have to recode every time the schema changes. Better to be flexible, generic, extensible.
Can I Parse the ESS XML Simply by Scanning for XML Tags I'm Interested In, Treating it Like a String?
Yes, but only if you absolutely have no other way of doing so. We strongly recommend using an off-the-shelf or native XML parser API if at all possible. XML is by definition extremely flexible, and so must any parser be. With the wealth of platform-specific parsers available (many freeware), it's hard to find an environment where you can't find one.
If you have no alternative, you can always treat the XML just like a string. You can use basic string parsing to scan for tags that you are interested in, and then harvest the contents. We can't stress enough how careful you must be to be as loosely coupled to the schema as possible. Save yourself future effort, write your parser as generically as possible, to handle any circumstance that XML can throw you. Don't bind yourself to specific tag names or a specific ordering scheme. Be as tolerant of schema changes as possible, and build in reliability so that unexpected XML doesn't force you to crash.
Or just use an off-the-shelf XML parser or API.
Backward Compatibility
How Can I Write Integration Code that Won't Break when the ESS Schema Changes?
Your job will be simplified if you use a parser API available on your platform, this will allow you to focus on the integration code. Here are a couple of other tips:
- Process what you know, ignore what you don't. Write your inegration code to skip over (ignore) elements and attributes which are unknown to you or irrelvant to your integration. This will allow your intgration code to be tolerant of future ESS XML schema changes involving new transactions types, new elements or new attributes.
- Don't rely on position, rely on the tags. You can cause problems if you introduce assumptions about the order that the tags will be in. If you assume that the elements or attributes follow a specific ordering scheme, you will most likely be wrong. It's better to scan the document looking for things by their tag name. If the tags are found, great. If not, then at least you can continue without crashing. If your integration requires you to process a transaction's content in a certain order, then extract and load the transaction data into an intermediate program data structure or object and then perform the order dependent processing.
- Treat optional elements/attributes as optional. If an element is declared as optional, be sure that your code will not crash if that element isn't present in a transaction. The same is true for attributes.
- Maximize Your Re-use. The ESS transaction schema make heavy re-use of certain complex elements such as <equipment> and <proximity>. You can save yourself time and write less code by creating corresponding data structures, functions or objects that handle those XML elements and attributes regardless of the transaction types they're associated with. Also, certain data such as timestamps are uniformly formatted across all ESS transactions so their conversion should be handled by a single common function. The same is true for position latitude and longitude information.
Exactly Which Parts of the ESS Schema Are Subject to Change?
The following aspects of the ESS schema are subject to change:
- New transaction types will be added (but none will be removed!)
- New elements may be added to existing transaction types (but none will be removed!)
- New attributes may be added to existing elements (but none will be removed!)
- Optional elements may change and become required (but not vice versa!)
Are There Parts of the ESS Schema that Can Never Change?
We consider backward-compatibility one of the most important aspects of ESS. Each release of the schema will be as compatible with existing integration as possible. Our goal is to break none of the existing integration during each new release.
To that end, these are some things that won't ever change:
- The ESS transaction data will always be contained within a <tranBlock> element.
- A single <tranBlock> element will always contain one or more <tran> elements
- A single <tran> element will always contain a single <T.n.n.n> element
- We won't ever remove a transaction type definition. We may, however, add additional ones.
- We won't ever remove an existing element or attribute. We may, however, add additional ones.
- Elements or attributes currently defined as "required" will never be made "optional".
How Long Will You Remain Backward Compatible with Previous Schema Versions?
We have no intention of ever breaking backward compatibility. Future versions of the schema should not break existing integration in place. Integrators will need to take steps to consume new data items revealed through the schema, but failure to do so the moment the new schema is released will not cause any harmful effects.
Can I Choose to "Stay Put" and Not Take Advantage of New Schema Changes?
Absolutely, you can "stay put". You don't have to subscribe to new transaction types. You don't have to handle new content (attributes, elements, data types, transaction types).
If you used an XML parser and your integration code is tolerant of changes (as previous described), you will not need to make any changes when new schemas are published. You may then add code to handle the new content whenever you're ready.
This is a key reason to ensure that your XML parsing includes the suggestions we've given in this FAQ. If the parsing is done right, additions to the schema will not break your existing integration.
Why Are Some Data Returned as XML Attributes, and Others as XML Elements?
Good question. There's not a really good answer. There doesn't appear to be an absolute standard for when to use attributes and when to use elements. If you Google it, you'll find plenty of differing opinions.
The good news is that it doesn't matter a lot. There's no "wrong" way, and it has minor impact no matter which way you do it.
We did our best to represent the data as accurately as possible in the most accessible and sensible manner.
Sample Code
Can You Provide a .NET-Based Example of ESS Integration Code that Uses an XML Parser?
Sure! Click below!
An Example Using .NET
Can You Provide an ILE RPG-Based Example of ESS Integration Code that Uses an XML Parser?
Sure! Click below!
An Example Using ILE RPG