Data Import Utility
Appendix F - Data Import Utility
This appendix describes the inter-platform data migration utility, which is a part of QTRACS/400. The migration characteristics, and file and record formats are described herein.
Included Data
It is possible to migrate only the following information to the QTRACS/400 databases using this utility. Note that only the 'major' data elements are migrated.
QTRACS Simple Entities:
- Users
- Vehicles
- Vehicle Message Groups
QTRACS Complex Entities (groupings of simple entities):
- Address Lists
- Vehicle Coverage Lists
Macro Template Information
- Forward Macro Templates
- Return Macro Templates
Message Information
Position History Information
Import Utility Characteristics
- Duplicate import data will not supersede existing data. Any duplicate data contained in the import shall be kicked-out to the exceptions file.
- The import utility performs data validation where necessary. For instance, the importation of message data requires that the message information (sender, addressees, macro number, etc.) be valid in the current QTRACS context. Sender and addressee entities must already exist. Macro templates must already exist.
- The import utility generates an exception file containing a copy of every record that was rejected during importation. The utility will fill in the transfer reason code (XFREASON) field for all rejected records. After resolving the reason for the rejection, and editing the exceptions file so that the data is correct, the exceptions file may be reprocessed via the very same import utility.
- The import utility produces a spooled audit log showing a summary of the import activity. This log will be found in the same location that all spooled files are sent for the user running the utility. The log will show up only after the utility has completed running.
- The utility will run only in batch mode.
OmniTRACS Account Issues
Certain QTRACS data is specific to the OmniTRACS Account. This means that there is data within QTRACS that is synchronized with data within the NMC. Examples of this data are vehicle group numbers and macro template definitions.
The data import utility was designed to support the migration of data into a new OmniTRACS account only.
This scenario involves the creation of a new OmniTRACS account into which existing QTRACS information (ideally from another platform or from another account) will be migrated. This also involves the transfer of vehicles from one account to another at the NMC. Two accounts will exist simultaneously.
The data import utility does not support the migration of data from an existing OmniTRACS account into the same OmniTRACS account. This scenario implies a one-shot migration from one platform to another in which the same OmniTRACS account will be used. All migration will be treated as if data was going to a new account. This means that the initial QTRACS/400 installation will be performed using standard methods. Data imported into the account will be treated as new information.
Invoking the QTRACS Data Import Utility
Before import can begin, data to be imported must first be placed on the target AS/400. All import data must be placed in a uniquely named member within the OMIMPORT file in the data library (OMNIDATA). The import file is a physical file (PF) with a single field of length 3000. This length accommodates the largest record format.
Using a standard QTRACS library list type the command OMIMPORT and press the F4 key to prompt. You will be asked to specify the name of the member in the OMIMPORT file containing the data that you wish to import to QTRACS/400. You will also be asked to specify the name of an exception file (a member of OMIMPORT) which will contain a log of all the records which did not get imported successfully. Note that the input member must exist, and the exception member must not exist, because it will be created during import.
After the import is complete, all exception file records will contain a reason code in the first four characters. This reason code corresponds to a message ID in the message file OMIMPDTAM. Append the letters "IMP" to the four character reason code, and find the corresponding message ID in the message file to determine what went wrong with the import of each particular record.
The exception file can be "reprocessed". Edit the exception file, fix the errors in the data, and save the exception file. You can then run the data import utility again, using the exception file as input, and naming a second exception file to contain any errors for this iteration. This process can be repeated until all errors have been removed from the input data.
Make sure that your OmniTRACS job subsystem (usually "QUALCOMM") is up and running when you invoke the data import utility. A connection with the HUB should be active when the importer runs, so that definitions of vehicles and vehicle message groups can be transmitted immediately to the HUB.
Import File Record Formats
The data import utility demands that the import files adhere to the following file-type and data-type rules.
When writing the non-AS/400-based export utilities, one must consider what transfer mechanism will be used to get the file to the AS/400. For instance, if the transferring mechanism is FTP, which handles ASCII to EBCDIC conversion, then the export utilities need not necessarily generate EBCDIC.
The bottom line is that regardless of how the import files arrive on the AS/400, they must adhere to the following specifications prior to being processed by the import utilities.
Data Types
Import data must be specified via the following data types only.
Numeric |
A positive numeric value represented as a string of EBCDIC characters. Valid characters are the numeric digits: |
Signed |
A signed numeric value represented as a string of EBCDIC characters. Valid characters are the numeric digits: |
Character |
Left-justified character string data in EBCDIC format padded with blanks. |
Time |
A 14-character (EBCDIC) value indicating the date and time in the form YYYYMMDDHHMMSS. Example: 19971222112713. The time zone is assumed to be the same as that which is defined in the QTRACS/400 Time Configuration. |
Note: All distances will be assumed to be in miles, not kilometers. The export utility must convert to miles if the source units are kilometers.
Optional Fields
Certain fields in the import record are required in order for the import to succeed. Required fields are marked with a in the following record definitions. The export utility must supply a valid value for every field in the record that is tagged as required. Optional fields are fields that are not required in order for the import to be successful. Character fields that are not included should be set to all blanks . Numeric fields that are not included should be set to blanks as well . A blank is defined as an EBCDIC space (hex 40).
Order of Definition
The order of the records in the import file is very significant because of the relationships of the entities. The recommended order of definition is as follows:
- Users
- Address Lists
- User to Address List Associations
- Vehicle Message Groups
- Vehicles
- Return Macros
- Forward Macros
- Messages
- Position History
- Drivers
Failure to follow this sequence could result in the rejection of huge amounts of import data.
QTRACS Import File Record Formats
In order to import the various types of entities to the QTRACS database, the import file can have several different types of record formats, each conveying information about a different type of entity.
All import records begin with the following fields:
Field ID | Description | Size | Data Type | Comment |
---|---|---|---|---|
XFREASON | Transfer failure reason code | 4 | Character | Reserved = "0000" |
XFACTION | Action Code | 4 | Character | A four-character value indicating the type of the export record. |
User Entity Import Record Format
The following table describes the User Entity import record. This record causes a new user entity to be added to the QTRACS database.
Note that not all of the fields defined in this input format are required. The optional fields will be replaced with data from the QTRACS Default User record when importing user data. Therefore, it is important that your QTRACS Default User record be set up to your specifications before beginning the user import.
Field ID | Description | Size | Data Type | Comment |
---|---|---|---|---|
XFREASON | Transfer failure reason code | 4 | Character | Reserved = "0000" |
XFACTION | Action Code | 4 | Character | "USER" |
EYID | User Identifier | 10 | Character | |
EYNAME | Name | 24 | Character | |
EYREDI | Redirected To Name | 10 | Character | |
EYCAPS | Capability Flags | 64 | Character | See table below |
EYMDRR | New Message Default Return Receipt | 1 | Character | 'N' = NO; 'Y' = YES |
EYMDPR | New Message Default Priority | 1 | Character | '0' = NORMAL; '1' = IMPORTANT; '2' = SLEEPY; '3' = SLEEPY+; '5' = ATTENTION; '9' = EMERGENCY |
EYMDRT | New Message Default Reply To | 10 | Character | '*DISPATCH'; '*PERSONAL'; or Coverage Identifier |
EYMDCL | Default Claim Messages | 1 | Character | 'N' = NO; 'Y' = YES |
EYMDIG | Default Ignore Claimed Messages | 1 | Character | 'N' = NO; 'Y' = YES |
EYMDAC | Default Message Mailbox Access | 10 | Character | '*ONLY' = my mailbox only; '*VEHICLE' = *ONLY plus vehicle mailboxes; '*ALL' = all mailboxes |
EYMDTO | Default Time Order | 1 | Character | 'A' = Ascending; 'D' = Descending |
EYMDVW | Default View | 1 | Character | 'N' = Normal; 'U' = Unseen; 'A' = All |
EYMDLC | Default Location | 1 | Character | 'C' = Current Location; 'M' = Message Location |
EYMNRV | Notify on Receipt from Vehicle | 1 | Character | 'N' = NO; 'Y' = YES with break msg; 'S' = YES with status msg; 'R' = YES but reply msgs only; 'W' = YES with break window |
EYMNRU | Notify on Receipt from User | 1 | Character | 'N' = NO; 'Y' = YES with break message; 'S' = YES with status message |
EYMMDL | Notify on Delivery | 1 | Character | 'N' = NO; 'Y' = YES with break message; 'S' = YES with status message |
EYMNUN | Notify on Undeliverable Message | 1 | Character | 'N' = NO; 'Y' = YES with break message; 'S' = YES with status message; 'M' = YES with QTRACS message |
EYMNRD | Notify on Message Read | 1 | Character | 'N' = NO; 'Y' = YES with break message; 'S' = YES with status message |
EYPQIV | Proximity Default Initial View | 1 | Character | 'A' = Alphabetic by ID; 'C' = Closest first; |
EYPQML | Proximity Default Mileage Limit | 5 | Numeric | 0-500 |
EYPQFL | Proximity Default Find Limit | 5 | Numeric | 0-500 |
EYPQLD | Proximity Default Look Direction | 3 | Character | N, S, E, W, NE, NW, SE, SW |
EYRLPP | Reference Limit to Preferred Place | 5 (1) | Numeric | 0-15.0 |
EYRLNS | Reference Limit to Next Stop | 5 | Numeric | 0-9999 |
EYRLBC | Reference Limit to Big City | 5 | Numeric | 0-999 |
EYPDIV | Place Directory Initial Sort | 1 | Character | 'N' = Name; 'G' = Geography; 'A' = Alias; 'P' = Postal Code; 'S' = Source |
EYPDIF | Place Directory Initial Filter | 5 | Character | '*ALL' = all types; or a valid place type |
EYVDIV | Vehicle Default Initial View | 10 | Character | '*ALL', or valid coverage or group |
EYTMZC | User Time Zone Code | 3 | Character |
User Capability Flags
The following table describes the user capability flags. Each flag occupies a single bit in the user import record. Set the bit off (0) to indicate that the user should not be granted the capability. Set the bit on (1) to indicate that the user should be granted the capability.
Capability Bit | Description |
---|---|
Byte1 Bit0 | Redirect message notifications of others |
Byte1 Bit1 | |
Byte1 Bit2 | Work with vehicle profiles |
Byte1 Bit3 | Work with vehicle coverage |
Byte1 Bit4 | Work with user profiles and preferences |
Byte1 Bit5 | Work with macro template definitions |
Byte1 Bit6 | Work with external application profiles |
Byte1 Bit7 | Send emergency priority messages |
Byte2 Bit0 | Resynchronize macros |
Byte2 Bit1 | Work with drivers |
Byte2 Bit2 | Work with system utilities |
Byte2 Bit3 | Add or change place information |
Byte2 Bit4 | Work with user capabilities |
Byte2 Bit5 | Send messages to *ALL vehicles |
Byte2 Bit6 | Work with binary message routing profiles |
Byte2 Bit7 | Work with QTRACS configuration |
Byte3 Bit0 | Work with SensorTRACS vehicle profiles |
Byte3 Bit1 | Work with SensorTRACS driver profiles |
Byte3 Bit2 | Work with SensorTRACS extraction cycle configuration |
Byte3 Bit3 | Work with SensorTRACS vehicle parameters |
Byte3 Bit4 | Work with SensorTRACS report generation |
Byte3 Bit5 | JTRACS Administrator |
Byte3 Bit6 | Send macro messages to vehicles |
Byte3 Bit7 | Send messages to un-grouped fleets |
Byte4 Bit0 | Send messages to the NMC |
Byte4 Bit1 | Claim messages addressed to others |
Byte4 Bit2 | Delete messages addressed to others |
Byte4 Bit3 | Delete messages that are un-seen |
Byte4 Bit4 | Work with vehicle phone book entries ** OBSOLETE ** |
Byte4 Bit5 | Work with fleet phone book entries ** OBSOLETE ** |
Byte4 Bit6 | Send freeform messages to vehicles |
Byte4 Bit7 | Send attention priority messages |
Byte 5 - all bits | Unused |
Byte 6 - all bits | Unused |
Byte 7 - all bits | Unused |
Byte 8 - all bits | Unused |
Address List Import Record Format
The following table describes the Address List Entity import record. This record causes a new address list entity to be added to the QTRACS database.
Field ID |
|
Description |
Size |
Data Type |
Comment |
---|---|---|---|---|---|
XFREASON |
? |
Transfer failure reason code |
4 |
Character |
Reserved = "0000" |
XFACTION |
? |
Action Code |
4 |
Character |
"LIST" |
EYID |
? |
Address List Identifier |
10 |
Character |
|
EYNAME |
? |
Name |
24 |
Character |
|
EYUSER |
? |
First User Member of Address List |
10 |
Character |
User ID |
Address List Association Import Record Format
The following table describes the Address List Association record. This record causes the user specified by EYUSER to become a member of the address list specified by EYLIST.
Field ID |
|
Description |
Size |
Data Type |
Comment |
---|---|---|---|---|---|
XFREASON |
? |
Transfer failure reason code |
4 |
Character |
Reserved = "0000" |
XFACTION |
? |
Action Code |
4 |
Character |
"AUSR" |
EYUSER |
? |
User Entity Identifier |
10 |
Character |
User ID |
EYLIST |
? |
Address List Entity Identifier |
10 |
Character |
Address List ID |
Vehicle Entity Import Record Format
The following table describes the Vehicle Entity import record. This record causes a new vehicle to be added to the QTRACS database. In addition, the appropriate transactions are sent to the NMC to define the vehicle and set the various mobile parameters. The latest position report is also retrieved for the vehicle.
Field ID | Description | Size | Data Type | Comment |
---|---|---|---|---|
XFREASON | Transfer failure reason code | 4 | Character | Reserved = "0000" |
XFACTION | Action Code | 4 | Character | "VEHL" |
EYID | Vehicle Identifier | 10 | Character | |
EYNAME | Entity Description | 24 | Character | |
EYMCT# | Comm Unit Address | 10 | Character | |
EYCVRG | Vehicle Coverage Name | 10 | Character | |
EYVMGN | Vehicle Message Group Name | 10 | Character | |
EYPSRC | Position Source Type | 5 | Character | |
EYMPR | MIPR Timeout | 7 | Character | '0000000' - '9995959' (HHH MM SS) |
EYPDT | Power Down Timer | 4 | Character | '0000'-'9999' |
EYASM | Screen Behavior | 2 | Character | '00','01','02','03' |
EYBCP1 | Normal Beep Timer Subsequent | 2 | Character | '00'-'63' |
EYBCP2 | Normal Beep Volume Subsequent | 1 | Character | '0','1','2' |
EYBCP3 | Emergency Beep Timer Subsequent | 2 | Character | '00'-'63' |
EYBCP4 | Emergency Beep Volume Subsequent | 1 | Character | '0','1','2' |
EYBCP5 | Normal Beep Volume Initial | 1 | Character | '0','1','2' |
EYBCP6 | Emergency Beep Volume Initial | 1 | Character | '0','1','2' |
EYTMZC | Mobile Time Zone | 3 | Character | |
EYINDS | Mobile in Daylight Savings Time | 1 | Character | 'Y','N' |
EYWTP1 | Wake up Timer Interval | 4 | Character | '0000','0060'-'9999' |
EYWTP2 | Total Wake up Power Time | 4 | Character | '0000'-'9999' |
EYDTTC | Capable | 1 | Character | '0','1' |
EYDTTE | Enable | 1 | Character | '0','1' |
Message Import Record Format
The following table describes the Message import record format. This record causes a message to be added to the QTRACS message database.
Field ID |
|
Description |
Size |
Data Type |
Comment |
---|---|---|---|---|---|
XFREASON |
? |
Transfer failure reason code |
4 |
Character |
Reserved = "0000" |
XFACTION |
? |
Action Code |
4 |
Character |
"MSSG" |
MHSRC |
? |
Message Source (Sender) |
10 |
Character |
Simple Entity ID |
MHPRI |
? |
Message Priority |
1 |
Character |
0=Normal; |
MHMAC# |
? |
Macro Number or Binary Data Type |
3 |
Numeric |
(See Note 1) |
MHRCPT |
? |
Return Receipt Request Flag |
1 |
Character |
'0'=No; |
MHRPLY |
? |
Reply-To Designator |
10 |
Character |
"*DISPATCH"; |
MHRCVR |
? |
Message Destination (Addressee) |
10 |
Character |
|
MHINDR |
? |
Indirection Flag |
1 |
Character |
0 = No Indirection, |
MLHUB# |
? |
Global Hub Reference Number |
11 |
Numeric |
GFMN or GRMN |
MHSENT |
? |
Time Message Sent |
14 |
Time |
|
MHRCVD |
? |
Time Message Received |
14 |
Time |
|
MHREAD |
? |
Time Message Read |
14 |
Time |
|
MHMLEN |
? |
Transmittable Message Length |
4 |
Numeric |
|
MHBTIN |
? |
Message Body Type |
1 |
Character |
B = Binary; |
MTBODY |
? |
Message Body |
2850 |
Character |
Transmittable format, |
Notes :
- The exporting software must insure that the macro template is current and valid when generating the message export record for macro messages. The data import utility will attempt to convert macro message bodies from their transmittable form into their displayable form based on the version of the macro template that is currently stored in the AS/400 database. The exporting software should convert expired macro message bodies into free format message bodies when it encounters macro messages for which the macro template has expired. The macro number should be set to zero (0), indicating free form, in these cases.
- The Global Message Handle (GMH or HMN) is not imported. It will be assigned by QTRACS/400 during the import. For example, a given message may have been #123456 in QTRACS/ESA, however, after it is imported to QTRACS/400 it may be known as #654321. All references via the 'old key' will be lost.
- The Indirection Flag (MHINDR) determines whether the recipient of the message was the entity named in the MHRCVR field (Receiver), or the vehicle(s) being covered by the entity named in the Receiver field. For example, if the Receiver is USER1 and the Indirection field is 0, the message was sent to the individual user named USER1. If the Indirection flag is 1, the message was sent to the vehicles that USER1 is covering.
Data Import Utility - Position History Import Record Format
The following table describes the Vehicle Position History import record. This record causes a vehicle position history record to be added to the QTRACS database.
Field ID | Description | Size | Data Type | Comment |
---|---|---|---|---|
XFREASON | Transfer failure reason code | 4 | Character | Reserved = "0000" |
XFACTION | Action Code | 4 | Character | "POSI" |
PHVEH | Vehicle Entity Identifier | 10 | Character | |
PHTIME | Position Time | 14 | Time | CCYYMMDDHHMMSS |
PHARRV | Position Arrival Time | 14 | Time | Blanks = same as PHTIME |
PHSIOT | Trip In/Out | 1 | Character | 'U' = Unknown; 'I' = In; 'O' = Out |
PHIGST | Ignition Status | 1 | Character | '0' = Unknown; '1' = On; '2' = Off; '*' = No Change |
PHLAT | Latitude (whole seconds) | 8 | Signed | |
PHLON | Longitude (whole seconds) | 8 | Signed |
Macro Template Import Record Format
The following table describes the Macro Template import record. This record causes a macro to be added or removed from the QTRACS database. In addition, transactions are sent to the NMC to in order synchronize the HUB database and the mobile unit's memory with each macro definition.
Field ID | Description | Size | Data Type | Comment |
---|---|---|---|---|
XFREASON | Transfer failure reason code | 4 | Character | Reserved = "0000" |
XFACTION | Action Code | 4 | Character | "AMAC" - Add Macro "RMAC" - Remove Macro |
FMDIR | Macro Direction | 1 | Character | F (forward) or R (return) |
FMMAC# | Macro Number | 3 | Numeric | |
FMNAME | Macro Name | 14 | Character | |
FMDESC | Macro Description | 30 | Character | |
FMASSO | Associated Return Macro Number | 3 | Numeric | Forward macro definitions only. |
FMCVRG | Return Macro Coverage Identifier | 10 | Character | Return macro definitions only. |
FMAPPL | External Application Name | 10 | Character | Return macro definitions only. |
FMVCVG | Route to Vehicle Coverage | 1 | Character | 'Y' = Yes; 'N' = No (Return only) |
FMBFMT | Format of Macro Definition | 1 | Character | 'F' = freeform 'T' = transmittable |
FMLENG | Length of Macro Definition | 5 | Numeric | Must be > 0 |
FMBODY | Macro Template Definition | 1900 | Character | May be freeform or transmittable. |
Notes:
- Freeform macros (FMMAC# = 0) may not be imported.
- Return macros should be imported before forward macros. (This is because some forward macros may reference a return macro.)
- FMASSO may only be specified on forward macros. Return macros must have this field set to blanks.
- FMCVRG and FMAPPL may only be specified on return macros. Forward macros must have these fields set to blanks.
Please be aware of the following items when deleting macros:
- QTRACS/400 will set the macro's status to "deleted" when the delete request is processed, however the record in the database only gets deleted after the NMC confirms the deletion.
- This process may take some time, considering the amount of traffic between the AS/400 and the NMC at the time that this utility is run.
- If the connection with the NMC is down none of the "deleted" macros will be removed from the database until the connection is reestablished. What this means is that if you remove macro #35, you may not get an error, but you still may not be able to re-add macro #35 yet !
Vehicle Message Group Import Record Format
The following table describes the Vehicle Message Group record. This record causes a Vehicle Message Group to be added to the QTRACS database, making use of one of the 49 vehicle group numbers available to each site.
Field ID |
|
Description |
Size |
Data Type |
Comment |
---|---|---|---|---|---|
XFREASON |
? |
Transfer failure reason code |
4 |
Character |
Reserved = "0000" |
XFACTION |
? |
Action Code |
4 |
Character |
"AGRP" |
GRUSER |
? |
Group ID |
10 |
Character |
|
GRNAME |
? |
Group Description |
24 |
Character |
|
Driver Entity Import Record Format
The following table describes the Driver Entity import record. This record causes a new driver entity to be added to the QTRACS database.
Field ID |
|
Description |
Size |
Data Type |
Comment |
---|---|---|---|---|---|
XFREASON |
? |
Transfer failure reason code |
4 |
Character |
Reserved = '0000' |
XFACTION |
? |
Action Code |
4 |
Character |
'DRVR' |
EYID |
? |
Driver Identifier |
10 |
Character |
|
EYNAME |
? |
Full Driver Name |
30 |
Character |
|
EYPWRD |
? |
Global Login Password |
8 |
Character |
|
EYSTID |
|
SensorTRACS ID |
9 |
Character |
Required if EYFLAG = 'Y' |
EYFLAG |
? |
Add Driver to SensorTRACS |
1 |
Character |
'N' = NO; |
Notes: If EYFLAG is set to 'Y' then the driver will be added to QTRACS only if the add to SensorTRACS is successful. In order for the SensorTRACS add to be successful, the SensorTRACS ID (EYSTID) must be specified.