Troubleshooting: "Error: Project Status: EXECUTION_FAILED"

📌 Problem

In LP360, while attempting to process the sensor trajectory solution for a TrueView 3DIS or microdrones payload in the Cloud run location, the user encounters an error message stating, "POSPac Cloud Processor failed", or "Project Status: EXECUTION_FAILED" or "Sensor Trajectory Processing Failed", or "An unexpected error occurred running Local POSPacCloud.exe" when running Trajectory Processing through the Job Manager , where the Project upload was successful, but the project status is "Execution_Failed", and the status type is FAILURE.

🔍 Root Cause

"Execution_Failed" is a catch all error message that occurs when POSPacCloud was unable to successfully process a sensor trajectory solution.

✅ Probable Resolutions

1. Run the QC Only (Realtime) to review the input data

If we do not know the source of the issue, we can run the sensor trajectory processing with QC Realtime instead of Single Base. In this case, we will establish if the issue is related to the rover (.T04 file) or to the base station (RINEX file) data. 
Be aware that this tool allows for creating a geocoded point cloud using an RNAV solution instead of an SBET solution. The resultant dataset is not intended for final processing but is a handy QC tool since you can create a point cloud from a flight without needing a complete PPK solution. In this case we will use it as a troubleshooting tool.

1. On the Sensor tab of the ribbon, select sensor Trajectory Processing .
2. Select "QC-Only (Realtime)".
3. Process the sensor trajectory.
4. Review the QC report.

At this point we will see two possible scenarios:

1. The processing fails: There is an issue with the trajectory. Either contact support or repeat the flight.
2. The processing succeeds: In this case continue with the processing -> Auto Create Flight Lines -> Create TV trajectories -> Geocoding.
   • During "Create TV trajectories", If you see the error "Trajectory File Time Range Warning", it could mean that not all the trajectory was processed.
   • Geocoding, review that all the point cloud was generated. Special attention if you saw the issue "Trajectory File Time Range Warning".

1. Base station: If the point cloud generated covers all the flight area, then the issue is related to the base station. At this point you could review:
   • Base station coordinates: If the coordinates are wrong, sensor trajectory processing will fail. If you do not know the coordinates you can use Trimble Centerpoint RTX, OPUS, or other similar post-processing service to compute them.
   • Base station file: Review that the file is not corrupt, that it covers all the flight without gaps, is working in Static mode and recording at 1Hz.
   • Reprocess without base station: using PPRTX or Smartbase services. These are paid services, that allows for processing without a base station.

• Gap in the trajectory: If the sensor trajectory has a gap the flight will be divided in two parts. Before the gap and after the gap. Read the POSPac QC report to find the gap (usually it will be at or near the beginning of the flight). Change the trajectory processing start time in the processing range. Continue with the TV workflow, and review that all the point cloud was generated.
• Alignment missing/failure: If the pilot did not perform the alignment either at the beginning/end, or it failed, you may see the first or last flight line missing, since the trajectory (sbet) starts after the IMU has been initialized. In this case, all data missed is lost.

Probable Resolution #2

A common cause of the "Execution_Failed" error is an incorrectly entered base station coordinate. Verify the base station value entered is correct. Signs are positive North and East. The approximate distance to base station as shown on the Base Station Settings page of the POSPac Processing wizard must be within 50m.

Probable Resolution #3

Verify the header of the base station RINEX file contains an approximate position that is close to the value entered, and correct, base station location. If the approximate location is more than 50m off of the real position the solution will have trouble processing.

Probable Resolution #4:

Locate the "YYMMDD_HHMMSS.log" file from the Area_\Cycle_YYMMDD_HHMMSS\POS\POSPac_YYMMDD_HHMMSS folder corresponding to the attempted execution in POSPacCloud that resulted in the "Execution_Failed" and review the contents for the following messages in the reason section:

"Primary and base observables data are incompatible. Please check base data against POS data."

Verify the base station is sampled at 1Hz to match the sample rate on the TrueView system. If the sampling rate is higher, then you may attempt to use a program to resample the observations to 1Hz. Ensure a 1Hz base station is used going forward.

TEQC, delivered with LP360 can be used to resample your base RINEX file. For example:

"C:\Program Files\Common Files\LP360\teqc.exe" -O.dec 1s D:\MyProject\mybase.20o > D:\MyProject\mybase_1s.20o

There is also a GUI interface of TEQC for Windows available, if desired.

In addition to being 1Hz data, POSPac requires the base station observations be exactly on the one second interval. For Emlid receivers, which typically record at some fraction of a second, it means using the Time Rounding option in Emlid Studio when converting the base observation file to RINEX.

“Primary and DEFAULT base observables data do not overlap in time. Please check base data against POS data.”

POSPac found that the base station RINEX file did not completely cover the T04 from the rover.

1. Verify the Time of First Obs through Time of Last Obs in the header of the base RINEX covers the time shown in the rover (T04) as displayed on the first page of the sensor Trajectory Processing Wizard in LP360.
2. Verify the first and last observation in the RINEX file match those in the header.
3. Verify the observations exist in the RINEX file for the full extent of the rover (T04) as displayed on the first page of the sensor Trajectory Processing Wizard in LP360.

Header: YYYY MM DD hh mm ss.sssssss GPS TIME OF FIRST/LAST OBS 2019 12 11 21 8 1.0000000 GPS TIME OF FIRST OBS 2019 12 11 21 8 1.0000000 GPS TIME OF LAST OBS

Observations: YY MM DD hh mm ss.sssssss ? Observed Satellites 19 12 11 21 8 1.0000000 0 14G03G14G16G22G23G26G31G32R01R02R08R10

“Smart Select terminated because more than 200 files cannot be downloaded”

The error message indicates the base stations are too far apart from the project site, or one another. The site is not suitable for processing using SMARTBase. Use the CyclePath.csv file to review the validity of the site for SMARTBase processing by following the Checking a Flown Mission method in the section Using SMARTBase Cloud to post-process your sensor trajectory in LP360.

This error message may also indicate that the nearest stations to your project area have only hourly files, but not daily files available at the time you tried to process. SMARTBase only uses stations with daily files available, causing the Smart Selector to look further and further away from your area. Wait until the daily file spanning your collection is uploaded and made available, usually shortly after GPS midnight, before trying to process again using this method. Or select a suitable nearby station and download the hourly files, using the provider's FTP site or other data transfer service, to process using the single base option. Or, if suitable for your projects and areas, process using PP-RTX to post-process your sensor trajectory in LP360.

"pointID\\r\\nParameter name: Could not set the base station position because a station with ID 'DEFAULT' was not found in the project."

Verify the base station being used for processing has a data interval of 1 second to match the processing rate on the TrueView POS system. A lower sampling rate may be rejected by POSPac and lead to no base being imported in the project, hence, no base station position can be set.

POSPac also doesn’t like the base station RINEX file if it believes it to be kinematic. Search the RINEX observation file for the word "Kinematic" or "Marker" to see if they exist partway through the file, indicating the start of another observation.

Kinematic

Verify the options and data used to create the RINEX file. If you can fix the base RINEX, then you should be able to get LP360 to successfully batch the processing. A specific example is the T04 file from a Trimble R12i base station with the INS enabled.

Marker

There is a known error due to a bug with TPS2RIN that happens periodically with the RINEX conversion of TopCon receivers which causes header data in the satellite range observation data. If you delete these lines, then your data should process normally.

"Ephemeris data is unavailable. Please import POS data first."

This an old error message that was typically caused by a POSPac bug in versions 8.4 through 8.5, where the rover broadcast ephemeris data gets incorrectly clipped. It can be worked around by adjusting the POSPac Processing Offset (LP360 -> Project Settings -> EXP -> TrueView Experimental Settings) value to a larger number (Default = 30s). Suggested resolution was to use 3600 or 7200 if there is an issue. This issue should no longer occur with POSPac v8.6 or later.

"Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt."

Though misleading, this error message has been known to be related to a bad base station RINEX file, which causes POSPac to crash. A giveaway is an abnormal distance to base coordinate warning in the sensor Trajectory Processing Wizard. Check your base RINEX file using a RINEX check and repair utility, such as GFZRNX by GFZ Potsdam. If your Cycle is long enough for your region (10min for fast RTX regions and 30min for standard RTX regions), try processing using PP-RTX, or, download data from a nearby CORS.

"Cannot open PPF"

This is an old error message that was caused by the IMU on the system not being supported by the version of POSPac running in Cloud.

"Error 26: The required data could not be retrieved."

Error 26 may indicate an issue with the supplied base station observation file. Verify the observations in the base overlap with the time of the rover data.

"TDGL_GET_OUTPUT_FILES_FAILED"

The error message indicates that POSPac Cloud encountered an issue processing this dataset to completion. Try again later.

"Processing error: please view the appropriate message log for details."

Starting in LP360 v2023.1.25.0, LP360 downloads the processing files when an “execution_failed” in POSPac Cloud to enable users to review the messages in the various processing logs when the reason for the failure isn’t in the main log. For help on reviewing those logs, please refer to Probable Resolution#6 in the "unexpected error running POSPac" post.

📬 Need Help?

If you're still stuck, please Contact Support and submit the logs found in \Project\Area\Cycle\POS\POSPac_YYMMDD_hhmmss\CYYMMDD_hhmmss and transfer the archived Cycle to support via LP360 Cloud or upload using Upload to Support and select the option to include project logs for further analysis.