A Day in the Life of an Iridium SNOC Engineer

Ever wonder what it’s like to work at our Satellite Network Operations Center (SNOC)? Satellite Systems Engineer Ken Rock took us behind the scenes in August, giving us a peek at the typical types of engineering and operations activities he encounters at the SNOC in Leesburg, Virginia. In this post, he shares the ins and outs of ensuring that Iridium^® satellites perform optimally, allowing our network to continue providing global coverage for our subscribers all around the world.

Nine O’clock Operations Coordination
We are starting the week discussing current progress on Launch 7 slot swap, software loads, service activations, de-boost, and other activities we’ll undertake for the next several days. There is a lot going on at the SNOC, so in attendance were participants from every team. It is amazing how much information is exchanged in this meeting, yet it runs very smoothly since we’ve been conducting this meeting daily since Launch 1. Today, we see that SV153 is arriving at Plane 1 after a long drift. We will be working on maneuvers to stop the drift and ascend to mission, as well as complete particular activities to prepare a few satellite vehicles (SVs) for de-boost. We have to work quickly, though, because we want to accomplish these tasks before the SVs drop out of crosslinks. My team is also preparing for new satellite flight software – we need to get the software onboard, and then methodically test it to make sure it is working properly. The initial upload was planned to take place on a storage satellite, but we recognized an opportunity to load the software onto a mission SV. Using this other satellite will streamline the overall activity. Several offline discussions will happen to coordinate with the Planning, K-band, Systems, and Platform Software (PFSW) teams.

Preparing SV056 (Block 1) for its Last Software Upload
New flight software requires a compatibility procedure be run prior to resetting the computer, so we had to build, check, and execute that procedure with the Ops floor. The new software provides advanced de-boost capabilities to safely lower the satellite’s orbit and passivate the vehicle. We want the SV to re-enter the Earth’s atmosphere in a predictable manner, with no propellant onboard, Solar arrays in a high-drag configuration, and batteries depleted. This is how we ensure the satellite is de-boosted safely and we contribute to reducing space debris (or “space junk”), sustaining space for the future.

Reviewing De-boosting SV Performance
On the topic of de-boost, three SVs are currently in various stages of maneuvers. The Attitude Control Systems (ACS) and Orbit Analysis (OA) teams have automated data collection and processing performed many times per day. Data is processed, and a data package generated and emailed to team members, so they can follow along with the progress each SV is making. I like to check the orbit apogee and perigee, the fuel remaining, and the attitude transients encountered.

Lunch & Learn: Operations
Julian Horvath, an Operations Systems Engineer from our RimRock facility in Tempe, Arizona was kind enough to host a Lunch & Learn session for our SNOC interns. He discussed cost/benefit trades for engineering versus operations, and provided some insight on how work is divided between Iridium and our Iridium NEXT mission partner (and Iridium NEXT satellite manufacturer) Thales Alenia Space. One of the interns asked about how work varies at different-sized companies. Julian shared that when working at a company like Iridium, there is a lot of responsibility, but also a lot of opportunity.

Event Monitoring
I was asked by Ahsen Abbasi, Senior Technical Manager of Real-Time Operations, to monitor how many events of a certain type were expected per day. Currently the Ops team manually runs a procedure for each event, and recently had seen a slight uptick in occurrences. I queried data from several weeks of our vehicle reports and observed about 8 to 10 events per day. For 65 satellites in orbit, this is about the number I expected, although it was much fewer when we only had only 20 Iridium NEXT SVs on orbit last year. In our next version of software, these events are handled autonomously, so the Operations team will not have to respond. Interestingly, this event is correlated with space weather. As it turns out, a solar coronal hole has rotated towards Earth over the last few days, according to Spaceweather.com. Just today, the solar wind is arriving at Earth, and particles are interacting with our satellites. This is all normal – we just don’t see these space weather reports here on Leesburg TV, but this weather may trigger some nice auroras near our ground stations in Alaska, Yellowknife, or Svalbard.

Counting Bit Errors
We audited single bit errors for Satellite Platform computers recently. We trended all SVs and cumulatively plotted the number of error per SV per day. It was nice to see a linear plot of this data, indicating no degradation in memory over time. It was also interesting to see that small differences in orbit altitude affect the rate of bit errors (fewer errors at lower altitudes). One of our SNOC interns created a MATLAB tool that captured telemetry changes and plotted them over top of a world map. We expected to see most bit errors occurring over the South Atlantic Anomaly (SAA), but it was amazing to see such a tight grouping of data! Our counterparts at TAS are also tasked to review this data as part of on-orbit checkout. Data access has already been approved for them, but they do not have direct access to it, so we query the data and send it to them.

Collecting External Hosted Payload (HPL) Data
All of our Iridium NEXT satellites have a Hosted Payload (Aireon) onboard. Those payloads have a dedicated ground system to manage their operation, which was built in parallel with the primary Iridium satellite ground system. We recently identified an opportunity to improve operations by allowing these systems to better share information. Since Launch 1, we’ve always stored 30 days of historical HPL data that was available to process, but in order to analyze it, we had to dig into our long-term archives. When we retrieved the data, we found it stored in three different formats that had changed throughout development. We wrote a script to parse and format the data into a common intermediate format. Another script was written to process that data and add it to our databases. The next step will be to automate the entire data collection and integrate this process into standard ground system processing.

AMPERE NEXT Ground System Performance
AMPERE NEXT is a data program for The Johns Hopkins Applied Physics Laboratory (JH/APL). They are analyzing magnetometer sensed magnetic field perturbations and inferring electric current flow in the Earth’s Magnetosphere-Ionosphere. I have been refining the data processing system for AMPERE NEXT for the past several months. Generally, the data processing works well and sends data to JHU/APL within minutes of acquisition on the ground, but today I found a small gap in the data – about 350 samples (out of 10,800) were missed during real-time processing, and filled in shortly after during post-processing. I spent some time searching for the root causes of the data gaps, then made some notes and recommendations for code changes that will allow the system to avoiding this kind of gap in the future. I also spent some time writing a data archiver script for this data, as archiving has previously been done manually. The script was simple, in that it would “zip up” some data on one day, then move other “zipped” data to long-term storage. I tested the script on data that needed to be archived, and manually executed the script on a day that needed to be zipped/moved. I then wrote a change request to officially release the script and add it to scheduled processing for automatic execution. I was able to catch a few members of the Control Facility System Operators (CFSOs) during lunch to discuss the AMPERE processes. After lunch, I added content to new AMPERE NEXT training packages that were co-developed with Aaron O’Connell, one of our SNOC interns. Aaron was able to brief the training package to one of the four Real Time Operations (RTO) crews today. We’ll get another crew tomorrow, and the final two next week. Training for the AMPERE NEXT ground processes will be more important, as we begin to hand over control to RTO, who will support these ground systems tools 24×7 for years to come.

SV166 Computer Reset
Computers in space reset just like any other computer, but when those computers are satellites flying through space that, together, are worth over $3 billion, we look a little closer at the root cause. This involves analyzing any observed error codes, memory dumps, and satellite telemetry. We happened to be completing the reset while TAS engineers were still involved in SV checkout, so they were able to participate in the discussions and analysis. This type of analysis can get very detailed very quickly, especially when looking through software memory maps…and once you are in there, you find all kinds of interesting things to learn about, like reviewing executable assembly code, breaking out software binary files, analyzing data header structures, etc. In the end, we determined this event to be a single event upset (SEU), which is a random upset. This SV is destined for mission!

Thank you to Ken for giving us a peek into what a day in the life of a Satellite Systems Engineer looks like!