A bunch of us gathered to do a post mortem on LV2 this evening. We spread tarps and white sheets down on the FAB 155 table and began trying to do a "crash analysis" on the remains. It was pretty brutal: there wasn't a single "ok" piece of the rocket. Surface mount components we stripped off boards, circuit boards chipped into tiny pieces and conformally smushed into crevaces in the Aluminum... and there was almost zero evidence of the compact flash card we hoped to find. We think we found the carier board, but the CF card itself seems to have disintigrated.
Here's a movie with Brian and Andrew giving a walk through of the remains.
A few things we know now:
- Not a single component of LV2 can be reused.
- The compact flash is a complete loss. We can't even find the bulk of it.
- The nose cone was still attached to the rocket on impact.
- The recovery charge - the pyrotechnic gas generator - did ignite and burn because it charred the parachute deployment bags. We can't tell whether it was ignited in the air at apogee as expected, or whether it was triggered on the ground by the crash.
- Tim is going to investigate the line cutters... if they didn't fire, then we need to rethink what exactly happened. If they did fire, then we can conclude that the nose cone didn't come off for most likely mechanical reasons.
Composite image of LV2 spread out on the FAB 155 oval conference table, complete with white sheets.
For the original composite without the labels, click here.
Tim's analysis of the line cutters is complete. Here's his email, which pretty much sums up our best guess as to what happened:
At the impact site several incongruities presented themselves. All four electric matches were expected to have been fired, but the nose failed to detach and the linecutters appeared not to have actuated. Clearly the black powder nose separation charge had fired, but was the ignition the result of impact with the ground? Had the recovery electronics really fired the matches at the right time? A quick examination of the linecutters revealed that the cutter knives had broken free of the plastic shear pins that hold them in place. This is normal if the cutters have been actuated, but the primary linecutter still had the un-cut 4mm line running through it. Had the linecutter been initiated on time and subsequently failed to cut? There was a disturbing lack of burnt powder smell coming from the linecutters. What was going on here? Both linecutters were partially crushed and so could not be disassembled normally. On Friday i cut them apart, and have since inspected both of them. ------------- Observations: The primary cutter did not cut the 4mm line. The backup cutter had the line pulled out during impact, but i suspect it also failed to cut. Both linecutter's electric matches have been fired Both linecutters contain un-burnt smokeless (mostly nitrocellulose) powder grains. The 4mm line protruding from the primary cutter shows evidence of melting, indicating moderately high temperatures, but only on one side of the cutter. The line sticking out the other end of the same cutter is un-melted. The primary linecutter's knife, which is hardened steel, shows no evident damage and may actually be re-usable. The secondary cutter's knife still needs to be cut free of the body, so its condition is not completely known. -------------- Thoughts The quantity of un-burnt powder in both cutters is similar. The primary has 42 un-burnt grains by my count, the backup 43 grains. Both were loaded with 55(+/-)1 grains so about a dozen grains are missing from each cutter. I suspect that when the electric match fired, the grains adjacent to the match also ignited, but the grains further removed from the match failed to ignite. This scenario is consistent with several observations. When found after impact, the cutter knives were out of place, their shear pins were broken but there was no smell of powder and the line was not cut. An incomplete combustion of the powder might generate sufficient pressure to break the shear pins and move the cutter knife, but not enough pressure to cut the line. If the line doesn't cut the knife can't move forward far enough in the body to reach the vent holes and the combustion gases can't easily escape, so the characteristic smell would be absent or muted. Why would the smokeless powder fail to ignite if the electric match fired? The most convenient explanation is the altitude at which the match fired. Telemetry indicates that the linecutters fired approximately 21,000 ft above MSL. At that altitude atmospheric density is about 44% of sea level density. Although it is somewhat surprising to me, there may be insufficient gas at that altitude to effectively transfer heat between adjacent powder grains. We have been worried about this before in connection with the nose ejection charge, but never with the linecutters since their operational design altitude is only ~1000 ft AGL. I speculate the top of the fault tree looks like this: 1) Software Succeeds in commanding recovery node to fire pyrotechnics 2) Recovery node connects charged HV capacitor to terminal block 3) Ignition wires conduct current to electric match 4) Electric match fires 5) Powder adjacent to match burns 6) Combustion gases transfer heat to surrounding powder 7) Powder charge completes combustion 8) Ejection charge gas pressure pushes nose cone forward breaking shear pins 9) Nose cone clears PVC ejection charge holder 10) Recovery module pressurizes 11) Aluminum sleeve on nose cone clears recovery module under the combined influence of momentum and the pressurized recovery module Logs indicate that (1) took place on schedule. (2) relies on the testing history (zero failures) and the fact that the recovery node firmware checks for a charged HV capacitor before attempting to fire. The log indicates that all fire attempts completed successfully. The recovery node hardware design is capable of detecting most actual failures up to (4) but the firmware has not been implemented. Therefore the fault tree branches at (3) and (4). (3) relies on a pre-flight Ohmmeter check and an otherwise successful history. If (4) failed the fault would probably not be at (3) because of the redundant wires and matches to the ejection charge. The observed firing of the cutter matches would be evidence of success at (4) unless the cutter matches were triggered on impact. This seems plausible since the core of the match contains a small amount of shock-sensitive lead picrate compound, but impact triggering should have burnt all the powder (zero failures in all previous tests). Also the quantity of picrate is quite small and well protected in an epoxy dip, further protected inside the linecutter body. However, striking an electric match between hammer and anvil sets it off with a bang (hearing protection recommended). So, for the failure to be at (4) we must have the electric matches fire on impact (plausible buy not certain) and the linecutter powder charge burn incompletely near ground level (unlikely), therefore failure at (4) is unlikely. Based on all previous tests (5) is assumed. Early electric match tests explored the possibility of barely-initiated matches exhibiting slow burning and low peak temperatures, and some effect was seen, but it was slight. The recovery node is designed to deliver the firing energy at power levels many hundreds of times greater than the minimum required to overcome the slow-burn effect, so the match firings are expected to be consistently hot which should assure (5). (6) appears to me a probable failure. I have no other explanation for the incomplete burn of the linecutter powder charge. We have successfully used the present nose ejection charge system at about the same altitude AGL attempted this time, but several things unique to this launch make a complete powder burn less likely. The powder was old. Prepared, i think, for the February launch attempt. It may have been wet or partially oxidized. Also there is a slightly higher base altitude in Brothers. I haven't compared actual pressure and temperature at apogee from the two launches. (7) likely did not take place. Although the sleeve fit of the nose could have jammed and prevented the ejection of the nose cone, this seems unlikely because the system has been tested and found to be fairly reliable. Also the fit was hand tested just prior to launch and seemed ok. The extent of burning seen on the nomex drogue cloth is also consistent with incomplete combustion of the ejection charge. Had the ejection charge burned completely at altitude the heat transfered to the nomex cloth should have been on order what we have observed before, where the cloth survives mostly intact and can be re-used. If, however, the charge had gone off on impact with the cloth crushed against the burning charge in the heated wreckage, it makes sense that the cloth would burn through. I don't have any black powder. Anyone know for sure that black powder ignites when struck? Anyone got some powder and a hammer? (Just a pinch, head clear of recoil please ;) Given success at stage (7), (8) is virtually assured, the volume to which the ejection charge is confined before breaking the shear pins is quite small, and the ejection charge pressure in the confined volume large. The shear pins are four weak ~3/32 inch PVC rods. I don't think we have ever observed a problem breaking the shear pins. Once the nose cone moves forward the pressure separating the PVC ejection charge container is probably un-stoppable. Even if the nose cone were to jam at this point the combination of high force and well-centered point-of-action would probably move the nose forward and separate the charge container. Even if the nose fully jammed the pressure is probably sufficient to break the PVC pipe and pressurize the recovery module. Once the recovery module is pressurized, there is yet another probable failure mode. The pressurized gas in the module leaks out around the drouge deployment bag and subsequently through the module cutouts and down the aeroshell. Therefore if the nose cone hangs up, or the deployment bag or drouge cloth is not tightly packed, or possibly if the altitude is high enough that insufficient air is pre-existing in the module, the module pressure may neither peak high enough nor maintain long enough to fully eject the nose cone. This is probably bad design, and we have said previously it ought to be changed. My feeling is that this particular problem probably did not cause the ejection failure we saw this time, but i have no way in hand to know that for sure. -------------- Wrap Up The most likely causes of failure in order are: Incomplete combustion of the ejection powder charge Jamming of the nose cone combined with pressure loss in the recovery module Recovery node failure or associated wiring To increase the system reliability in future do these things (in priority order): Use only fresh or properly stored powder charges for flight Test ejection system under vacuum Re-design ejection system to avoid relying on whole-module pressurization Test ejection more often Finish recovery node firmware
Note that some people have noted that we probably should move away from gunpowder gas generators and move to cold gas systems; we may very well do that.