You can see it is attached, but loose up top. Is this some kind of test device left behind or not connected somewhere it needs to be?

 

Someone left part of their fuel pressure tester attached to your fuel rail?

Depending on the fuel pressure tester, some have an adapter that is needed to be used on particular engines. Seems that adapter was left on your engine. Your dust cap is missing now as well.

 

 

Dust cap?

 

There is a cap that goes onto the fuel pressure test port to keep dust and other debris from getting down into the Schrader valve.

Given that the fuel pressure tester adapter is on there currently, your dust cap is not. Should be easy enough to grab from a junkyard though if yours isn't somewhere in the vehicle.

 

Does it look like a big valve stem cap? I think I found it on the lip at the top of the fire wall.

 

Yes, same function as a tire valve stem cap, just quite a bit larger. You should be good then to just unscrew the adapter hose and the thread on the cap.