Draft: Track job failures correctly
Up until now, we have used Health::Complete to indicate job success
in Gitlab pipelines, and Heath::Incomplete to indicate job failure.
This is not in fact how LAVA uses these health values, because whether
a job completed does not directly indicate whether the tests within it
passed. An incomplete job will always be a failure, because it will
have aborted. But a complete job may contain many (even all) test
cases having failed, even if the job itself did not abort.
The actual logic for whether the job should be reported to Gitlab as
failed is unfortunately slightly more complicated. We need to identify
results messages within the log, and check whether any of them are
failures. That means we need to make sure we check the logs, before we
report any successful status to Gitlab.
The biggest change is that the monitor mode now outputs the logs
(for all jobs) directly. There's no particular reason not to do this,
since it has to query the logs anyway in order to accurately report.
success or failure.
This MR is draft pending final testing of the submit command,
and full testing of the monitor command.