#3881 diagnostic tools: output most recent trace logs on crash

[Anonymous]

Originally created by: marta-lokhova
Originally owned by: graydon

This would be useful if we aren't able to reproduce the issue locally. The idea is that core keeps a circular buffer of trace logs that gets dumped on crash. This issue is tracking two tasks:

• [ ] Implementation of circular buffer and the log-on-crash mechanism
• [ ] Log audit to ensure helpful information is outputted. It's probably reasonable to focus on the transaction subsystem for this one (e.g. information about apply time, host functions called, etc)

[Anonymous]

Originally posted by: graydon

An interesting thing here is that .. spdlog has a facility for this, and it looks like we're not using it. They call it a "backtrace" (not the best term) but it's a ring buffer you can attach to the sink network, see spdlog::enable_backtrace and spdlog::dump_backtrace.

What's weird is that I totally knew that existed when I added spdlog and, I suspect, I might even have chosen it partly for that facility .. and yet we do not use it currently. Did I forget to enable it when landing? Was there a reason it doesn't work with our crash-handler? I am unclear on this. It seems like it should already be turned on if it can be.

[Anonymous]

Originally posted by: MonsieurNicolas

We probably would need a new log partition for this and then decide which things we'd like to dump in there (so that particular exercise may be quite time consuming).

I think trace level for everything (like transactions) adds a lot of overhead to nodes, even if they don't end up in log files:
it's a lot of allocations/string operations so we'd have to be careful with this.

[Anonymous]

Originally posted by: MonsieurNicolas

may be related to [#3980] that we probably need to do first

[Anonymous]

• assigned_to: graydon