dle-develop — Dependable Linux Effort mailing list for developers

[Dle-develop] disableprobe test

[Satoru M. <sat...@hd...>]

Hi,

This is a test for kprobes per-probe disabling support.
For details, please see README in tarball.

Thanks,
Satoru

Re: [Dle-develop] [PATCH 2/2] kmsg_dump: adding to reboot, halt, poweroff and emergency_restart path

[KOSAKI M. <kos...@jp...>]

>  We need to know the reason why system rebooted in support service.
>  However, we can't inform our customers of the reason because
>  final messages are lost on current Linux kernel.
>  
>  This patch improves the situation above because the final messages 
>  are saved by adding kmsg_dump() to reboot, halt, poweroff and 
>  emergency_restart path.
> 
>  Signed-off-by: Seiji Aguchi <sei...@hd...>

This patch doesn't have any notification consumer. It's too bad.
As far as I know, some poeple are now discussing to make generic
firmware access interface. perhaps your one can be integrate it.
Please see and join following discussion.


From: "Luck, Tony" <ton...@in...>
Subject: [RFC] persistent store

Re: [Dle-develop] [PATCH 1/2] kmsg_dump: adding to reboot, halt, poweroff and emergency_restart path

[KOSAKI M. <kos...@jp...>]

> This patch alters mtdoops and ramoops to perform their actions only for KMSG_DUMP_PANIC,
> KMSG_DUMP_OOPS and KMSG_DUMP_KEXEC because they would like to log crashes only.
> 
>  Signed-off-by: Seiji Aguchi <sei...@hd...>
> 
> ---
>  drivers/char/ramoops.c |    5 +++++
>  drivers/mtd/mtdoops.c  |    5 +++++
>  2 files changed, 10 insertions(+), 0 deletions(-)
> 
> diff --git a/drivers/char/ramoops.c b/drivers/char/ramoops.c index 73dcb0e..8998b29 100644
> --- a/drivers/char/ramoops.c
> +++ b/drivers/char/ramoops.c
> @@ -69,6 +69,11 @@ static void ramoops_do_dump(struct kmsg_dumper *dumper,
>  	char *buf;
>  	struct timeval timestamp;
>  
> +	if (reason != KMSG_DUMP_OOPS &&
> +	    reason != KMSG_DUMP_PANIC &&
> +	    reason != KMSG_DUMP_KEXEC)
> +		return;
> +
>  	/* Only dump oopses if dump_oops is set */
>  	if (reason == KMSG_DUMP_OOPS && !dump_oops)
>  		return;
> diff --git a/drivers/mtd/mtdoops.c b/drivers/mtd/mtdoops.c index 1ee72f3..c948150 100644
> --- a/drivers/mtd/mtdoops.c
> +++ b/drivers/mtd/mtdoops.c
> @@ -307,6 +307,11 @@ static void mtdoops_do_dump(struct kmsg_dumper *dumper,
>  	unsigned long l1_cpy, l2_cpy;
>  	char *dst;
>  
> +	if (reason != KMSG_DUMP_OOPS &&
> +	    reason != KMSG_DUMP_PANIC &&
> +	    reason != KMSG_DUMP_KEXEC)
> +		return;
> +
>  	/* Only dump oopses if dump_oops is set */
>  	if (reason == KMSG_DUMP_OOPS && !dump_oops)
>  		return;

Looks good.
	Reviewed-by: KOSAKI Motohiro <kos...@jp...>

[Dle-develop] Hwziavit Ohlyedxu

[Wpdb <zz...@gx...>]

ÄúºÃ£¡ÖÐ\»ª\ËÕ\ÑÌ\»Æ\º×\Â¥µÈ£¡ÍêÈ«·Ö±æ³ö²»À´£¨Çë¿´¸½¼þ£©   lTD84j12v9vY864iPa29K5G7ZoLQdaeI278AfsV6e9bXs2D6450VZlNPR4n20Vng9kwxy12j1860X9B

Re: [Dle-develop] [PATCH 2/2] kmsg_dump: adding to reboot, halt, poweroff and emergency_restart path

[Artem B. <Art...@no...>]

On Wed, 2010-11-17 at 10:03 -0500, Seiji Aguchi wrote:
> We need to know the reason why system rebooted in support service.
>  However, we can't inform our customers of the reason because
>  final messages are lost on current Linux kernel.
>  
>  This patch improves the situation above because the final messages 
>  are saved by adding kmsg_dump() to reboot, halt, poweroff and 
>  emergency_restart path.
> 
>  Signed-off-by: Seiji Aguchi <sei...@hd...>

Looks ok,

Reviewed-by: Artem Bityutskiy <Art...@no...>

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [Dle-develop] [PATCH 1/2] kmsg_dump: adding to reboot, halt, poweroff and emergency_restart path

[Artem B. <Art...@no...>]

On Wed, 2010-11-17 at 10:00 -0500, Seiji Aguchi wrote:
> This patch alters mtdoops and ramoops to perform their actions only for KMSG_DUMP_PANIC,
> KMSG_DUMP_OOPS and KMSG_DUMP_KEXEC because they would like to log crashes only.
> 
>  Signed-off-by: Seiji Aguchi <sei...@hd...>

Looks OK,

Reviewed-by: Artem Bityutskiy <Art...@no...>
-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [Dle-develop] [PATCH 0/2] kmsg_dump: adding to reboot, halt, poweroff and emergency_restart path

[Andrew M. <ak...@li...>]

On Wed, 17 Nov 2010 09:58:18 -0500
Seiji Aguchi <sei...@hd...> wrote:

> Hi,
> 
> This series aims to develop logging facility for enterprise use.
> 
> It is important to save kernel messages reliably on enterprise system 
> because they are helpful for diagnosing system.
> 
> This series add kmsg_dump() to the paths loosing kernel messages.
> The use case is the following.
> 
> [Use case of reboot/poweroff/halt/emergency_restart]
> 
>  My company has often experienced the followings in our support service.
>  - Customer's system suddenly reboots.
>  - Customers ask us to investigate the reason of the reboot.
> 
>  We recognize the fact itself because boot messages remain in /var/log/messages.
>  However, we can't investigate the reason why the system rebooted,  because the last messages don't remain.
>  And off course we can't explain the reason.
>  
>  
>  We can solve above problem with this patch as follows.
>  Case1: reboot with command 
>    - We can see "Restarting system with command:" or ""Restarting system.".
> 
>  Case2: halt with command
>    - We can see "System halted.".
> 
>  Case3: poweroff with command
>    - We can see " Power down.".
> 
>  Case4: emergency_restart with sysrq.
>    - We can see "Sysrq:" outputted in __handle_sysrq().
> 
>  Case5: emergency_restart with softdog.
>    - We can see "Initiating system reboot" in watchdog_fire().
> 
>  So, we can distinguish the reason of reboot, poweroff, halt and emergency_restart.
> 
>  If customer executed reboot command, you may think the customer should know the fact.
>  However, they often claim they don't execute the command when they rebooted system by mistake.
> 
>  No evidential message remain on current Linux kernel, so we can't show the proof to the customer.
>  This patch improves this situation.
> 
> 
> The first patch alters mtdoops and ramoops to perform their actions only for KMSG_DUMP_PANIC,
> KMSG_DUMP_OOPS and KMSG_DUMP_KEXEC because they would like to log crashes only.
> 
> The latter patch adds kmsg_dump() to reboot, halt, poweroff and emergency_restart path.

Damn, that's a good changelog.  We can actually understand why you
wrote the patch, and see what its value is!

One thing: please don't send multiple patches with the same title. 
See Documentation/SubmittingPatches section 15.  I renamed these two
patches to

kmsg_dump: constrain mtdoops and ramoops to perform their actions only for KMSG_DUMP_PANIC

and

kmsg_dump: add kmsg_dump() calls to the reboot, halt, poweroff and emergency_restart paths

[Dle-develop] [PATCH 1/2] kmsg_dump: adding to reboot, halt, poweroff and emergency_restart path

[Seiji A. <sei...@hd...>]

This patch alters mtdoops and ramoops to perform their actions only for KMSG_DUMP_PANIC,
KMSG_DUMP_OOPS and KMSG_DUMP_KEXEC because they would like to log crashes only.

 Signed-off-by: Seiji Aguchi <sei...@hd...>

---
 drivers/char/ramoops.c |    5 +++++
 drivers/mtd/mtdoops.c  |    5 +++++
 2 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/drivers/char/ramoops.c b/drivers/char/ramoops.c index 73dcb0e..8998b29 100644
--- a/drivers/char/ramoops.c
+++ b/drivers/char/ramoops.c
@@ -69,6 +69,11 @@ static void ramoops_do_dump(struct kmsg_dumper *dumper,
 	char *buf;
 	struct timeval timestamp;
 
+	if (reason != KMSG_DUMP_OOPS &&
+	    reason != KMSG_DUMP_PANIC &&
+	    reason != KMSG_DUMP_KEXEC)
+		return;
+
 	/* Only dump oopses if dump_oops is set */
 	if (reason == KMSG_DUMP_OOPS && !dump_oops)
 		return;
diff --git a/drivers/mtd/mtdoops.c b/drivers/mtd/mtdoops.c index 1ee72f3..c948150 100644
--- a/drivers/mtd/mtdoops.c
+++ b/drivers/mtd/mtdoops.c
@@ -307,6 +307,11 @@ static void mtdoops_do_dump(struct kmsg_dumper *dumper,
 	unsigned long l1_cpy, l2_cpy;
 	char *dst;
 
+	if (reason != KMSG_DUMP_OOPS &&
+	    reason != KMSG_DUMP_PANIC &&
+	    reason != KMSG_DUMP_KEXEC)
+		return;
+
 	/* Only dump oopses if dump_oops is set */
 	if (reason == KMSG_DUMP_OOPS && !dump_oops)
 		return;
--
1.7.2.2

[Dle-develop] [PATCH 2/2] kmsg_dump: adding to reboot, halt, poweroff and emergency_restart path

[Seiji A. <sei...@hd...>]

 We need to know the reason why system rebooted in support service.
 However, we can't inform our customers of the reason because
 final messages are lost on current Linux kernel.
 
 This patch improves the situation above because the final messages 
 are saved by adding kmsg_dump() to reboot, halt, poweroff and 
 emergency_restart path.

 Signed-off-by: Seiji Aguchi <sei...@hd...>

---
 include/linux/kmsg_dump.h |    4 ++++
 kernel/printk.c           |    4 ++++
 kernel/sys.c              |    6 ++++++
 3 files changed, 14 insertions(+), 0 deletions(-)

diff --git a/include/linux/kmsg_dump.h b/include/linux/kmsg_dump.h
index 24b4414..2a0d7d6 100644
--- a/include/linux/kmsg_dump.h
+++ b/include/linux/kmsg_dump.h
@@ -18,6 +18,10 @@ enum kmsg_dump_reason {
 	KMSG_DUMP_OOPS,
 	KMSG_DUMP_PANIC,
 	KMSG_DUMP_KEXEC,
+	KMSG_DUMP_RESTART,
+	KMSG_DUMP_HALT,
+	KMSG_DUMP_POWEROFF,
+	KMSG_DUMP_EMERG,
 };
 
 /**
diff --git a/kernel/printk.c b/kernel/printk.c
index b2ebaee..c5648b4 100644
--- a/kernel/printk.c
+++ b/kernel/printk.c
@@ -1516,6 +1516,10 @@ static const char * const kmsg_reasons[] = {
 	[KMSG_DUMP_OOPS]	= "oops",
 	[KMSG_DUMP_PANIC]	= "panic",
 	[KMSG_DUMP_KEXEC]	= "kexec",
+	[KMSG_DUMP_RESTART]	= "restart",
+	[KMSG_DUMP_HALT]	= "halt",
+	[KMSG_DUMP_POWEROFF]	= "poweroff",
+	[KMSG_DUMP_EMERG]	= "emergency_restart",
 };
 
 static const char *kmsg_to_str(enum kmsg_dump_reason reason)
diff --git a/kernel/sys.c b/kernel/sys.c
index 7f5a0cd..3eab886 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -43,6 +43,8 @@
 #include <linux/kprobes.h>
 #include <linux/user_namespace.h>
 
+#include <linux/kmsg_dump.h>
+
 #include <asm/uaccess.h>
 #include <asm/io.h>
 #include <asm/unistd.h>
@@ -285,6 +287,7 @@ out_unlock:
  */
 void emergency_restart(void)
 {
+	kmsg_dump(KMSG_DUMP_EMERG);
 	machine_emergency_restart();
 }
 EXPORT_SYMBOL_GPL(emergency_restart);
@@ -312,6 +315,7 @@ void kernel_restart(char *cmd)
 		printk(KERN_EMERG "Restarting system.\n");
 	else
 		printk(KERN_EMERG "Restarting system with command '%s'.\n", cmd);
+	kmsg_dump(KMSG_DUMP_RESTART);
 	machine_restart(cmd);
 }
 EXPORT_SYMBOL_GPL(kernel_restart);
@@ -333,6 +337,7 @@ void kernel_halt(void)
 	kernel_shutdown_prepare(SYSTEM_HALT);
 	sysdev_shutdown();
 	printk(KERN_EMERG "System halted.\n");
+	kmsg_dump(KMSG_DUMP_HALT);
 	machine_halt();
 }
 
@@ -351,6 +356,7 @@ void kernel_power_off(void)
 	disable_nonboot_cpus();
 	sysdev_shutdown();
 	printk(KERN_EMERG "Power down.\n");
+	kmsg_dump(KMSG_DUMP_POWEROFF);
 	machine_power_off();
 }
 EXPORT_SYMBOL_GPL(kernel_power_off);
-- 
1.7.2.2

[Dle-develop] [PATCH 0/2] kmsg_dump: adding to reboot, halt, poweroff and emergency_restart path

[Seiji A. <sei...@hd...>]

Hi,

This series aims to develop logging facility for enterprise use.

It is important to save kernel messages reliably on enterprise system 
because they are helpful for diagnosing system.

This series add kmsg_dump() to the paths loosing kernel messages.
The use case is the following.

[Use case of reboot/poweroff/halt/emergency_restart]

 My company has often experienced the followings in our support service.
 - Customer's system suddenly reboots.
 - Customers ask us to investigate the reason of the reboot.

 We recognize the fact itself because boot messages remain in /var/log/messages.
 However, we can't investigate the reason why the system rebooted,  because the last messages don't remain.
 And off course we can't explain the reason.
 
 
 We can solve above problem with this patch as follows.
 Case1: reboot with command 
   - We can see "Restarting system with command:" or ""Restarting system.".

 Case2: halt with command
   - We can see "System halted.".

 Case3: poweroff with command
   - We can see " Power down.".

 Case4: emergency_restart with sysrq.
   - We can see "Sysrq:" outputted in __handle_sysrq().

 Case5: emergency_restart with softdog.
   - We can see "Initiating system reboot" in watchdog_fire().

 So, we can distinguish the reason of reboot, poweroff, halt and emergency_restart.

 If customer executed reboot command, you may think the customer should know the fact.
 However, they often claim they don't execute the command when they rebooted system by mistake.

 No evidential message remain on current Linux kernel, so we can't show the proof to the customer.
 This patch improves this situation.


The first patch alters mtdoops and ramoops to perform their actions only for KMSG_DUMP_PANIC,
KMSG_DUMP_OOPS and KMSG_DUMP_KEXEC because they would like to log crashes only.

The latter patch adds kmsg_dump() to reboot, halt, poweroff and emergency_restart path.

[Dle-develop] Rlougxs Bklslj

[Jthayu <bir...@ya...>]

qbCppKnQqaSp0KmkqdCppKnQqaSp0KmkqdCppKnQqaSp0KmkqdCppKnQqaSptA0KqaaxvqmqueOp
qsnDqarS1KmqvvmpqsGqqaqhoamqo82pqqPDqaqj06mqo8Kppg0Kqaa5q6mquOapqrOkqarJz6mq
v8mpqs+1qaqhoamqo9KpqqPPqaqjzamqo8qppg0KqabLvqmqt9GpqrT6qarGsamqyc+pqs31qaqh
oamqo8GpqqPIqaqj2amqo8appg0Kqaa/yamqoaKpqsDtqaq+3amqzfipqsn6qaqhoamqo8WpqqPL
qaqjzKmqo8Kppg0KqabNqKmqu+Gpqsqxqaq++amqsumpqqGhqaqhoamqo9mpqqPWqaqj0amqo8up
pg0Kqaa5/amqzvGpqrzkqarOqqmq0a+pqrXnqaqhoamqo9CpqqPFqaqjwamqo9Wppg0Kqaa52Kmq
t9GpqrOkqaq496mqu/Kpqruwqaqhoamqo9qpqqPGqaqjwqmqo9appg0KqabPtamqoaKpqqOsqaq1
pamqtb2pqqOxqaqhoamqo8mpqqPIqaqjz6mqo9eppg0KqabTxamqt/6pqrfRqarOu6mqy7CpqqOz
qaqhoamqo8ipqqPJqaqj0Kmqo9Wppg0Kqaa73amqzvGpqtPDqarU2qmqzvGpqqO1qaqhoamqo8Kp
qqPKqaqj0amqoaGppg0Kqaa0+qmq0rWpqtanqarLsKmq0empqqO5qaqhoamqo8ypqqGhqaqj06mq
oaGppg0Kqaa/qqmqvLCpqrP2qarO8amq1qSpqqOwqaqhoamqo9apqqGhqaqj1KmqoaGppg0Kqaa4
96mqycypqri0qaq+1qmquvOpqqOzqaqhoamqo8WpqqGhqaqj1amqoaGppg0Kqaa12Kmq0rWpqtTT
qarL+amquLapqqO0qaqhoamqo8+pqqGhqaqj1qmqoaGppg0Kqaa3oqmqz/qpqrXEqarJ6qmqv+6p
qqOyqaqhoamqo9ipqqGhqaqj2KmqoaGppg0KqabGr6mqytupqsavqarB7Kmqo6GpqqOxqaqhoamq
oaGpqqGhqaqj2amqoaGppg0Kqaaho6mqoaGpqr7dqaqho6mqoaGpqqO5qaqhoamqoaGpqqGhqaqj
16mqoaGppg0KqaahoamqoaGpqqGhqaqhoamqoaGpqqO1qaqhoamqoaGpqqGhqaqhoamqoaGppg0K
qaahoamqoaGpqqGhqaqhoamqoaGpqqGhqaqhoamqoaGpqqGhqaqhoamqoaGppg0KqaahoamqoaGp
qqGhqaqhoamqoaGpqqGhqaqhoamqoaGpqqGhqaqhoamqoaGppg0KqbippKnYqaSp2KmkqdippKnY
qaSp2KmkqdippKnYqaSp2KmkqdippKnYqaSpvA0K
__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Dle-develop] [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Artem B. <Art...@no...>]

On Wed, 2010-11-03 at 22:50 +0100, ext Aaron Durbin wrote:
> Hope that helps others know that we think such a mechansim is vital. I 
> can post the patches for the common infrastructure if people are interested.

I think Andrew replied that this patch is OK, it only needs some
tweaking. Just tweak it and send out.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [Dle-develop] [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Aaron D. <ad...@go...>]

On 10/27/10 12:44, Seiji Aguchi wrote:
> Hi,
>
>> What actual problem are we solving here?  Why is the current code
>> inadequate?  It would help to demonstrate some use-case and to explain
>> how the situation improved with this patch.
>
> [Purpose]
>   My purpose is developing highly reliable logging facility for
enterprise use.
>
>   I'm planning to add the following triggers of kmsg_dumper().
>      - reboot/poweroff/halt/emergency_restart (this patch)
>      - Machine check
>
>   I'm also planning to add an feature outputting kernel messages to
NVRAM,
>   because NVRAM is equipped with enterprise servers.
>   We can realize highly reliable logging facility by outputting
kernel messages to NVRAM.
>   (NVRAM is commonly used on Mainframe and Commercial Unix as well.)
>
> [Use case of reboot/poweroff/halt/emergency_restart]
>
>   My company has often experienced the followings in our support service.
>   - Customer's system suddenly reboots.
>   - Customers ask us to investigate the reason of the reboot.
>
>   We recognize the fact itself because boot messages remain in
/var/log/messages.
>   However, we can't investigate the reason why the system rebooted,
>   because the last messages don't remain.
>   And off course we can't explain the reason.
>
>
>   We can solve above problem with this patch as follows.
>   Case1: reboot with command
>     - We can see "Restarting system with command:" or ""Restarting
system.".
>
>   Case2: halt with command
>     - We can see "System halted.".
>
>   Case3: poweroff with command
>     - We can see " Power down.".
>
>   Case4: emergency_restart with sysrq.
>     - We can see "Sysrq:" outputted in __handle_sysrq().
>
>   Case5: emergency_restart with softdog.
>     - We can see "Initiating system reboot" in watchdog_fire().
>
>   So, we can distinguish the reason of reboot, poweroff, halt and
emergency_restart.
>
>   If customer executed reboot command, you may think the customer
should know the fact.
>   However, they often claim they don't execute the command when they
rebooted system by mistake.
>
>   No evidential message remain on current Linux kernel, so we can't
show the proof to the customer.
>   This patch improves this situation.
>
> Seiji

We carry patches in our kernels that do very similar things. The reason 
is essentially the same as what you have cited. On our platforms we have 
two different ways of storing events to an event log. One communicates 
with the BIOS itself; the other writes bit flags to a known area of 
non-volatile storage. That way when the machine comes back up we have a 
clear eventlog (with times) as to what happened when. Piecing these 
events together has proven to be invaluable for finding issues.

For both of the drivers that log these events they use a shared 
interface that collect various events in the kernel and present them 
through a single notifier chain for the drivers' consumption.

The things we currently track and log are the following:
- clean reboot/shutdown
- panic
- oops
- die
- NMI watchdog

An example eventlog produced by our systems looks like the following 
(63-67 are the boot numbers of the system in question):

2010-10-14 10:26:06 | System Reset | 63
2010-10-14 10:26:19 | System boot | 63
2010-10-14 11:36:43 | Kernel Shutdown | 63 | Unknown Shutdown Reason
2010-10-14 11:36:43 | System Reset | 64
2010-10-14 11:36:56 | System boot | 64
2010-10-18 14:51:54 | Kernel Shutdown | 64 | Clean
2010-10-18 14:52:38 | System Reset | 65
2010-10-18 14:52:51 | System boot | 65
2010-10-26 02:44:48 | Kernel Shutdown | 65 | Oops
2010-10-26 02:44:48 | Kernel Shutdown | 65 | Die
2010-10-26 02:44:49 | Kernel Shutdown | 65 | Panic
2010-10-26 02:45:43 | System Reset | 66
2010-10-26 02:45:56 | System boot | 66
2010-10-26 02:49:22 | Kernel Shutdown | 66 | Clean
2010-10-26 02:50:05 | System Reset | 67
2010-10-26 02:50:18 | System boot | 67
2010-10-26 11:39:20 | Kernel Shutdown | 67 | Clean

Hope that helps others know that we think such a mechansim is vital. I 
can post the patches for the common infrastructure if people are interested.

-Aaron

Re: [Dle-develop] [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Artem B. <ded...@gm...>]

On Wed, 2010-10-27 at 16:35 -0700, Andrew Morton wrote:
> > > I think mtdoops user don't want to dump logs if a system reboot/poweroff
> > > safely because mtd device have write count limitation. and they are main
> > > user of kmsg_dump().
> 
> mtdoops was the only user until ramoops came along.
> 
> > Right, at least this is how we use mtdoops - we log crashes.
> 
> Fair enough.  So I suppose the existing clients (mtdoops_do_dump and
> ramoops_do_dump) should be altered to inspect the `reason' argument,
> and return if it isn't KMSG_DUMP_OOPS, KMSG_DUMP_PANIC or
> KMSG_DUMP_KEXEC (I assume?)

I agree, then mtdoops/ramoops can be tweaked to handle other messages.

> If that sounds suitable then I'd ask Seiji to update and resend the
> patch, along with a complete changelog as earlier dicussed.

Yes, I think the patch makes perfect sense, just the existing
functionality should not be changed as a side effect.

> I'd suggest doing it as two patches, the first of which alters mtdoops
> and ramoops to perform their actions only for the appropriate `reason'
> values.

Right, thanks.

-- 
Best Regards,
Artem Bityutskiy (Битюцкий Артём)

Re: [Dle-develop] [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Andrew M. <ak...@li...>]

On Tue, 19 Oct 2010 11:51:12 +0300
Artem Bityutskiy <Art...@no...> wrote:

> On Tue, 2010-10-19 at 10:52 +0200, ext KOSAKI Motohiro wrote:
> > > Hi,
> > > 
> > > Final messages from reboot, halt and poweroff aren't output to disk because klogd/syslogd 
> > > has been killed and root file system has been turned read-only.
> > > 
> > > Final messages from emergency_restart aren't output to disk as well because system may
> > > reboot before klogd/syslogd outputs messages to disk.
> > > 
> > > Therefore, it is better to put kmsg_dumper in reboot/halt/poweroff/emergency_restart path.
> > > Any comments/advices are welcome.
> > 
> > I think mtdoops user don't want to dump logs if a system reboot/poweroff
> > safely because mtd device have write count limitation. and they are main
> > user of kmsg_dump().

mtdoops was the only user until ramoops came along.

> Right, at least this is how we use mtdoops - we log crashes.

Fair enough.  So I suppose the existing clients (mtdoops_do_dump and
ramoops_do_dump) should be altered to inspect the `reason' argument,
and return if it isn't KMSG_DUMP_OOPS, KMSG_DUMP_PANIC or
KMSG_DUMP_KEXEC (I assume?)

If that sounds suitable then I'd ask Seiji to update and resend the
patch, along with a complete changelog as earlier dicussed.

I'd suggest doing it as two patches, the first of which alters mtdoops
and ramoops to perform their actions only for the appropriate `reason'
values.

Re: [Dle-develop] [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Seiji A. <sei...@hd...>]

Hi,

> What actual problem are we solving here?  Why is the current code
> inadequate?  It would help to demonstrate some use-case and to explain
> how the situation improved with this patch.

[Purpose]
 My purpose is developing highly reliable logging facility for enterprise use.

 I'm planning to add the following triggers of kmsg_dumper().
    - reboot/poweroff/halt/emergency_restart (this patch)
    - Machine check

 I'm also planning to add an feature outputting kernel messages to NVRAM, 
 because NVRAM is equipped with enterprise servers.
 We can realize highly reliable logging facility by outputting kernel messages to NVRAM.
 (NVRAM is commonly used on Mainframe and Commercial Unix as well.)

[Use case of reboot/poweroff/halt/emergency_restart]

 My company has often experienced the followings in our support service.
 - Customer's system suddenly reboots.
 - Customers ask us to investigate the reason of the reboot.

 We recognize the fact itself because boot messages remain in /var/log/messages.
 However, we can't investigate the reason why the system rebooted, 
 because the last messages don't remain.
 And off course we can't explain the reason.
 
 
 We can solve above problem with this patch as follows.
 Case1: reboot with command 
   - We can see "Restarting system with command:" or ""Restarting system.".

 Case2: halt with command
   - We can see "System halted.".

 Case3: poweroff with command
   - We can see " Power down.".

 Case4: emergency_restart with sysrq.
   - We can see "Sysrq:" outputted in __handle_sysrq().

 Case5: emergency_restart with softdog.
   - We can see "Initiating system reboot" in watchdog_fire().

 So, we can distinguish the reason of reboot, poweroff, halt and emergency_restart.

 If customer executed reboot command, you may think the customer should know the fact.
 However, they often claim they don't execute the command when they rebooted system by mistake.

 No evidential message remain on current Linux kernel, so we can't show the proof to the customer.
 This patch improves this situation.

Seiji

Re: [Dle-develop] [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Artem B. <Art...@no...>]

On Tue, 2010-10-19 at 10:52 +0200, ext KOSAKI Motohiro wrote:
> > Hi,
> > 
> > Final messages from reboot, halt and poweroff aren't output to disk because klogd/syslogd 
> > has been killed and root file system has been turned read-only.
> > 
> > Final messages from emergency_restart aren't output to disk as well because system may
> > reboot before klogd/syslogd outputs messages to disk.
> > 
> > Therefore, it is better to put kmsg_dumper in reboot/halt/poweroff/emergency_restart path.
> > Any comments/advices are welcome.
> 
> I think mtdoops user don't want to dump logs if a system reboot/poweroff
> safely because mtd device have write count limitation. and they are main
> user of kmsg_dump().

Right, at least this is how we use mtdoops - we log crashes.

-- 
Best Regards,
Artem Bityutskiy (Артём Битюцкий)

Re: [Dle-develop] [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[KOSAKI M. <kos...@jp...>]

> Hi,
> 
> Final messages from reboot, halt and poweroff aren't output to disk because klogd/syslogd 
> has been killed and root file system has been turned read-only.
> 
> Final messages from emergency_restart aren't output to disk as well because system may
> reboot before klogd/syslogd outputs messages to disk.
> 
> Therefore, it is better to put kmsg_dumper in reboot/halt/poweroff/emergency_restart path.
> Any comments/advices are welcome.

I think mtdoops user don't want to dump logs if a system reboot/poweroff
safely because mtd device have write count limitation. and they are main
user of kmsg_dump().

But if they ack this one, I have no objection.

Thanks.

Re: [Dle-develop] [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Andrew M. <ak...@li...>]

On Mon, 18 Oct 2010 18:24:11 -0400
Seiji Aguchi <sei...@hd...> wrote:

> Hi,
> 
> Final messages from reboot, halt and poweroff aren't output to disk because klogd/syslogd 
> has been killed and root file system has been turned read-only.
> 
> Final messages from emergency_restart aren't output to disk as well because system may
> reboot before klogd/syslogd outputs messages to disk.
> 
> Therefore, it is better to put kmsg_dumper in reboot/halt/poweroff/emergency_restart path.
> Any comments/advices are welcome.
> 

I don't get it.

What actual problem are we solving here?  Why is the current code
inadequate?  It would help to demonstrate some use-case and to explain
how the situation improved with this patch.

[Dle-develop] [RFC][Patch] Adding kmsg_dump() to reboot/halt/poweroff/emergency_restart path

[Seiji A. <sei...@hd...>]

Hi,

Final messages from reboot, halt and poweroff aren't output to disk because klogd/syslogd 
has been killed and root file system has been turned read-only.

Final messages from emergency_restart aren't output to disk as well because system may
reboot before klogd/syslogd outputs messages to disk.

Therefore, it is better to put kmsg_dumper in reboot/halt/poweroff/emergency_restart path.
Any comments/advices are welcome.

Seiji

 Signed-off-by: Seiji Aguchi <sei...@hd...>

---
 include/linux/kmsg_dump.h |    4 ++++
 kernel/printk.c           |    4 ++++
 kernel/sys.c              |    6 ++++++
 3 files changed, 14 insertions(+), 0 deletions(-)

diff --git a/include/linux/kmsg_dump.h b/include/linux/kmsg_dump.h
index 24b4414..2a0d7d6 100644
--- a/include/linux/kmsg_dump.h
+++ b/include/linux/kmsg_dump.h
@@ -18,6 +18,10 @@ enum kmsg_dump_reason {
 	KMSG_DUMP_OOPS,
 	KMSG_DUMP_PANIC,
 	KMSG_DUMP_KEXEC,
+	KMSG_DUMP_RESTART,
+	KMSG_DUMP_HALT,
+	KMSG_DUMP_POWEROFF,
+	KMSG_DUMP_EMERG,
 };
 
 /**
diff --git a/kernel/printk.c b/kernel/printk.c
index 6d6b09f..6a63848 100644
--- a/kernel/printk.c
+++ b/kernel/printk.c
@@ -1516,6 +1516,10 @@ static const char const *kmsg_reasons[] = {
 	[KMSG_DUMP_OOPS]	= "oops",
 	[KMSG_DUMP_PANIC]	= "panic",
 	[KMSG_DUMP_KEXEC]	= "kexec",
+	[KMSG_DUMP_RESTART]	= "restart",
+	[KMSG_DUMP_HALT]	= "halt",
+	[KMSG_DUMP_POWEROFF]	= "poweroff",
+	[KMSG_DUMP_EMERG]	= "emergency_restart",
 };
 
 static const char *kmsg_to_str(enum kmsg_dump_reason reason)
diff --git a/kernel/sys.c b/kernel/sys.c
index 7f5a0cd..3eab886 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -43,6 +43,8 @@
 #include <linux/kprobes.h>
 #include <linux/user_namespace.h>
 
+#include <linux/kmsg_dump.h>
+
 #include <asm/uaccess.h>
 #include <asm/io.h>
 #include <asm/unistd.h>
@@ -285,6 +287,7 @@ out_unlock:
  */
 void emergency_restart(void)
 {
+	kmsg_dump(KMSG_DUMP_EMERG);
 	machine_emergency_restart();
 }
 EXPORT_SYMBOL_GPL(emergency_restart);
@@ -312,6 +315,7 @@ void kernel_restart(char *cmd)
 		printk(KERN_EMERG "Restarting system.\n");
 	else
 		printk(KERN_EMERG "Restarting system with command '%s'.\n", cmd);
+	kmsg_dump(KMSG_DUMP_RESTART);
 	machine_restart(cmd);
 }
 EXPORT_SYMBOL_GPL(kernel_restart);
@@ -333,6 +337,7 @@ void kernel_halt(void)
 	kernel_shutdown_prepare(SYSTEM_HALT);
 	sysdev_shutdown();
 	printk(KERN_EMERG "System halted.\n");
+	kmsg_dump(KMSG_DUMP_HALT);
 	machine_halt();
 }
 
@@ -351,6 +356,7 @@ void kernel_power_off(void)
 	disable_nonboot_cpus();
 	sysdev_shutdown();
 	printk(KERN_EMERG "Power down.\n");
+	kmsg_dump(KMSG_DUMP_POWEROFF);
 	machine_power_off();
 }
 EXPORT_SYMBOL_GPL(kernel_power_off);
-- 
1.7.2.2

Re: [Dle-develop] [RFC][PATCH] Fix kexec abort due to IPI from panic().

[Seiji A. <sei...@hd...>]

Hi Eric

Thank you for your detailed explanation.
I understood historic backdrop of kexec in upstream kernel.
I also understood the reason why you are worried about 
kmsg_dump in kexec path.

The problem is that kmsg_dumper or notifier calls are used by people 
who are unfamiliar with kexec because they use blocking-functions 
such as do_gettimeofday unconsciously.

And I still believe that kmsg_dumper in kexec path is an useful 
A troubleshooting feature.

When kexec on panic fails, it is difficult to investigate the reason 
why kexec fails. It comes from lack of materials for the investigation.

Kernel buffer is helpful for the investigation.
So, I would like to continue the discussion about a way of getting 
kernel buffer in kexec path.

I'll participate in End User Summit and Linux Plumbers Conference.
I'd like to meet with you if you join these conferences. 

Seiji

Re: [Dle-develop] [RFC][PATCH] Fix kexec abort due to IPI from panic().

[<ebi...@xm...>]

Seiji Aguchi <sei...@hd...> writes:

> Hi Eric,
>
> This is a patch which makes kmsg_dump() non-blocking.
> Please give me your comments and suggestions.
>
> I improved it as follows.
>
> (1) Improvement of dump_list_lock
>     (1-1) I changed dump_list to RCU for deleting dump_list_lock in kmsg_dump().
>     (1-2) I moved kmsg_dump(KMSG_DUMP_KEXEC) behind machine_crash_shutdown() 
>           for avoiding concurrent execution of dump_list functions.
>     (1-3) I also moved kmsg_dump(KMSG_DUMP_PANIC) behind smp_send_stop() for the 
>           same reason.
>  
> (2) Improvement of logbuf_lock
>     I added spinlock_init(&logbuf_lock) when executing kmsg_dump() in kexec or panic path
>     for preventing dead lock.
>
> We can delete blocking kmsg_dump call in crash_kexec and panic path.

This looks better, but it still gives me the willies.

I tried tracing through the ramoops code to see if there were anything
else that could block, but I couldn't make it through do_gettimeofday.

I couldn't even make it that far with the mtd oops tracer.

The fact that the code is exported and modular doesn't make me feel
safe because there have been people in the past who have asked for an
notifier on crash so they could do stupid things when the kernel is
broken.

The fact that this wasn't noticed until we actually had a hang, doesn't
give me an especially great feeling about long term stability.

Most of all I don't see the use case of calling kmsg_dump when you have
kexec on panic setup to do the same thing.  Having kmsg_dump not on
the kexec on panic code path would let me sleep much easier at night.

Then there is the historical side of this.  Through many failed attempts
it has been show that dumpers in the kernel are fragile beasts that work
up until you actually have a real world failure and then they let you
down.  Kexec on panic is better as it works 65% or so of the time,
and definitely won't corrupt your bits if it fails.  I don't see what
makes kmsg_dump better than all of the past failed and useless kernel
dumpers.

Eric

[Dle-develop] [RFC][PATCH] Fix kexec abort due to IPI from panic().

[Seiji A. <sei...@hd...>]

Hi Eric,

This is a patch which makes kmsg_dump() non-blocking.
Please give me your comments and suggestions.

I improved it as follows.

(1) Improvement of dump_list_lock
    (1-1) I changed dump_list to RCU for deleting dump_list_lock in kmsg_dump().
    (1-2) I moved kmsg_dump(KMSG_DUMP_KEXEC) behind machine_crash_shutdown() 
          for avoiding concurrent execution of dump_list functions.
    (1-3) I also moved kmsg_dump(KMSG_DUMP_PANIC) behind smp_send_stop() for the 
          same reason.
 
(2) Improvement of logbuf_lock
    I added spinlock_init(&logbuf_lock) when executing kmsg_dump() in kexec or panic path
    for preventing dead lock.

We can delete blocking kmsg_dump call in crash_kexec and panic path.

 Signed-off-by: Seiji Aguchi <sei...@hd...>

---
 kernel/kexec.c  |   10 ++++++++--
 kernel/panic.c  |    4 ++--
 kernel/printk.c |   19 ++++++++++---------
 3 files changed, 20 insertions(+), 13 deletions(-)

diff --git a/kernel/kexec.c b/kernel/kexec.c
index c0613f7..fdc6bfc 100644
--- a/kernel/kexec.c
+++ b/kernel/kexec.c
@@ -1075,16 +1075,22 @@ void crash_kexec(struct pt_regs *regs)
 	 * sufficient.  But since I reuse the memory...
 	 */
 	if (mutex_trylock(&kexec_mutex)) {
+		unsigned long flags;
+
+		local_irq_save(flags);
+
 		if (kexec_crash_image) {
 			struct pt_regs fixed_regs;
 
-			kmsg_dump(KMSG_DUMP_KEXEC);
-
 			crash_setup_regs(&fixed_regs, regs);
 			crash_save_vmcoreinfo();
 			machine_crash_shutdown(&fixed_regs);
+			kmsg_dump(KMSG_DUMP_KEXEC);
 			machine_kexec(kexec_crash_image);
 		}
+
+		local_irq_restore(flags);
+
 		mutex_unlock(&kexec_mutex);
 	}
 }
diff --git a/kernel/panic.c b/kernel/panic.c
index 4c13b1a..e14d1d6 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -87,8 +87,6 @@ NORET_TYPE void panic(const char * fmt, ...)
 	 */
 	crash_kexec(NULL);
 
-	kmsg_dump(KMSG_DUMP_PANIC);
-
 	/*
 	 * Note smp_send_stop is the usual smp shutdown function, which
 	 * unfortunately means it may not be hardened to work in a panic
@@ -96,6 +94,8 @@ NORET_TYPE void panic(const char * fmt, ...)
 	 */
 	smp_send_stop();
 
+	kmsg_dump(KMSG_DUMP_PANIC);
+
 	atomic_notifier_call_chain(&panic_notifier_list, 0, buf);
 
 	bust_spinlocks(0);
diff --git a/kernel/printk.c b/kernel/printk.c
index 8fe465a..6d6b09f 100644
--- a/kernel/printk.c
+++ b/kernel/printk.c
@@ -1478,7 +1478,7 @@ int kmsg_dump_register(struct kmsg_dumper *dumper)
 	/* Don't allow registering multiple times */
 	if (!dumper->registered) {
 		dumper->registered = 1;
-		list_add_tail(&dumper->list, &dump_list);
+		list_add_tail_rcu(&dumper->list, &dump_list);
 		err = 0;
 	}
 	spin_unlock_irqrestore(&dump_list_lock, flags);
@@ -1502,10 +1502,11 @@ int kmsg_dump_unregister(struct kmsg_dumper *dumper)
 	spin_lock_irqsave(&dump_list_lock, flags);
 	if (dumper->registered) {
 		dumper->registered = 0;
-		list_del(&dumper->list);
+		list_del_rcu(&dumper->list);
 		err = 0;
 	}
 	spin_unlock_irqrestore(&dump_list_lock, flags);
+	synchronize_rcu();
 
 	return err;
 }
@@ -1541,6 +1542,10 @@ void kmsg_dump(enum kmsg_dump_reason reason)
 	unsigned long l1, l2;
 	unsigned long flags;
 
+
+	if (reason == KMSG_DUMP_KEXEC || reason == KMSG_DUMP_PANIC)
+		spin_lock_init(&logbuf_lock);
+
 	/* Theoretically, the log could move on after we do this, but
 	   there's not a lot we can do about that. The new messages
 	   will overwrite the start of what we dump. */
@@ -1563,13 +1568,9 @@ void kmsg_dump(enum kmsg_dump_reason reason)
 		l2 = chars;
 	}
 
-	if (!spin_trylock_irqsave(&dump_list_lock, flags)) {
-		printk(KERN_ERR "dump_kmsg: dump list lock is held during %s, skipping dump\n",
-				kmsg_to_str(reason));
-		return;
-	}
-	list_for_each_entry(dumper, &dump_list, list)
+	rcu_read_lock();
+	list_for_each_entry_rcu(dumper, &dump_list, list)
 		dumper->dump(dumper, reason, s1, l1, s2, l2);
-	spin_unlock_irqrestore(&dump_list_lock, flags);
+	rcu_read_unlock();
 }
 #endif
-- 
1.7.2.2


Seiji

[Dle-develop] Nuubzm Ibbrpci Swwdtpnd

[Akwyqj <aji...@ph...>]

Rar!

Re: [Dle-develop] [PATCH] Fix kexec abort due to IPI from panic().

[Seiji A. <sei...@hd...>]

Hi Eric,

Thank you for your quick response!

>Disabling interrupts is fine,

Thanks!

>However that call to kmsg_dump(KMSG_DUMP_KEXEC) is a bug as it introduces locks 
>into a path that should not be taking locks.

I'd like to find a way that kexec coexists with kmsg_dump(KMSG_DUMP_KEXEC) because 
kmsg_dump is a useful troubleshooting feature as well.

So, I will improve kmsg_dump(KMSG_DUMP_KEXEC) if there are some bugs.
Could you please let me know your concern?
It is helpful for me if you have an example scenario kexec fails.

>Nothing in the crash_kexec path should even have the option of blocking.

Do you mean I need to change kmsg_dump(KMSG_DUMP_KEXEC) to lockless?

Seiji