nbd-general Mailing List for Network Block Device
Brought to you by:
yoe
You can subscribe to this list here.
2004 |
Jan
|
Feb
|
Mar
|
Apr
|
May
(10) |
Jun
(23) |
Jul
(20) |
Aug
(1) |
Sep
|
Oct
|
Nov
|
Dec
(1) |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2005 |
Jan
(2) |
Feb
(10) |
Mar
(7) |
Apr
(12) |
May
(14) |
Jun
(2) |
Jul
(4) |
Aug
(3) |
Sep
(6) |
Oct
(21) |
Nov
(2) |
Dec
(12) |
2006 |
Jan
(5) |
Feb
(13) |
Mar
(11) |
Apr
(11) |
May
(15) |
Jun
(21) |
Jul
(5) |
Aug
(7) |
Sep
(2) |
Oct
(17) |
Nov
(12) |
Dec
(9) |
2007 |
Jan
(15) |
Feb
(12) |
Mar
(5) |
Apr
(7) |
May
(18) |
Jun
(45) |
Jul
(8) |
Aug
(13) |
Sep
(15) |
Oct
(35) |
Nov
(11) |
Dec
(2) |
2008 |
Jan
(5) |
Feb
(19) |
Mar
(20) |
Apr
(13) |
May
(14) |
Jun
(11) |
Jul
(8) |
Aug
(19) |
Sep
(25) |
Oct
(13) |
Nov
(26) |
Dec
(49) |
2009 |
Jan
(16) |
Feb
(22) |
Mar
(27) |
Apr
(28) |
May
(69) |
Jun
(23) |
Jul
(47) |
Aug
(14) |
Sep
(5) |
Oct
(8) |
Nov
(9) |
Dec
(13) |
2010 |
Jan
(11) |
Feb
(12) |
Mar
(8) |
Apr
(3) |
May
(9) |
Jun
(7) |
Jul
(18) |
Aug
(34) |
Sep
(21) |
Oct
(14) |
Nov
(11) |
Dec
(13) |
2011 |
Jan
(19) |
Feb
(8) |
Mar
(17) |
Apr
(19) |
May
(184) |
Jun
(47) |
Jul
(27) |
Aug
(63) |
Sep
(127) |
Oct
(22) |
Nov
(21) |
Dec
(34) |
2012 |
Jan
(36) |
Feb
(45) |
Mar
(59) |
Apr
(52) |
May
(43) |
Jun
(68) |
Jul
(29) |
Aug
(45) |
Sep
(41) |
Oct
(4) |
Nov
(13) |
Dec
(37) |
2013 |
Jan
(59) |
Feb
(41) |
Mar
(104) |
Apr
(76) |
May
(29) |
Jun
(101) |
Jul
(18) |
Aug
(8) |
Sep
(23) |
Oct
(42) |
Nov
(36) |
Dec
(27) |
2014 |
Jan
(33) |
Feb
(40) |
Mar
(50) |
Apr
(40) |
May
(51) |
Jun
(15) |
Jul
(48) |
Aug
(42) |
Sep
(98) |
Oct
(106) |
Nov
(51) |
Dec
(21) |
2015 |
Jan
(50) |
Feb
(47) |
Mar
(31) |
Apr
(106) |
May
(66) |
Jun
(29) |
Jul
(36) |
Aug
(57) |
Sep
(22) |
Oct
(35) |
Nov
(33) |
Dec
(18) |
2016 |
Jan
(75) |
Feb
(159) |
Mar
(332) |
Apr
(740) |
May
(241) |
Jun
(107) |
Jul
(54) |
Aug
(33) |
Sep
(162) |
Oct
(106) |
Nov
(61) |
Dec
(187) |
2017 |
Jan
(94) |
Feb
(62) |
Mar
(74) |
Apr
(95) |
May
(74) |
Jun
(47) |
Jul
(51) |
Aug
(60) |
Sep
(30) |
Oct
(3) |
Nov
(2) |
Dec
|
2018 |
Jan
|
Feb
(1) |
Mar
|
Apr
|
May
(4) |
Jun
|
Jul
(2) |
Aug
(2) |
Sep
|
Oct
(1) |
Nov
|
Dec
|
From: Adriana K. <an...@li...> - 2018-10-03 18:50:51
|
Some environments like embedded systems may not care about about the man pages, which require the SGML tools to be installed. So add an option to make the man pages optional to make it easier to build nbd-server and nbd-client without the need for installing the SGML tools. Signed-off-by: Adriana Kobylak <an...@li...> --- Makefile.am | 2 +- configure.ac | 19 ++++++++++++++----- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/Makefile.am b/Makefile.am index 4fcfd63..788f9f1 100644 --- a/Makefile.am +++ b/Makefile.am @@ -1,5 +1,5 @@ ACLOCAL_AMFLAGS = -I support -SUBDIRS = . man doc tests systemd gznbd +SUBDIRS = . $(MAN_SUBDIR) doc tests systemd gznbd bin_PROGRAMS = nbd-server nbd-trdump EXTRA_PROGRAMS = nbd-client make-integrityhuge noinst_LTLIBRARIES = libnbdsrv.la libcliserv.la diff --git a/configure.ac b/configure.ac index c015ba0..a517877 100644 --- a/configure.ac +++ b/configure.ac @@ -333,6 +333,19 @@ else AC_DEFINE(HAVE_NETLINK, 0, [Define to 1 if we have netlink support]) fi +AC_ARG_ENABLE([manpages], + AS_HELP_STRING([--disable-manpages], [Do not install man pages])) +AS_IF([test "x$enable_manpages" != "xno"], [ + AC_SUBST([MAN_SUBDIR],["man"]) + AC_SUBST([MAN_CONFIG_FILES],["\ + man/nbd-client.8.sh \ + man/nbd-server.5.sh \ + man/nbd-server.1.sh \ + man/nbd-trdump.1.sh \ + man/nbdtab.5.sh \ + "]) + ]) + AC_HEADER_SYS_WAIT AC_TYPE_OFF_T AC_TYPE_PID_T @@ -351,11 +364,7 @@ AC_CONFIG_FILES([Makefile tests/Makefile tests/code/Makefile tests/run/Makefile - man/nbd-client.8.sh - man/nbd-server.5.sh - man/nbd-server.1.sh - man/nbd-trdump.1.sh - man/nbdtab.5.sh + $MAN_CONFIG_FILES systemd/Makefile systemd/nbd@.service.sh ]) -- 1.8.3.1 |
From: Eric B. <eb...@re...> - 2018-08-01 14:40:53
|
On 08/01/2018 09:13 AM, Richard W.M. Jones wrote: > On Wed, Aug 01, 2018 at 04:49:21PM +0300, Nir Soffer wrote: >> All this is an issue only when the underlying file is a file using raw >> format, right? It's an issue for any setup where the server and client disagree on the minimum block size, and where the file size is not aligned to one (or both) of the two endpoints' notion of minimum block size. > > NBD should only really be used to send raw format data over the wire. > While it is possible to serve a qcow2 file over NBD as qcow2, it's not > really how NBD is intended to be used. Well, if I ever get time to implement my proposed NBD_CMD_RESIZE support, then exporting qcow2 over NBD as qcow2 will be a lot more feasible than it currently is, but that's a side discussion. > >> Since NBD is meant for block devices, and block devices cannot have >> partial blocks, would it be simpler to round down unaligned file size, and >> treat possible unaligned range at the end of the file as area that can not >> be accessed via NBD? NBD is not just for block devices, but yes, block devices are much less likely to have a reported size that is not aligned to the minimum block size. So yes, the issue I'm describing (and in particular, qemu's bug of splitting NBD_CMD_READ and NBD_CMD_BLOCK_STATUS on non-sector boundaries) tends to occur mainly when using unaligned files, and not block devices, as the basis of the export that the NBD server is exposing. > > For nbdkit I would like to preserve the ability to serve arbitrary- > sized files. nbdkit allows people to write plugins, and we cannot be > sure that every plugin everyone writes will always use sensible block > sizes. > > In a forthcoming nbdkit release you can add the "truncate" filter on > top of any plugin to round up the size to a multiple of any power of 2 > (or round down if you really want). > > https://www.redhat.com/archives/libguestfs/2018-August/msg00006.html > > Rich. > -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org |
From: Eric B. <eb...@re...> - 2018-08-01 13:33:31
|
Rich Jones pointed me to questionable behavior in qemu's NBD server implementation today: qemu advertises a minimum block size of 512 to any client that promises to honor block sizes, but when serving up a raw file that is not aligned to a sector boundary, attempting to read that final portion of the file results in a structured read with two chunks, the first for the data up to the end of the actual file, and the second reporting a hole for the rest of the sector. If a client is promising to obey block sizes on its requests, it seems odd that the server is allowed to send a result that is not also aligned to block sizes. Right now, the NBD spec says that when structured replies are in use, then for a structured read: The server MAY split the reply into any number of content chunks; each chunk MUST describe at least one byte, although to minimize overhead, the server SHOULD use chunks with lengths and offsets as an integer multiple of 512 bytes, where possible (the first and last chunk of an unaligned read being the most obvious places for an exception). I'm wondering if we should tighten that to require that the server partition the reply chunks to be aligned to the advertised minimum block size (at which point, qemu should either advertise 1 instead of 512 as the minimum size when serving up an unaligned file, or else qemu should just send the final partial sector as a single data chunk rather than trying to report the last few bytes as a hole). For comparison, on block status, we require: The server SHOULD use descriptor lengths that are an integer multiple of 512 bytes where possible (the first and last descriptor of an unaligned query being the most obvious places for an exception), and MUST use descriptor lengths that are an integer multiple of any advertised minimum block size. And qemu as a client currently hangs up on any server that violates that requirement on block status (that is, when qemu as the server tries to send a block status that was not aligned to the advertised block size, qemu as the client flags it as an invalid server - which means qemu as server is currently broken). So I'm thinking we should copy that requirement onto servers for reads as well. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org |
From: Wouter V. <w...@ut...> - 2018-07-21 11:10:33
|
On Thu, Jul 19, 2018 at 09:22:03PM +0530, Jude Pereira wrote: > Is it possible to compile the client under the Mac with OSXFuse? No. NBD is not a FUSE application; in fact, it predates FUSE. It works at a different layer, too (FUSE is for file systems, NBD is for the layer that lies below a file system). You'd need to write a kext to be able to use NBD on OSX. -- Could you people please use IRC like normal people?!? -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008 Hacklab |
From: Jude P. <co...@ju...> - 2018-07-19 16:22:31
|
Is it possible to compile the client under the Mac with OSXFuse? Are there instructions on doing this? I read that it is possible on the homepage, but couldn’t find any docs on this. In my setup, the server is going to be a Raspberry Pi, and the client will be a Mac. Cheers, Jude |
From: Wouter V. <w...@ut...> - 2018-05-20 17:28:37
|
Both applied, thanks On Fri, May 18, 2018 at 05:10:16PM -0300, Thadeu Lima de Souza Cascardo wrote: > When fork fails, we return without closing the socket pair that was just > created. This might cause file descriptor leaks. > > Signed-off-by: Thadeu Lima de Souza Cascardo <cas...@ca...> > --- > nbd-server.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/nbd-server.c b/nbd-server.c > index f5b244f..c2e20c2 100644 > --- a/nbd-server.c > +++ b/nbd-server.c > @@ -2807,6 +2807,8 @@ spawn_child(int* socket) > pid = fork(); > if (pid < 0) { > msg(LOG_ERR, "Could not fork (%s)", strerror(errno)); > + close(sockets[0]); > + close(sockets[1]); > goto out; > } > if (pid > 0) { /* Parent */ > -- > 2.17.0 > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Nbd-general mailing list > Nbd...@li... > https://lists.sourceforge.net/lists/listinfo/nbd-general > -- Could you people please use IRC like normal people?!? -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008 Hacklab |
From: Thadeu L. de S. C. <cas...@ca...> - 2018-05-18 20:10:33
|
The parent will receive the servename from the child to verify if it has reached the max number of connections. When the servename is the empty name, it will try to allocate a 0-sized buffer, which will return a NULL pointer, and that segfaults when running strcmp. Signed-off-by: Thadeu Lima de Souza Cascardo <cas...@ca...> --- nbd-server.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nbd-server.c b/nbd-server.c index c2e20c2..1d1f4c8 100644 --- a/nbd-server.c +++ b/nbd-server.c @@ -2952,7 +2952,8 @@ static int handle_childname(GArray* servers, int socket) break; } } - buf = g_malloc0(len); + buf = g_malloc0(len + 1); + buf[len] = 0; readit(socket, buf, len); for(i=0; i<servers->len; i++) { SERVER* srv = &g_array_index(servers, SERVER, i); -- 2.17.0 |
From: Thadeu L. de S. C. <cas...@ca...> - 2018-05-18 20:10:31
|
When fork fails, we return without closing the socket pair that was just created. This might cause file descriptor leaks. Signed-off-by: Thadeu Lima de Souza Cascardo <cas...@ca...> --- nbd-server.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/nbd-server.c b/nbd-server.c index f5b244f..c2e20c2 100644 --- a/nbd-server.c +++ b/nbd-server.c @@ -2807,6 +2807,8 @@ spawn_child(int* socket) pid = fork(); if (pid < 0) { msg(LOG_ERR, "Could not fork (%s)", strerror(errno)); + close(sockets[0]); + close(sockets[1]); goto out; } if (pid > 0) { /* Parent */ -- 2.17.0 |
From: Eric B. <eb...@re...> - 2018-05-03 16:18:16
|
Reviving this discussion, as it still seems useful to incorporate now that BLOCK_STATUS is only recently part of the standard. On 12/14/2016 11:09 AM, Wouter Verhelst wrote: > One thing I've been thinking about that we might want to add: > > There may be cases where a server, in performing the required calls to > be able to handle a BLOCK_STATUS request, will end up with more > information than the client asked; e.g., if the client asked information > in the base:allocation context on an extent at offset X of length Y, > then the server might conceivably do an lseek(SEEK_DATA) and/or > lseek(SEEK_HOLE). The result of that call might be that the file offset > will now point to a location Z, where Z > (X+Y). > > Currently, our spec says "the sum of the *length* fields MUST not be > greater than the overall *length* of request". This means that in > essense, the server then has to throw away the information it has on the > range Z - (X + Y). In case a client was interested in that information, > that seems like a waste. I would therefore like to remove that > requirement, by rephrasing that "sum of the *length* fields" thing into > something along the following lines: > > In case the server returns N extents, the sum of the *length* fields > of the first N-1 extents MUST NOT be greater than the overall *length* > of the request. The final extent MAY exceed the length of the request > if the server has that information anyway as a side effect of looking > up the required information and wishes to share it. > > This would then result in the fact that the "length" field in the > BLOCK_STATUS command would be little more than a hint, since we're > saying that a server can return more data than requested (if it's > available anyway) and less data than requested (if it would be too > resource-intensive to provide all the information). Not sure whether all > that makes much sense anymore, but hey. > > In addition, the combination of a server providing more information than > requested with a "REQ_ONE" flag and a length field of zero could be an > interesting way to enumerate a whole export, too. Essentially, we could > define that as a client saying "I'm interested in what the size of the > extent at offset X is, and what its properties are". > > Thoughts? > -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org |
From: Eric B. <eb...@re...> - 2018-02-08 15:11:34
|
On 02/08/2018 07:23 AM, Edgar Kaziakhmedov wrote: > Upstream NBD protocol implementation supports an efficient zero out > mechanism over the wire, along with the ability to check whether a > client allows using a hole. > > Accordingly, since PWRITE_ZERO doesn't involve any payload on the wire, > increase a maximum size of the PWRITE_ZERO request up to 1Gb (aligned). > Moreover, such change will decrease the number of PWRITE_ZERO NBD commands > in comparison with the current 32M limit. The benefits of > the larger constraint can be examined in a block mirroring over NBD. We've got a potential problem. Unless you have out-of-band communication of the maximum NBD_CMD_WRITE_ZEROES sizing (or if the NBD protocol is enhanced to advertise that as an additional piece of block size information during NBD_OPT_GO), then a client CANNOT assume that the server will accept a request this large. We MIGHT get lucky if all existing servers that accept WRITE_ZEROES requests either act on large requests or reply with EINVAL but do not outright drop the connection (which is different from servers that DO outright drop the connection for an NBD_CMD_WRITE larger than 32M). But I don't know if that's how all servers behave, so sending a too-large WRITE_ZEROES request may have the unintended consequence of killing the connection. I'm adding the NBD list; perhaps before accepting this into qemu, I should revive my earlier attempt at codifying an NBD_OPT_GO info advertisement for maximum trim/zero sizing, which would let clients have a guarantee that their choice of sizing won't cause unexpected failures. > > Signed-off-by: Edgar Kaziakhmedov <edg...@vi...> > --- > block/nbd.c | 2 +- > include/block/nbd.h | 3 +++ > 2 files changed, 4 insertions(+), 1 deletion(-) > > diff --git a/block/nbd.c b/block/nbd.c > index 94220f6d14..3641d9244e 100644 > --- a/block/nbd.c > +++ b/block/nbd.c > @@ -477,7 +477,7 @@ static void nbd_refresh_limits(BlockDriverState *bs, Error **errp) > uint32_t max = MIN_NON_ZERO(NBD_MAX_BUFFER_SIZE, s->info.max_block); > > bs->bl.max_pdiscard = max; > - bs->bl.max_pwrite_zeroes = max; > + bs->bl.max_pwrite_zeroes = NBD_MAX_PWRITE_ZERO_SIZE; max_discard should get the same treatment, whatever we decide (yes, it is feasible that a server may have different limits for NBD_CMD_TRIM than for NBD_CMD_WRITE_ZEROES, but in general, it's probably easier if both commands share theh same limit as both have very similar implementations). > bs->bl.max_transfer = max; > > if (s->info.opt_block && > diff --git a/include/block/nbd.h b/include/block/nbd.h > index ee74ec391a..e2f18e2332 100644 > --- a/include/block/nbd.h > +++ b/include/block/nbd.h > @@ -182,6 +182,9 @@ enum { > /* Maximum size of a single READ/WRITE data buffer */ > #define NBD_MAX_BUFFER_SIZE (32 * 1024 * 1024) > > +/* Maximum size of a single PWRITE_ZERO request 1Gb */ > +#define NBD_MAX_PWRITE_ZERO_SIZE (1024 * 1024 * 1024) > + > /* Maximum size of an export name. The NBD spec requires 256 and > * suggests that servers support up to 4096, but we stick to only the > * required size so that we can stack-allocate the names, and because > -- > 2.11.0 > > -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org |
From: Mehul V. <meh...@gm...> - 2017-11-07 15:33:56
|
Hey NBD Experts, I ran into below kernel crash while testing NBD client/server failover. Here is the stack dump I see on my Ubuntu-16.04 box. [10554.029187] nbd: registered device at major 43 [10573.523556] EXT4-fs (nbd0): mounting ext2 file system using the ext4 subsystem [10573.524366] EXT4-fs (nbd0): warning: mounting unchecked fs, running e2fsck is recommended [10573.524500] EXT4-fs (nbd0): mounted filesystem without journal. Opts: (null) [10591.278962] block nbd0: Receive control failed (result -512) [10591.278971] block nbd0: pid 115995, nbd-client, got signal 9 [10591.278974] block nbd0: shutting down socket [10638.646904] block nbd0: Attempted send on closed socket [10638.646908] blk_update_request: I/O error, dev nbd0, sector 4632 [10638.646912] EXT4-fs warning (device nbd0): htree_dirblock_to_tree:958: inode #2: lblock 0: comm ls: error -5 reading directory block [10662.102399] ------------[ cut here ]------------ [10662.102420] kernel BUG at /build/linux-0XAgc4/linux-4.4. 0/fs/buffer.c:3005! [10662.102427] invalid opcode: 0000 [#1] SMP [10662.102434] Modules linked in: nbd ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc aufs snd_hda_codec_hdmi binfmt_misc hp_wmi snd_hda_codec_realtek sparse_keymap snd_hda_codec_generic input_leds intel_rapl x86_pkg_temp_thermal intel_powerclamp snd_hda_intel coretemp snd_hda_codec kvm_intel snd_hda_core snd_hwdep kvm snd_pcm irqbypass snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq serio_raw snd_seq_device snd_timer sb_edac edac_core lpc_ich snd mei_me mei soundcore shpchp tpm_infineon 8250_fintek mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi [10662.102572] scsi_transport_iscsi parport_pc ppdev lp parport autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid hid nouveau crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 mxm_wmi lrw video gf128mul glue_helper ablk_helper i2c_algo_bit cryptd ttm drm_kms_helper syscopyarea sysfillrect e1000e sysimgblt psmouse fb_sys_fops ptp ahci drm pps_core libahci wmi fjes [last unloaded: nbd] [10662.102673] CPU: 7 PID: 188844 Comm: umount Not tainted 4.4.0-78-generic #99-Ubuntu [10662.102679] Hardware name: Hewlett-Packard HP Z440 Workstation/212B, BIOS M60 v02.31 12/14/2016 [10662.102686] task: ffff8807deb47000 ti: ffff8807d9e00000 task.ti: ffff8807d9e00000 [10662.102692] RIP: 0010:[<ffffffff81247a62>] [<ffffffff81247a62>] submit_bh_wbc+0x152/0x160 [10662.102706] RSP: 0018:ffff8807d9e03d40 EFLAGS: 00010246 [10662.102711] RAX: 0000000000000005 RBX: ffff88079bffbd00 RCX: 0000000000000000 [10662.102719] RDX: 0000000000000000 RSI: ffff88079bffbd00 RDI: 0000000000001411 [10662.103016] RBP: ffff8807d9e03d68 R08: 0000000000000000 R09: 0000000000000fff [10662.103755] R10: 0000000000002d7c R11: 000000000000ef31 R12: 0000000000001411 [10662.104491] R13: 0000000000000008 R14: ffff8800c973a400 R15: ffff880802f83800 [10662.105227] FS: 00007f538a5ba840(0000) GS:ffff88080c7c0000(0000) knlGS:0000000000000000 [10662.105959] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [10662.106697] CR2: 0000000001a34878 CR3: 00000007deb1b000 CR4: 00000000001406e0 [10662.107431] Stack: [10662.108161] ffff88079bffbd00 0000000000001411 0000000000000008 ffff8800c973a400 [10662.108902] ffff880802f83800 ffff8807d9e03d88 ffffffff812497bc ffffffff81f38d80 [10662.109639] ffff88079bffbd00 ffff8807d9e03dd0 ffffffff812bbf42 0000000000000034 [10662.110378] Call Trace: [10662.111100] [<ffffffff812497bc>] __sync_dirty_buffer+0x6c/0x100 [10662.111825] [<ffffffff812bbf42>] ext4_commit_super+0x1d2/0x290 [10662.112553] [<ffffffff812bccb1>] ext4_put_super+0xe1/0x390 [10662.113276] [<ffffffff812111ef>] generic_shutdown_super+0x6f/0x100 [10662.113988] [<ffffffff8121157c>] kill_block_super+0x2c/0xa0 [10662.114694] [<ffffffff812116d3>] deactivate_locked_super+0x43/0x70 [10662.115399] [<ffffffff81211bac>] deactivate_super+0x5c/0x60 [10662.116095] [<ffffffff8122fc0f>] cleanup_mnt+0x3f/0x90 [10662.116777] [<ffffffff8122fca2>] __cleanup_mnt+0x12/0x20 [10662.117458] [<ffffffff8109f011>] task_work_run+0x81/0xa0 [10662.118138] [<ffffffff81003242>] exit_to_usermode_loop+0xc2/0xd0 [10662.118805] [<ffffffff81003c6e>] syscall_return_slowpath+0x4e/0x60 [10662.119469] [<ffffffff81840b90>] int_ret_from_sys_call+0x25/0x8f [10662.120121] Code: 44 89 ef e8 81 14 18 00 5b 31 c0 41 5c 41 5d 41 5e 41 5f 5d c3 40 f6 c7 01 0f 84 1c ff ff ff f0 80 63 01 f7 e9 12 ff ff ff 0f 0b <0f> 0b 0f 0b 0f 0b 0f 0b 0f 0b 0f 1f 40 00 0f 1f 44 00 00 55 31 [10662.121487] RIP [<ffffffff81247a62>] submit_bh_wbc+0x152/0x160 [10662.122161] RSP <ffff8807d9e03d40> I have both nbd-server and nbd-client running on the same system, and issue can be reproduced with following commands, **Server truncate -s 10G /mnt/nbddisk mkfs.ext4 /mnt/nbddisk nbd-server 127.0.0.1@9000 /mnt/nbddisk **Client modprobe nbd nbd-client 127.0.0.1 9000 /dev/nbd0 mount /dev/nbd0 /mnt/ kill -9 <pid of nbd-client> After killing nbd-client, remounting /dev/nbd0 to different folder fails with "/dev/nbd0 is already mounted or /mnt1/" busy". Unmounting "/mnt" leads to above kernel crash. I found below thread reporting the similar crash. I see thread concluded with suggestions, but not sure if the fix is pushed to the mainstream kernel or not. https://sourceforge.net/p/nbd/mailman/message/34486113/ Is there any way this can be fixed in the driver? I would be glad to help in verifying the fix if needed. Thanks, Mehul. |
From: YLT <yan...@gm...> - 2017-11-02 21:40:59
|
> Yes, but it's not clear the bug is in NBD. > Can you try to reproduce the bug with different filesystems, and/or > different partition types? > If you can, then you're probably right. If not though, you'll have more > luck looking at whatever you swapped out. You're right, I couldn't reproduce with other filesystems. I did some more homework, and isolated the issue between 4.11.5 and 4.11.6. And then I saw this commit: https://patchwork.kernel.org/patch/9749695/ Reverting the patch resolve the situation. :) Thanks! On Wed, Oct 18, 2017 at 12:16 PM, Wouter Verhelst <w...@ut...> wrote: > On Sat, Oct 14, 2017 at 05:36:30PM -0400, YLT wrote: > [...] >> 0x5306550 - 0x52fe750 = 0x7E00 = 32,256 >> So it looks like the partition is offset by 32,256 bytes in 4.11. >> >> Do I need to submit a bug report? > > Yes, but it's not clear the bug is in NBD. > > Can you try to reproduce the bug with different filesystems, and/or > different partition types? > > If you can, then you're probably right. If not though, you'll have more > luck looking at whatever you swapped out. > > Regards, > > -- > Could you people please use IRC like normal people?!? > > -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008 > Hacklab |
From: Wouter V. <w...@ut...> - 2017-10-18 16:16:18
|
On Sat, Oct 14, 2017 at 05:36:30PM -0400, YLT wrote: [...] > 0x5306550 - 0x52fe750 = 0x7E00 = 32,256 > So it looks like the partition is offset by 32,256 bytes in 4.11. > > Do I need to submit a bug report? Yes, but it's not clear the bug is in NBD. Can you try to reproduce the bug with different filesystems, and/or different partition types? If you can, then you're probably right. If not though, you'll have more luck looking at whatever you swapped out. Regards, -- Could you people please use IRC like normal people?!? -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008 Hacklab |
From: YLT <yan...@gm...> - 2017-10-14 21:36:37
|
Hello, I am facing what I think is a regression with kernels > 4.10. With kernel-4.10, I am able to mount a UFS image as follow: host # qemu-nbd -c /dev/nbd0 /opt/images/user_images/instances/d40_R1.qcow2 host # partx -l /dev/nbd0 # 1: 63- 41929649 ( 41929587 sectors, 21467 MB) # 5: 63- 7547376 ( 7547314 sectors, 3864 MB) # 6: 7547377- 21384118 ( 13836742 sectors, 7084 MB) # 7: 21384119- 23480594 ( 2096476 sectors, 1073 MB) # 8: 23480595- 41929586 ( 18448992 sectors, 9445 MB) host # mount -t ufs -o ufstype=44bsd /dev/nbd0p7 /mnt/img/ host # df | grep nbd /dev/nbd0p7 1030638 2 948188 1% /mnt/img Unfortunately, this doesn't work anymore with 4.11+ (tried 4.13.6 as well, same qemu version): host # qemu-nbd -c /dev/nbd0 /opt/images/user_images/instances/d40_R1.qcow2 host # partx -l /dev/nbd0 # 1: 63- 41929649 ( 41929587 sectors, 21467 MB) # 5: 63- 7547376 ( 7547314 sectors, 3864 MB) # 6: 7547377- 21384118 ( 13836742 sectors, 7084 MB) # 7: 21384119- 23480594 ( 2096476 sectors, 1073 MB) # 8: 23480595- 41929586 ( 18448992 sectors, 9445 MB) host # mount -t ufs -o ufstype=44bsd /dev/nbd0p7 /mnt/img/ mount: wrong fs type, bad option, bad superblock on /dev/nbd0p7, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. host # dmesg ... [ 507.252786] ufs: ufs_fill_super(): bad magic number <-- Looking for magic number on 4.10: host # hexdump /dev/nbd0p7 | grep -w '1954' 0002550 0001 0000 0000 0000 0000 0000 1954 0001 << magic number at low offset 0004550 0001 0000 0000 0000 0000 0000 1954 0001 << 5306550 0001 0000 0000 0000 0000 0000 1954 0001 541ed60 0000 0000 0000 0000 0000 0000 1954 2b49 5447ee0 0000 0000 0000 0000 0000 0000 4ea8 1954 a608550 0001 0000 0000 0000 0000 0000 1954 0001 f90a550 0001 0000 0000 0000 0000 0000 1954 0001 14c0c550 0001 0000 0000 0000 0000 0000 1954 0001 19f0e550 0001 0000 0000 0000 0000 0000 1954 0001 1f210550 0001 0000 0000 0000 0000 0000 1954 0001 1f2c9960 0000 0000 0000 0000 0000 0000 6e0c 1954 1f3037e0 0000 0000 0000 0000 0000 0000 6ff2 1954 24512550 0001 0000 0000 0000 0000 0000 1954 0001 29814550 0001 0000 0000 0000 0000 0000 1954 0001 29873fe0 0000 0000 0000 0000 0000 0000 1954 3292 2eb16550 0001 0000 0000 0000 0000 0000 1954 0001 33e18550 0001 0000 0000 0000 0000 0000 1954 0001 3911a550 0001 0000 0000 0000 0000 0000 1954 0001 3e41c550 0001 0000 0000 0000 0000 0000 1954 0001 3e47fe60 0000 0000 0000 0000 0000 0000 1954 1261 <-- Not there anymore on 4.11: host # hexdump /dev/nbd0p7 | grep -w '1954' 52fe750 0001 0000 0000 0000 0000 0000 1954 0001 << high offset on the first match 5416f60 0000 0000 0000 0000 0000 0000 1954 2b49 54400e0 0000 0000 0000 0000 0000 0000 4ea8 1954 a600750 0001 0000 0000 0000 0000 0000 1954 0001 f902750 0001 0000 0000 0000 0000 0000 1954 0001 14c04750 0001 0000 0000 0000 0000 0000 1954 0001 19f06750 0001 0000 0000 0000 0000 0000 1954 0001 1f208750 0001 0000 0000 0000 0000 0000 1954 0001 1f2c1b60 0000 0000 0000 0000 0000 0000 6e0c 1954 1f2fb9e0 0000 0000 0000 0000 0000 0000 6ff2 1954 2450a750 0001 0000 0000 0000 0000 0000 1954 0001 2980c750 0001 0000 0000 0000 0000 0000 1954 0001 2986c1e0 0000 0000 0000 0000 0000 0000 1954 3292 2eb0e750 0001 0000 0000 0000 0000 0000 1954 0001 33e10750 0001 0000 0000 0000 0000 0000 1954 0001 39112750 0001 0000 0000 0000 0000 0000 1954 0001 3e414750 0001 0000 0000 0000 0000 0000 1954 0001 3e478060 0000 0000 0000 0000 0000 0000 1954 1261 3ffa5f50 0001 0000 0000 0000 0000 0000 1954 0001 << magic number now at the end 3ffa7f50 0001 0000 0000 0000 0000 0000 1954 0001 << 0x5306550 - 0x52fe750 = 0x7E00 = 32,256 So it looks like the partition is offset by 32,256 bytes in 4.11. Do I need to submit a bug report? Thanks in advance, Yannick |
From: Eric B. <eb...@re...> - 2017-10-05 13:37:35
|
On 10/05/2017 06:30 AM, Vladimir Sementsov-Ogievskiy wrote: > 21.09.2017 15:18, Vladimir Sementsov-Ogievskiy wrote: >> Hi all! >> >> I'm about this: >> >> "A server SHOULD try to minimize the number of chunks sent in a reply, >> but MUST NOT mark a chunk as final if there is still a possibility of >> detecting an error before transmission of that chunk completes" >> >> What do we mean by "possibility"? Formally, such possibility exists >> always, so, we'll never mark a chunk as final. >> > > One more question: > > for |NBD_REPLY_TYPE_ERROR and ||NBD_REPLY_TYPE_ERROR_OFFSET, why do we > need message_length field? why not to calc it as chunk.lenght - 4 for > ||NBD_REPLY_TYPE_ERROR and chunk.lenght - 12 for > ||NBD_REPLY_TYPE_ERROR_OFFSET? For consistency. If _all_ NBD_REPLY_TYPE_ERROR* message have a message_length field, then it is easier to write a generic handler that knows how to deal with an unknown error, no matter what command the error is sent in response to. Ideally, a server should never send an error message that the client is not expecting, but having a robust protocol that lets clients deal with bad servers is worth the redundancy caused by being consistent, and we are more likely to add additional error modes to existing commands than we are to add more success modes. > > For example, with NBD_REPLY_TYPE_OFFSET_DATA variable data length is > calculated, not specified separately. That's because non-error types don't have the same consistency concerns; if we want to introduce a new success response, we'll probably introduce it via a new command, rather than as a reply to an existing command. Furthermore, while error replies are likely to have a free-form text error description, success replies tend to not need it. The layout of the error types is designed to make it easy to grab the free-form error message from a known location for display to the user, even if the client has no idea what the rest of the error means, as that may be a useful debugging aid. > > What is the reason for server to send NBD_REPLY_TYPE_ERROR with > message_lenght < chunk.lenght - 4? In all likelihood, all well-written servers will never send garbage bytes (possible only when setting chunk.length larger than message_length + sizeof(documented fields)). But we wrote the spec to be conservative, in case we want to add a later defined field that earlier clients will still gracefully ignore, rather than strict (allowing inequality, instead of requiring exact lengths, lets a client skip over what it considers garbage bytes rather than dropping the connection because a too-new server tried to send useful information in those bytes). -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org |
From: Wouter V. <w...@ut...> - 2017-09-22 10:03:26
|
On Wed, Sep 20, 2017 at 02:41:52PM +0000, Josef Bacik wrote: > Nope I missed it, and I can’t find it in my linux-block or lkml folders, > where did you send it? To you personally, and to the new and old nbd lists. Yeah, should have included linux-block too, sorry. > If you re-send make sure you cc linux-block and send it to Jens, and you can > add my Reviewed-by: to it. Thanks, Right. -- Could you people please use IRC like normal people?!? -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008 Hacklab |
From: Vladimir Sementsov-O. <sem...@mi...> - 2017-09-21 12:25:08
|
Hi all! I'm about this: "A server SHOULD try to minimize the number of chunks sent in a reply, but MUST NOT mark a chunk as final if there is still a possibility of detecting an error before transmission of that chunk completes" What do we mean by "possibility"? Formally, such possibility exists always, so, we'll never mark a chunk as a final. -- Best regards, Vladimir |
From: Vladimir Sementsov-O. <vse...@vi...> - 2017-09-21 12:19:08
|
Hi all! I'm about this: "A server SHOULD try to minimize the number of chunks sent in a reply, but MUST NOT mark a chunk as final if there is still a possibility of detecting an error before transmission of that chunk completes" What do we mean by "possibility"? Formally, such possibility exists always, so, we'll never mark a chunk as final. -- Best regards, Vladimir |
From: Josef B. <jb...@fb...> - 2017-09-20 14:42:11
|
Nope I missed it, and I can’t find it in my linux-block or lkml folders, where did you send it? If you re-send make sure you cc linux-block and send it to Jens, and you can add my Reviewed-by: to it. Thanks, Josef On 9/20/17, 3:51 AM, "Wouter Verhelst" <w...@ut...> wrote: Ping. Josef, did you see this? On Tue, Sep 12, 2017 at 12:45:24AM +0200, Wouter Verhelst wrote: > nbd...@so... becomes nb...@ot..., because > sourceforge is just a spamtrap these days. > > Signed-off-by: Wouter Verhelst <w...@ut...> > --- > MAINTAINERS | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/MAINTAINERS b/MAINTAINERS > index fbb269415f06..2f8824995a93 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -9326,7 +9326,7 @@ NETWORK BLOCK DEVICE (NBD) > M: Josef Bacik <jb...@fb...> > S: Maintained > L: lin...@vg... > -L: nbd...@li... > +L: nb...@ot... > F: Documentation/blockdev/nbd.txt > F: drivers/block/nbd.c > F: include/uapi/linux/nbd.h > -- > 2.14.1 > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! https://urldefense.proofpoint.com/v2/url?u=http-3A__sdm.link_slashdot&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=sDzg6MvHymKOUgI8SFIm4Q&m=wJsUkVpb7IOV1dZ52HeUZkawsEy88nrSk2kyzJI1B1w&s=WvloF9SdMnUYtpJulUZ45appr-CwRlCKfQ4drnIvnIs&e= > _______________________________________________ > Nbd-general mailing list > Nbd...@li... > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_nbd-2Dgeneral&d=DwIBAg&c=5VD0RTtNlTh3ycd41b3MUw&r=sDzg6MvHymKOUgI8SFIm4Q&m=wJsUkVpb7IOV1dZ52HeUZkawsEy88nrSk2kyzJI1B1w&s=Sy5OOud41XNAu1LFgrQcvitIPDx8yqsH-fsP8hI90Ho&e= > -- Could you people please use IRC like normal people?!? -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008 Hacklab |
From: 戴尔 ・ 老. <Del...@ch...> - 2017-09-20 08:03:36
|
From: Wouter V. <w...@ut...> - 2017-09-20 07:51:30
|
Ping. Josef, did you see this? On Tue, Sep 12, 2017 at 12:45:24AM +0200, Wouter Verhelst wrote: > nbd...@so... becomes nb...@ot..., because > sourceforge is just a spamtrap these days. > > Signed-off-by: Wouter Verhelst <w...@ut...> > --- > MAINTAINERS | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/MAINTAINERS b/MAINTAINERS > index fbb269415f06..2f8824995a93 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -9326,7 +9326,7 @@ NETWORK BLOCK DEVICE (NBD) > M: Josef Bacik <jb...@fb...> > S: Maintained > L: lin...@vg... > -L: nbd...@li... > +L: nb...@ot... > F: Documentation/blockdev/nbd.txt > F: drivers/block/nbd.c > F: include/uapi/linux/nbd.h > -- > 2.14.1 > > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Nbd-general mailing list > Nbd...@li... > https://lists.sourceforge.net/lists/listinfo/nbd-general > -- Could you people please use IRC like normal people?!? -- Amaya Rodrigo Sastre, trying to quiet down the buzz in the DebConf 2008 Hacklab |
From: IndiaLends.com <in...@em...> - 2017-09-20 04:47:44
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type" /> <title></title> <style type="text/css"> body {margin: 10px 0; padding: 0 10px; background: #ffffff;} table {border-collapse: collapse;} td {font-family: Calibri,Candara,Segoe,Segoe UI,Optima,Arial,sans-serif;} @media only screen and (max-width: 480px) { body,table,td,p,a,li,blockquote { -webkit-text-size-adjust:none !important; font-family: Calibri,Candara,Segoe,Segoe UI,Optima,Arial,sans-serif; } table {width: 100% !important;} img { height: auto; max-width: 100%; width: 100%; } } </style> </head> <body style="margin: 0px; padding: 0px; font-family: 'Calibri';"><img src="http://em.adsancharmails.com/tr/p.gif?uid=17066609920&mid=1700240159&msd=1505882645255&st=0" width="1" height="1" alt="" /> <table style="width: 580px; border: 1px solid silver;background-color:#fafafa;" cellspacing="0" cellpadding="0" align="center"> <tbody><tr> <td> <table style="width: 100%" cellspacing="0" cellpadding="0" align="center"> <tbody> <tr> <td style="padding:0px 10px 0px 0px; text-align:left; line-height:40px; border-bottom:1px solid silver;" height="60"> <a href="http://em.adsancharmails.com/re?l=D0Is4a1pbI7u90ohsI0" target="_blank" style="text-decoration:none;"><img src="https://indialends.com/images/indialends-logo.png" alt="IndiaLends.com" style="width:auto; color:#fea84b; font-size:24px; "></a> </td> </tr> <tr> <td> <p style="font-size:15px; color: #555555; text-align:left; margin-top:10px; margin-bottom:0px; padding:0 30px 0px 10px; font-weight:normal;"> Hello $Customer$,</p> <p style="font-size:15px; color: #555555; text-align:left; margin:5px 0px 10px 0px; padding:0 10px 0px 10px; font-weight:normal;"> If you use a credit card cleverly then it?s possible to borrow at no cost and earn cashbacks for spendings on the card. Be it dining, shopping or travel vouchers, get the perfect card according to your needs by applying at IndiaLends. </p> <p style="font-size:15px; color: #555555; text-align:left; margin:5px 0px 10px 0px; padding:0 10px 0px 10px; font-weight:normal;"> We have hand-picked 3 cards that give you best in class deals: </p> </td> </tr> </tbody></table> </td></tr> <tr> <td style="padding:0px 10px;"> <table style="width:100%; text-align:center; margin-top:0px; border:1px solid grey; font-size:13px;" cellspacing="0" cellpadding="0"> <tbody><tr style="background-color:#81cce4;"> <th style="padding:14px 3px; vertical-align:top;"> <p style="font-size:14px; color: #000000; text-align:center; margin-top:0px; margin-bottom:0px; padding:0 10px 0px 10px; font-weight:bold;">Credit Card</p></th> <th style="padding:14px 3px; vertical-align:top;width:35%;"> <p style="font-size:14px; color: #000000; text-align:center; margin-top:0px; margin-bottom:0px; padding:0 10px 0px 10px; font-weight:bold;">Rewards and Benefits</p></th> <th style="padding:14px 3px; vertical-align:top; width:25%;"> <p style="font-size:14px; color: #000000; text-align:center; margin-top:0px; margin-bottom:0px; padding:0 10px 0px 10px; font-weight:bold;">Why This Card?</p></th> <th style="padding:3px; width:20%;"> </th> </tr> <tr> <td colspan="4" style="border-top:1px solid grey;"></td> </tr> <tr> <td style="padding:5px 5px;"> <p style="font-size:14px; color: #555555; text-align:center; margin-top:0px; margin-bottom:0px; padding:0px; font-weight:bold;">SBI SimplySAVE</p></td> <td style="padding:5px 5px;"> <ul style="font-size:14px; color: #555555; text-align:left; font-weight:normal; padding-left:15px; margin:0px;"> <li>Earn reward points on every spends & <b>extra 2000 points </b> as welcome gift</li> </ul></td> <td style="padding:5px 5px;"> <ul style="font-size:14px; color: #555555; text-align:left; font-weight:normal; padding-left:15px; margin:0px;"> <li>Because you can <b>save</b> on every spend</li> </ul> </td> <td style="padding:5px 5px;"> <a href="http://em.adsancharmails.com/re?l=D0Is4a1pbI7u90ohsI1" target="_blank" style="text-decoration:none; color:#81cce4;"><span style="background-color:#f88e1c; color:#fff; padding: 7px 20px; border-radius:5px;">Apply</span> </a> </td> </tr> <tr> <td colspan="4" style="border-top:1px solid grey;"></td> </tr> <tr> <td style="padding:5px 5px;"> <p style="font-size:14px; color: #555555; text-align:center; margin-top:0px; margin-bottom:0px; padding:0px; font-weight:bold;">SBI IRCTC</p></td> <td style="padding:5px 5px;"> <ul style="font-size:14px; color: #555555; text-align:left; font-weight:normal; padding-left:15px;margin:0px;"> <li>Get <b>10% cashback</b> on all rail tickets & save 1.8% transaction charges on rail bookings</li> </ul> </td> <td style="padding:5px 5px;"> <ul style="font-size:14px; color: #555555; text-align:left; font-weight:normal; padding-left:15px; margin:0px;"> <li>Because now <b>booking tickets</b> on irctc is cost-effective</li> </ul> </td> <td style="padding:5px 5px;"> <a href="http://em.adsancharmails.com/re?l=D0Is4a1pbI7u90ohsI2" target="_blank" style="text-decoration:none; color:#81cce4;"><span style="background-color:#f88e1c; color:#fff; padding: 7px 20px; border-radius:5px;">Apply</span> </a> </td> </tr> <tr> <td colspan="4" style="border-top:1px solid grey;"></td> </tr> <tr> <td style="padding:5px 5px;"> <p style="font-size:14px; color: #555555; text-align:center; margin-top:0px; margin-bottom:0px; padding:0px; font-weight:bold;">Yatra SBI</p></td> <td style="padding:5px 5px;"> <ul style="font-size:14px; color: #555555; text-align:left; font-weight:normal; padding-left:15px; margin:0px;"> <li>Get Rs 8,250 worth Yatra.com vouchers on joining</li> </ul> </td> <td style="padding:5px 5px;"> <ul style="font-size:14px; color: #555555; text-align:left; font-weight:normal; padding-left:15px; margin:0px;"> <li>Because you can save more on every travel </li> </ul> </td> <td style="padding:5px 5px;"> <a href="http://em.adsancharmails.com/re?l=D0Is4a1pbI7u90ohsI3" target="_blank" style="text-decoration:none; color:#81cce4;"><span style="background-color:#f88e1c; color:#fff; padding: 7px 20px; border-radius:5px;">Apply</span> </a> </td> </tr> </tbody></table> </td> </tr> <tr> <td style="width:100%; text-align:center;padding:0px 10px 20px 0px; "> <p style="font-size:15px; color: #555555; text-align:left; margin-top:10px; margin-bottom:0px; padding:0 10px 0px 10px; font-weight:normal;"> Find yourself the perfect card suitable according to your lifestyle and spending.<br><br> Regards,<br> IndiaLends Team </p> </td> </tr> </tbody></table> </body> </html> |
From: Dell C. <Del...@ch...> - 2017-09-19 01:28:20
|