#63 sqsh/​freetds segfaults after failure to connect

[Slava K.]

The latest sqsh for FreeTDS (loaded as sqsh-2.5-1.el6.x86_64.x86_64.rpm) consistently generates "Segmentation fault" error every time sqsh fails on connection to MS SQL server. For instance:

iebdev21 [05:19 PM] ~> ~/sa/MSSQL/sqsh-ms/sqsh-2.5/RPM/usr/bin/sqsh -v
sqsh-2.5
iebdev21 [05:20 PM] ~> ~/sa/MSSQL/sqsh-ms/sqsh-2.5/RPM/usr/bin/sqsh -C "set nocount on" -S MSSQL777 -U xxx -P yyy
Open Client Message
Layer 0, Origin 0, Severity 78, Number 44
Server name not found in configuration files.
Open Client Message
Layer 0, Origin 0, Severity 78, Number 45
Unknown host machine name.
Segmentation fault
iebdev21 [05:20 PM] ~> echo $?
139

Since this is the binary from prebuild package (downloaded from sourceforge), it hopefully would not be a problem to reproduce the reported behavior. Please note: it doesn't matter what was the cause of connection failure - wrong login name, password or server was down. In any case sqsh segfaults.
Actually same happens with every sqsh version (for freetds) I tested: 2.1.4, 2.1.7, 2.1.8 and 2.4.

[Martin Wesdorp]

Unfortunately I am not able to reproduce this behaviour. I have a strong feeling this segmentation fault comes from your freetds implementation. Which version of FreeTDS have you installed? Is it installed as a pre-built package or did you built it yourself? The sqsh package is built against freetds-0.91-2.el6.x86_64.

I will explain a little bit further how sqsh interacts with the FreeTDS libct library. When you start sqsh with a couple of connection specific parameters like -S, -U, -P (but also parameter settings regarding language, TDS package size, character set, etc.), these are being handled by the cmd_connect() function of sqsh and that will make subsequent function calls to CT-library (FreeTDS) to set/activate these provided parameters and finally a call will be made to ct_connect() that will actually perform the connection setup to the database server. If something with ct_connect() fails, it will be reported by the callback message handlers coded in sqsh and control is then returned back to the CT-library again. Normally speaking, when all messages/errors are processed by the callback handlers, the ct_connect() call should return back to sqsh with a return code not equal to CS_SUCCEED and sqsh would then cleanup and exit with returncode 255. In your case a segmentation fault occurs after the error messages are displayed. So the fault could still be in ct_connect() or cleanup processing of sqsh, which also makes some CT-library calls.

If you have gdb installed you could try to analyze the core file or run sqsh through gdb to find the failing function. To inspect a core file you could do:

ulimit -c 4096
/usr/bin/sqsh -Sxxx -Uyyy -Pzzz
gdb /usr/bin/sqsh core

or

gdb
(gdb) file /usr/bin/sqsh
(gdb) run -Sxxx -Uyyy -Pzzz
(gdb) where

to find out which libraries are loaded by sqsh you could run

ldd /usr/bin/sqsh

The libct.so.4 specifies the related shared object file of FreeTDS. Please let me know your results.
PS, I note you have installed sqsh in another location so you have to provide your own path to sqsh of course in the above examples

[Slava K.]

Hi Martin!
Thanks a lot for detail instructions. I will follow them all, but first I'd like to make sure we're on the same page. I wonder if you can try to run sqsh command with "-C" option, like the one I showed in initial post. Something like
sqsh -S xxx -U yyy -P zzz -C "any command or anything"

Now about software I used. Previously I ran the test against freetds loaded from www.freetds.org (freetds-stable.tgz), which then I built myself. I couldn't find there freetds version you mentioned in your post, but then got freetds-0.91-2.el6.x86_64.rpm from http://ftp.linux.ncsu.edu/mirror/ftp.redhat.com/pub/epel/6/x86_64. This is pre-built package, so I didn't need to build it myself. sqsh itself is also loaded as RPM package sqsh-2.5-1.el6.x86_64.x86_64.rpm from https://sourceforge.net/projects/sqsh/files/sqsh/sqsh-2.5 - i.e. it's your build.

In gdb we are not getting all details:

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./bin/sqsh -S xxx -U yyy -C anything -¨3/4'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007ffff7dadf8d in ct_cmd_alloc () from ./freetds/lib64/libct.so.4
(gdb) bt
#0  0x00007ffff7dadf8d in ct_cmd_alloc () from ./freetds/lib64/libct.so.4
#1  0x0000000000409e9f in cmd_go ()
#2  0x00000000004215c0 in jobset_run ()
#3  0x000000000040bf50 in cmd_input ()
#4  0x000000000040d060 in cmd_loop ()
#5  0x00000000004215c0 in jobset_run ()
#6  0x00000000004257f7 in main ()

Running sqsh under gdb basically gives the same output.
But previously I had this test run with sqsh and freetds I build myself with debug options.
And then I've got a bit more info:

Program terminated with signal 11, Segmentation fault.
#0  ct_cmd_alloc (con=0x0, cmd=cmd@entry=0x7fffadd1b708) at ct.c:683
683             if ( con->cmds == NULL ) {
(gdb) bt
#0  ct_cmd_alloc (con=0x0, cmd=cmd@entry=0x7fffadd1b708) at ct.c:683
#1  0x0000000000409d7d in cmd_go (argc=<optimized out>, argv=argv@entry=0x1adcf10) at cmd_go.c:468
#2  0x000000000042002d in jobset_run (js=<optimized out>, cmd_line=<optimized out>, cmd_line@entry=0x1add5f0 "\\go\n", exit_status=exit_status@entry=0x7fffadd1c860) at sqsh_job.c:508
#3  0x000000000040be2f in cmd_input () at cmd_input.c:494
#4  0x000000000040ceed in cmd_loop (argc=<optimized out>, argv=argv@entry=0x1acc860) at cmd_loop.c:183
#5  0x000000000042002d in jobset_run (js=<optimized out>, cmd_line=<optimized out>, cmd_line@entry=0x42f63c "\\loop -e \"$sql\"", exit_status=exit_status@entry=0x7fffadd1dd6c)
    at sqsh_job.c:508
#6  0x00000000004246c4 in main (argc=<optimized out>, argv=<optimized out>) at sqsh_main.c:793

Please let me know if you need more data.

[Martin Wesdorp]

Hi Slava,

You are absolutely right. When using the -C option I can reproduce the segmentation fault indeed. What happens when you add the -C option to the commandline is that another codepath is executed and cmd_go is called first. This functions checks if a connection is already established and if not, calls cmd_connect to setup a server connection (via a call to jobset_run(..,"\connect",..)). However, the return status of cmd_connect is not checked correctly and code processing in cmd_go() continues and finally fails if a ct_cmd_alloc() is called with a NULL connection pointer. The difference here with Sybase CT-Library is that the Sybase functions do some sanity checks and just return CS_FAIL when called with a NULL connection pointer, where FreeTDS does not check on this and runs into the segmentation fault.

The solution is to add an additional check on the return status of cmd_connect (via jobset_run(g_jobset,"\connect",&exit_status)). Not only the return value of jobset_run itself needs to be checked, but also the value of exit_status. There are a couple of other flaws in the code with the same error.

I will provide a fix in the next release.

Thanks for your report and your help with this.

Best Regards,
Martin.

[Slava K.]

Thanks for prompt feedback and cooperation, Martin! Hope this fix will be out soon. Now it's up to you to close this ticket or leave open. I have one more note about option "-I" not working with sqsh/freetds, but probably it should go to a separate ticket. Please advise.

[Martin Wesdorp]

Hi Slava,

Ik will close the ticket when the bug is fixed in the next release.

Regarding the -I option with sqsh on FreeTDS, that is a known issue. FreeTDS currently does not support this and silently ignores interfaces or freetds.conf files provided with the -I parameter. Please have a look at thread

https://sourceforge.net/p/sqsh/discussion/365924/thread/e56075e9

So what happens if you provide the -I parameter is that sqsh makes a call to ct_config() with the CS_IFILE property and a pointer to the expanded parameter string. However, the ct_config function in freetds-dev.0.92.377/src/ctlib/ct.c does not check for the CS_IFILE property in the switch statement and returns CS_SUCCEED by default. To solve this the FreeTDS project should add something minimalistic like this on line 2456 for example of the named source file:

    case CS_IFILE:
            tds_set_interfaces_file_loc (buffer);
            ret = CS_SUCCEED;
            break;

This works for me.

Cheers,
Martin.

[Slava K.]

Sounds good on all counts, Martin. I think it might be useful to report "-I" issue to freetds, maybe we'll get to that later. Thanks for your help!

[Slava K.]

Hi Martin,

While I was looking for the ways sqsh and freetds handle interfaces file, I spotted that in line 1283 of cmd_connect.c the code looks a little unusual (to my ignorant opinion):

    if (ct_cmd_alloc( g_connection, &cmd ) != CS_SUCCEED)
        goto connect_succeed;

In other modules like cmd_go.c, cmd_do.c or freetds unit tests the similar condition is treated as a failure. I'm totally unfamiliar with this code, so it maybe all right. Just wanted to check with you whether it's what was supposed to be there.

Thanks for looking,
Slava.

[Martin Wesdorp]

Hi Slava,

Sorry for a late response but I was on holiday and not able to reply any sooner.

Yes, you are right that this code looks a bit strange. This part of the code will arrange for a "use <dbname>" command, (if required because of a -D parameter for example), after a connection is successfully setup. So basically the connection succeeded but if ct_cmd_alloc fails here, it will not issue the "use" command but go to connect_succeed anyway. I assume the author of this piece of code decided to go for a connect_succeed as the connection was still successful and handle CT problems later on. But this might still lead to undesired side-effects. So I think it is better to goto connect_fail here. I will fix it in the next release.

Thanks for pointing this out.

Best regards,
Martin.

[Slava K.]

Hope you had good holidays, Martin. And now I'm also back from vacation. :)
Thanks for looking into issue I reported in the last post. Seems like it should not be hard to fix.

All the best,
Slava.