Gpg-agent caching config for pysvn
pysvn is the pythonic interface to subversion
Brought to you by:
barry-scott
Menu
▾
▴
#2 Gpg-agent caching config for pysvn
Thank you so much for pysvn. It has been a lifesaver for us.
Recently we were trying to movefrom plaintext authentication to gpggagent password store.
It works when we use svn commandline with the pinentry. The gpg agent daemon is running. I've set the config file and servers file to avoid plaintext and allow storing of passwords. The password stores setting is set to gpggagent.
It works fine with svn but not through pysvn. I saw that you got it to work with gpg. Could you please provide the instructions required to get this to work? I can't find these anywhere on the web.
Thanks!
It seems that the environment of the process that is runing svn or pysvn is critical to getting this too work. I quick search found this:
https://stackoverflow.com/questions/41142125/svn-setup-with-gpg-agent-and-pinentry-ttycurses
Let me know if this helps.
Thanks for the quick reply!
I tried adding the allow-preset-passphrase to ~/.gnupg/gpg-agent.conf with the same results:
We've got an X server and pinentry dialog is showing fine, so it's not exactly the same issue as that link.
Here is the behavior I'm seeing (basically that svn commandline works with gpg-agent but pysvn does not use it):
Gpg agent is running:
The first time it asks for a password inline:
The second time it opens a Pinentry dialog asking for the password:
...instead of asking from the command line (this was reported by others too, so it's normal and acceptable for us as long as it stores it in gpg):
The third time it does not ask for a password or open a dialog, so it has stored that password successfully in gpg:
And it can be verified as saved in the svn.simple auth file:
But when trying it through our python script which contains the following pysvn related snippets:
And our ~/.subversion/config contains the following:
And our ~/.subversion/servers contains the following:
The result is that it keeps calling the callback_get_login and asking for the username and password and does not store it using gpgagent like the svn commandline does.
But the minute I comment the store-plaintext-passwords line in the servers file, it stores the plaintext auth and does not repeat the callback for login after storing it in plaintext inside the 2aauthfile.
Sorry, I haven't dug into the pysvn source code yet for this, I'm hoping it is a simple config issue.
Last edit: John Snow 2019-11-13
pysvn wraps around the svn code. pysvn has no GPG specific code in it.
It would be helpful if you could do some debugging on this.
I read a bit of svn code in subversion/libsvn_subr/gpg_agent.c to get a sense of what
might be going on. Looks like the svn code relies on the a GPG "pipentry" program.
I did see this about problems with pinentry:
https://superuser.com/questions/520980/how-to-force-gpg-to-use-console-mode-pinentry-to-prompt-for-passwords
Just a wild thought. Does your code work if you do not setup the callback_get_login?
Seeing as the pinentry reads and stores the password why would svn use the get_login
callback?
Ok, it's good to know it has no GPG specific code in it. Is it possible to check which svn binary pysvn calls?
I tried reloading GPG with pinentry set in the config as mentioned in the article. It's the same behavior. I'm using GNOME and a terminal within CentOS 7.5.1804 and there don't appear to be any issues with Pinentry when using svn. It shows the dialog for Pinentry with the password input and stores it afterwards in GPG. The problem might be that when invoked through pysvn, svn does not consider GPG at all. I see why you are trying to be explicit about settings for svn and GPG.
Disabling the callback_get_login throws an exception "callback_get_login required". I remember trying this before too :-) The call back is supposedly thrown any time svn does not have credentials cached. But why would svn see the credentials as cached but pysvn overwrite it and not use any password store or caching? I'll try the set-auth-cache thing too with all the recent changes. Also, is there a way to check if my pysvn is using a different svn or check the commands that pysvn is running finally or enable logging or debug mode?
All the gnupg and pinentry stuff came preinstalled in CentOS.
I'm guessing you are on a Centos 7.
What repos are you using to get svn, gnupg, pysvn, etc from?
What versions of python, pysvn and svn are you using?
pysvn does not use the svn binary it uses the svn libraries.
Then use and look for the libsvn_XXX.so files:
Last edit: Barry Alan Scott 2019-11-11
Thanks, yes, CentOS 7.5.1804.
svn was gotten through the Wandisco repo.
$ rpm -qa | grep subversion
subversion-1.9.12-1.x86_64
subversion-javahl-1.9.12-1.x86_64
pysvn I'm not sure as it was a part of the conda environment and already preinstalled.
Confirm that the libs are from the RPM that you expect with:
$ rpm -qf /usr/lib64/libsvn_client-1.so.0.0.0It should tell you its the subversion-1.9.12-1.
Are you ok using gdb? Next you need to find out what svn is upto when
it needs a password frpm gpg-agent.
There are 3 entry points in the gpg_agent.c files.
simple_gpg_agent_first_creds, simple_gpg_agent_next_creds, simple_gpg_agent_save_creds.
Can you see how there operate from svn and from pysvn and find the difference?
I wonder if under python the gpg agent socket cannot be found?
Last edit: Barry Alan Scott 2019-11-11
$ rpm -qf /usr/lib64/libsvn_client-1.so.0.0.0
subversion-1.9.12-1.x86_64
I'm not sure how I can use gdb to check on those entry points. I was looking at a tutorial here: https://web.eecs.umich.edu/~sugih/pointers/summary.html
Do I need to get the gpg_agent.c source code and compile it and run with gdb or can I use the existing .so files or directly invoke the command somehow?
[lnxusr@localhost .gnupg]$ gdb svn
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/bin/svn...Reading symbols from /usr/lib/debug/usr/bin/svn.debug...done.
done.
(gdb) ^Z
[3]+ Stopped gdb svn
Ok just figured out i can run with arguments inside the gdb command line. Might need more debug info packages. The below ran the first run (inline password request) and the second run (which shows pinentry dialog):
$ gdb svn
...
(gdb) run info https://svnserver:443
Starting program: /usr/bin/svn info https://svnserver:443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Authentication realm: https://svnserver:443 VisualSVN Server
Password for 'svn_user': ****
svn: E170013: Unable to connect to a repository at URL 'https://svnserver'
svn: E175009: The XML response contains invalid XML
svn: E130003: Malformed XML: no element found
[Inferior 1 (process 3377) exited with code 01]
Missing separate debuginfos, use: debuginfo-install gssproxy-0.7.0-17.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-19.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64 libselinux-2.5-12.el7.x86_64 libuuid-2.23.2-52.el7_5.1.x86_64 nspr-4.19.0-1.el7_5.x86_64 nss-3.36.0-7.el7_5.x86_64 nss-softokn-freebl-3.36.0-5.el7_5.x86_64 nss-util-3.36.0-1.el7_5.x86_64 openldap-2.4.44-15.el7_5.x86_64 openssl-libs-1.0.2k-12.el7.x86_64 pcre-8.32-17.el7.x86_64
(gdb) run info https://svnserver:443
Starting program: /usr/bin/svn info https://svnserver:443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
svn: E170013: Unable to connect to a repository at URL 'https://svnserver'
svn: E175009: The XML response contains invalid XML
svn: E130003: Malformed XML: no element found
[Inferior 1 (process 3383) exited with code 01]
(gdb) run info https://svnserver:443
Starting program: /usr/bin/svn info https://svnserver:443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Authentication realm: https://svnserver:443 VisualSVN Server
Password for 'svn_user': ****
svn: E170013: Unable to connect to a repository at URL 'https://svnserver'
svn: E175009: The XML response contains invalid XML
svn: E130003: Malformed XML: no element found
[Inferior 1 (process 3377) exited with code 01]
Missing separate debuginfos, use: debuginfo-install gssproxy-0.7.0-17.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-19.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64 libselinux-2.5-12.el7.x86_64 libuuid-2.23.2-52.el7_5.1.x86_64 nspr-4.19.0-1.el7_5.x86_64 nss-3.36.0-7.el7_5.x86_64 nss-softokn-freebl-3.36.0-5.el7_5.x86_64 nss-util-3.36.0-1.el7_5.x86_64 openldap-2.4.44-15.el7_5.x86_64 openssl-libs-1.0.2k-12.el7.x86_64 pcre-8.32-17.el7.x86_64
(gdb) run info https://svnserver:443
Starting program: /usr/bin/svn info https://svnserver:443
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
svn: E170013: Unable to connect to a repository at URL 'https://svnserver'
svn: E175009: The XML response contains invalid XML
svn: E130003: Malformed XML: no element found
[Inferior 1 (process 3383) exited with code 01]
Last edit: John Snow 2019-11-11
I'm running that debuginfos command right now and it's getting those packages. I'll run gdb with the above svn command again and run it with pysvn next (might get tricky with a conda environment so I'll do a test without conda first)
Here's a python gdb output which is also high level like the previous one:
GDB is only showing high level thread info. I'm looking into how I can get it to show more details?
I'm guessing your new to gdb. What you need to do is the following (untested):
$ gdb python
...
(gdb) b simple_gpg_agent_first_creds
Function "simple_gpg_agent_first_creds" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
and repeat for the other two entry points.
(gdb) run ...
When it hits the break point there are some useful command.
list - show you the sources where you stopped
step - execute one line of code going into any subroutines
next - execute one line of code going over any subroutines
print - display variables
continue - run until the next break point is hit
Now watch the flow of the logic and where it changes between the svn and pysvn cases.
Thanks for doing this.
Thanks, yes new to gdb. This is right after doing the pinentry through svn
Last edit: John Snow 2019-11-12
Definitely not the same when running it through python. I can't see a single breakpoint get called through python.
See below:
Here's the svntest.py that I executed:
The pysvn code looks to be written mostly in C. I have no idea how to gdb or modify the pysvn library and determine the svn command or library its invoking. I don't really have much more to add at this point. If you have some ideas, please let me know and I will try those too.
If the config used is differnce between svn and pysvn then that would explain things.
When you create the pysvn.Client() you can pass in the config like this:
c = pysvn.Client( config_dir )
Can you try that and see if that makes things work?
Same exact behavior with the config_dir set. I found something that looked bleak for pysvn caching:
It looks like pysvn calls "svn_client_cat2()" from libsvn and this function does not cache username/passwd between invoking even for the same url and same realms.
https://stackoverflow.com/questions/3325489/how-to-cache-username-and-passwd-in-pysvn
Is this true? Has anyone ever gotten this to work?
Last edit: John Snow 2019-11-17
I just confirmed that the gnome keyring password store also results in identical behavior (works through svn but not through pysvn):
The 2a auth file never has a password entry in it through pysvn unless it's a plaintext password.
Last edit: John Snow 2019-11-17
It would be very beneficial for us if this worked.
I looked into other keyring solutions, but can't use other keyring solutions because that would require handing over the password in plaintext to pysvn
From gpg_agent.c:
*
I'm gonna focus my efforts on using GNOME Keyring now. GPG does not look to be a secure method
In this bug it appears you got it to work through GPG!
http://pysvn.tigris.org/issues/show_bug.cgi?id=149
Do you remember how?
Looking at pysvn_svnenv.cpp where the providers get populated, it seems that maybe it's using the provider setting from 1.6
Specifically this line:
if defined( PYSVN_HAS_AUTH_GET_SIMPLE_PROVIDER2 )?
Last edit: John Snow 2019-11-20
I just ran a GDB and it does hit the svn_auth_get_simple_provider2 which is meant for SVN v1.6 and greater. I guess it was just an upgrade to the way this function worked after 1.6, never mind!
Last edit: John Snow 2019-11-20
A lot of plaintext stuff in the gdb though, so it's justnot picking the right simple provider (baton?) for a keyring:
Last edit: John Snow 2019-11-20
Never mind, the difference is somewhere else. SVN also calls plaintext.
Here is the gdb for svn (first run without a stored password and 2nd run with a stored pw):
Last edit: John Snow 2019-11-20