#1298 Support HTTPS

[Anonymous]

As far as I can tell, isfdb.org only supports unencrypted HTTP. This was enough to prevent me from signing up, because I didn't want to submit a password over the internet unencrypted. You could get a free SSL cert from e.g. https://letsencrypt.org/ and depending on your web server, probably also set up a redirect from http:// -> https://

Thanks for making such a cool website! Sorry if I submitted this to the wrong place.

[Ahasuerus]

Thanks for raising the issue! When it was last discussed on the ISDFB Wiki, I wrote:

"Migrating to HTTPS is certainly the right thing to do. There are some administrative issues that we need to resolve first, but it's definitely on the agenda. Unfortunately, there is no ETA at this time."

Unfortunately, nothing has changed in the last 10 months :-( I will keep this Feature Request open to make sure that we don't forgte.

[ErsatzCulture]

I wrote up some brain dump notes on the ISFDB wiki about what might need doing to move to HTTPS here: http://www.isfdb.org/wiki/index.php/User:ErsatzCulture/DevelopmentNotes

Probably most/all of that is already known, but hopefully it's useful to have stuff documented.

My guess is that the cert and webserver work is fairly quick and straightforward, but cleaning up all - or at least, as many as possible - of the off-site links is the real pain point.

[Klaus Elsbernd]

Well, I setup a local isfdb with the current mediawiki 1.35.2, created a CA/Certificate for a https-support.
I had to change entries in the database as follows:

UPDATE notes SET note_note =  REPLACE(note_note,"http://www.isfdb.org/","https://isfdb.fritz.box/") WHERE note_note LIKE '%http://www.isfdb.org/%';
UPDATE pubs SET pub_frontimage = REPLACE(pub_frontimage,"http://www.isfdb.org/","https://isfdb.fritz.box/") WHERE pub_frontimage LIKE 'http://www.isfdb.org/%';
UPDATE authors SET author_image = REPLACE(author_image,"http://www.isfdb.org/","https://isfdb.fritz.box/") WHERE author_image LIKE 'http://www.isfdb.org/%';
UPDATE authors SET author_note =  REPLACE(author_note,"http://www.isfdb.org/","https://isfdb.fritz.box/") WHERE author_note LIKE '%http://www.isfdb.org/%';

It might be, that I left some tables unchanged.

Next I had to change .cgi and .py scripts (in my installation, should be straight forward in the svn-repository) (with command line in unix)
The files are searched for references to http://(%s or http:/%s , copied to <file>.html; then changed and stored into the original files.
Python-Modules will be compiled in the second statement</file>

for i in `find . -type f | xargs egrep -l "http://%s|http:/%s`; do cp -p $i $i.http; sed -e 's%http://\%s%https://\%s%' -e 's%http:/\%s%https:/\%s%' $i.http > `basename $i .http`; done
for i in `find . -name *py.http`; do python -m py_compile  `basename $i .http`; done

since I use Mediawiki 1.35.2, changes to login.cgi had to be made

[Ahasuerus]

Thanks for looking into this issue! A quick note about using "find" and "sed" to change "http" links to "https" ones. As I mentioned over on http://www.isfdb.org/wiki/index.php/User:ErsatzCulture/DevelopmentNotes , the vast majority of hard-coded "http://" occurrences in the ISFDB code should be migrated to "ISFDBLink" in common/library.py. Once it's done, we'll be able to switch between HTTP and HTTPS and between absolute URLs and relative URLs with ease.

[Klaus Elsbernd]

Well, I looked into your notes at ISFDB.
My current setup with replacements of http with https in the python code and the database modifications (I some more) result in a usable ISFDB.
Most unresolved references in the database are to websites with little references, which I can't check manually. Amazon/Google/goodreads/worldcat... have been updated. Of course, all ISFDB-references.
As you mentioned in the article, absolute URLs should be converted in relative URLs. Within the source code, HTMLLOC, HTFAKE and HTMLHOST define references to FQDN of the database. These variables will be inserted at runtime into http://%s and http:/%s strings.
Currently I replaced http: by https:, but I will check, if I can replace them both to "%s", and modify localdefs.py accordingly.
(HTFAKE = "/cgi-bin", ...)
I don't know, if this can be managed by converting these requests to ISFDBLink.
Rem: within ISFDBLink, I changed the occurrence of http to https.

Rem: But I'm not a python-programmer, since programming experiences are long ago. Instead I fix problems in established code, if I can find them. So converting HTTP-references to ISFDBLink is currently beyond my possibilities. Some hints would therefore be appreciated.

As I already mentioned in another article, you can try my installation. Leave a note.

Current database summaries (I use the SQL-statements from the wiki:

MariaDB [ISFDB]> select substring_index(pub_frontimage, '/', 1) domain, count(1) from pubs group by domain;
+--------+----------+
| domain | count(1) |
+--------+----------+
| NULL   |    78824 |
|        |    10003 |
| http:  |   230785 |
| https: |   351004 |
+--------+----------+
4 rows in set (5.951 sec)

MariaDB [ISFDB]> select substring_index(author_image, '/', 1) domain, count(1) from authors group by domain;
+--------+----------+
| domain | count(1) |
+--------+----------+
| NULL   |   211175 |
| http:  |      483 |
| https: |     5227 |
+--------+----------+
3 rows in set (1.266 sec)

MariaDB [ISFDB]> select substring_index(site_url, '/', 1) domain, count(1) from websites group by domain;
+--------+----------+
| domain | count(1) |
+--------+----------+
| http:  |        5 |
| https: |       37 |
+--------+----------+
2 rows in set (0.004 sec)

MariaDB [ISFDB]> select substring_index(url, '/', 1) domain, count(1) from webpages group by domain;
+---------+----------+
| domain  | count(1) |
+---------+----------+
| http:   |   108044 |
| https:  |    72388 |
| test12  |        1 |
| test123 |        1 |
+---------+----------+
4 rows in set (1.199 sec)

[Ahasuerus]

One problem that we have with full ISFDB URLs in record notes is that they make it harder to host the ISFDB system on a different domain. It makes testing, cloning and potential migration to another domain unnecessarily difficult. We have over 4,200 references to "title.cgi" and over 14,300 references to "pl.cgi" in Notes. I am thinking that we need new ISFDB templates for title and publication records, which would be similar to the currently existing OCLC/BL/LCCN/etc templates. They would take ISFDB record numbers and display properly formed URLs using ISFDBLink. It would also eliminate the issue of the title fields in the database and the titles that we use in notes getting out of sync over time.

[Ahasuerus]

Re: "ISFDBLink", it's a function in common/library.py -- see https://sourceforge.net/p/isfdb/code-svn/HEAD/tree/trunk/common/library.py . It's the central function which builds links to other ISFDB Web pages, adds mouse-over text when needed, brackets (if requested), etc.

[Ahasuerus]

To take a step back, what we probably need to do first is get a free cert, install it on the main server and implement 301 redirects for all HTTP traffic. Al von Ruff looked into it earlier this (2021) year, but he has been mostly unavailable lately, so it hasn't been done yet. Once we confirm that everything is working on that side, we can start changing the HTTP links which the ISFDB software generates to HTTPS.

[Klaus Elsbernd]

As an external to the system, I'd like to make only some comments.

You're probably right about the links in Notes. Maybe a template is the solution for the future. But here my thoughts for the past :-) :
The HTTPS problem needs two major steps.
1) get let's encrypt up and running. Redirection of index.html is easy then
2) convert isfdb.org using (only) HTTPS
3) introduce ISFDB-template and convert (new) entries and make database-references entries local

2) The second step can be "divided and conquered" into 2/3 parts
1. change .py/.cgi-scripts
2. change (old) database entries
3. change/update mediawiki entries/data

to 1) I already outlined the conversion of all HTTP-references in .py/CGI-scripts. A polished version of what I've done repeatedly.
All files, which contain http:// | http:/ strings will be copied to files with extensions -HTTP
All files, which contain www.isfdb.org strings will be copied to files with extensions -ISFDB, and then changed.
diff-files will be created in /tmp to show, what has been done
Script attached

to 2) I have written a shell script, which can be used to transfer all(most all) HTTP-references of isfdb.org to a new site. That means, if you like to clone the ISFDB to a new host, you can update all references in the new host to use the new host, including references to pictures and wiki-notes. I think the current hints on cloning the database in http://www.isfdb.org/wiki/index.php/ISFDB_Downloads doesn't cover this.
It can be used to convert to HTTPS too. I used it for myself and polished it somewhat. I'll put the shell script in the attachment. (It can be included in the wiki, if you like)
One can use parameters to add HTTPS support too. Use --help to get some descriptions.

to 3) I've already outlined conversion to media-wiki 1.35.2. There are two problems.
1) First one will be eliminated by the upgrade (in the MySQL-DB, there is on column/table, which is obsolete and referred to by script in a creation of an account in SQLparsing, if I remember correctly.
2) Login-Password check has been modernized by media-wiki and needs some additional code, which I already mentioned in another thread.

[Ahasuerus]

Something fairly obvious -- at least in retrospect -- occurred to me earlier today. The ISFDB software already uses a single function, "FormatNote", to display all notes and synopses. All we have to do is to add a line of code to transparently convert all note references to "http://www.isfdb.org" to "%s//%s" % (PROTOCOL, HTMLHOST) and the note/synopsis problem will be solved :-)

[Ahasuerus]

Part 1 - Add a PROTOCOL global variable to the configuration file to allow switching between HTTP and HTTPS seamlessly. Implemented in common/localdefs.py, installed in SVN 703 on 2021-08-24.

[Ahasuerus]

Part 2 - Use PROTOCOL in common/library.py:

biblio/languages.py
common/library.py

Installed in SVN 704 on 2021-08-24.

[Ahasuerus]

Part 3 - Use PROTOCOL in HTTP headers. Implemented in common/isfdb.py , installed in SVN 705 on 2021-08-24.

[Ahasuerus]

Part 4 - SFE3 links, Awards and Award categories:

common/awardClass.py
common/awardcatClass.py
common/library.py
common/sfe3.py

Installed in SVN 707 on 2021-08-25.

[Ahasuerus]

Part 5 - Award Types and the navigation bar:

common/awardtypeClass.py
common/navbar.py

Installed in SVN 708 on 2021-08-25.

[Klaus Elsbernd]

Well, just updated svn to version 708. Looks good. A diligent but routine piece of work. Thanks a lot.

May I suggest to update although index.html and some links in the mainpage, which points outside of isfdb.org. Like links in blilio/common.py to Creative Common, Vim, DBDesigner, Python, and Venkman (seems not to exist any more)

Rem: MediaWiki 1.36.1 works.
Within my locall installation, it seems, that isfdb.gif, which should be installed into $wgLogo = "$wgStylePath/common/images/isfdb.gif";
The Folder common/images, as described in https://isfdb.fritz.box/wiki/index.php/ISFDB:Full_Linux_Website, doesn't exist. And I assume, that the image I found in css/isfdb.gif isn't the same, isfdb.org uses. Maybe, this is intended?

[Ahasuerus]

Re: $wgStylePath/common/images/isfdb.gif, I have checked the live server. You were right, it's a slightly different image compared to the identically named file in the css subdirectory. I have created SR 197 to add this file to the SVN repository and to update the installation instructions.

Thanks for identifying this problem!

[Ahasuerus]

Thanks for the tip about the links in biblio/common.py! I am currently working on the code that resides in the "common" subdirectory. I am correcting obsolete/changed links as I go through them, e.g. our links to IMDB. Once I finish "common", I will move on to the "biblio" subdirectory and fix the links on the front page. I may even move some of them to the ISFDB FAQ because we haven't used 3 out of the 5 listed tools in many years.

I'll take a look at the gif issue later today.

[Ahasuerus]

Part 6 - Login links and DTD link correction:

common/isfdb.py
common/login.py

Installed in SVN 709 on 2021-08-26.

[Klaus Elsbernd]

Could you please have a look at my suggestion for submitlogin.cgi, which includes authentication for the new MediaWiki 1.26(!?) onwards support. (https://en.wikipedia.org/wiki/PBKDF2 )
Implementation is straight forward: if it seems, that pbkdf2 is needed, then use the new algorithm, otherwise use the old md5

[Ahasuerus]

I don't have the PHP/MediaWiki software installed on my development server, so I am not in a good position to look into it at the moment. I have asked MartyD to take a look when he has a moment -- http://www.isfdb.org/wiki/index.php/User_talk:MartyD#MediaWiki_upgrade .

[Ahasuerus]

Part 7 - SQLparsing, PubClass and SeriesClass:

common/SQLparsing.py
common/pubClass.py
common/seriesClass.py

Installed in SVN 710 on 2021-08-26.

[Ahasuerus]

Part 8 - Fix the Moderator link in the Other Pages section. Implemented in common/navbar.py , installed in SVN 711 on 2021-08-26.

[Ahasuerus]

Part 9 - View submission. Implemented in biblio/view_submission.py, installed in SVN 712 on 2021-08-26.

[Ahasuerus]

Part 10 [the commit says "9" due to an error] - Server-side redirects:

biblio/external_id_search_results.py
biblio/google_search_redirect.py
biblio/se.py
biblio/submitmylanguages.py
biblio/submitmywebsites.py
biblio/submitpreferences.py
common/library.py
edit/addquicktag.py
edit/isfdblib.py
edit/submit_select_award_type.py
edit/submittags.py
edit/submitvote.py
mod/bad_images.py
mod/change_tag_status.py
mod/remove_tag.py
mod/resolve_bad_url.py
mod/resolve_cleanup.py
mod/resolve_empty_containers.py
mod/resolve_sfe3_url.py

Installed in SVN 713 on 2021-08-27.