Menu

#5822 Calculate download sizes rather than hardcoding them

Fixed
2020-03-12
2020-03-06
No

Calculate download sizes rather than hardcoding them

This has several problems yet. The most relevant problem may be that
I have no clue how our actual webpages are being compiled, so this
likely would render the web pages silent about download sizes.

It works on the offline-root web pages intended as HTML documentation.
However, at the completion of make doc, numerous files are flagged as
not being available for size calculations. This is not really a
problem with this patch: it turns out that the respective links would
actually all be dead. So this is something that should be fixed
independently.

In contrast, the live webpage may have some links where the script
complains that aren't dead as much as dependent on the browser's
language settings (which is a bad idea since it does not allow
browsing documentation that does not correspond to your selected
browser language).

Contains two commits:

Replace hardwired download sizes with HTML comment

Download sizes given in web page and HTML pages tended to have
diverged a lot from reality. This patch removes all of the download
sizes but adds special HTML comments into the HTML files that
previously contained size estimates. These comments can then be
replaced by the actual sizes with a postprocessing script.

As a consequence, PDF and Info versions now no longer contain download
size estimates.

Replace HTML size comments with human-readable size

This calculates download sizes as the last step of HTML generation and
puts them into the HTML files for reference.

The work is done by scripts/build/fix-docsize.sh .

It does that by replacing the size-requesting HTML comments in the
files given on the command line by the respective human-readable size.

scripts/build/fix-docsize.sh is run in the respective directories as
the final step of generating the HTML versions.

This script uses ls -sh in order to get a human-readable size spec.

http://codereview.appspot.com/567340043

Discussion

  • Anonymous

    Anonymous - 2020-03-06
    • Description has changed:

    Diff:

    
    
    • Needs: -->
    • Patch: new --> needs_work
     
  • Anonymous

    Anonymous - 2020-03-06

    This fails right at the end of a full make doc.

    ...
    Making Documentation/zh/out-www/web-big-page.html < texi
    /home/jlowe/lilypond-git/build/.././Documentation/po/GNUmakefile:30: warning: overriding recipe for target 'po-update'
    /home/jlowe/lilypond-git/build/../stepmake/stepmake/podir-targets.make:14: warning: ignoring old recipe for target 'po-update'
    Making Documentation/out-www/web-big-page.zh.html (hard link)
    Making Documentation/out-www/web/index.zh.html (hard links)
    Making .htaccess 
    Making out-www/offline-root/index.html 
    Mirroring...
    Processing HTML pages for offline target...
    fix-docsize: web.pdf not accessible from /home/jlowe/lilypond-git/build/out-www/offline-root/Documentation/web-big-page.it.html
    fix-docsize: web.pdf not accessible from /home/jlowe/lilypond-git/build/out-www/offline-root/Documentation/web-big-page.fr.html
    
    ... more of the same for all the languages ... then ends as:
    
    fix-docsize: music-glossary.es.pdf not accessible from /home/jlowe/lilypond-git/build/out-www/offline-root/Documentation/web-big-page.es.html
    fix-docsize: snippets.es.pdf not accessible from /home/jlowe/lilypond-git/build/out-www/offline-root/Documentation/web-big-page.es.html
    fix-docsize: web.es.pdf not accessible from /home/jlowe/lilypond-git/build/out-www/offline-root/Documentation/web-big-page.es.html
    fix-docsize: web.pdf not accessible from /home/jlowe/lilypond-git/build/out-www/offline-root/Documentation/web-big-page.de.html
    fix-docsize: přispěvatel-big-page.html not accessible from /home/jlowe/lilypond-git/build/out-www/offline-root/Documentation/web-big-page.cs.html
    fix-docsize: přispěvatel.pdf not accessible from /home/jlowe/lilypond-git/build/out-www/offline-root/Documentation/web-big-page.cs.html
    fix-docsize: web.pdf not accessible from /home/jlowe/lilypond-git/build/out-www/offline-root/Documentation/web-big-page.cs.html
    

    The messages repeat for all the languages

     
    • David Kastrup

      David Kastrup - 2020-03-06

      What do you mean with "it fails"? A load of diagnostics at the end is expected because our big pages are broken to a good degree. If you try navigating the pages, the links for which the diagnostics are output are dead. That is not new but the preexisting state of things that nobody has fixed so far.

      A failure would be if Make exits with a non-zero status. I don't see that from what you quote.

      Yes, the ultimate state we want is no diagnostic at all. And hopefully this load of notifications will make people work towards it, and fix the dead links we have had for years to decades.

       
      • Anonymous

        Anonymous - 2020-03-07

        Well, without that first patch the make doc run would end with

        ...
        Making Documentation/out-www/web/index.zh.html (hard links)
        Making .htaccess 
        Making out-www/offline-root/index.html 
        Mirroring...
        Processing HTML pages for offline target...
        

        and that would be it, I'd get a prompt.

        So when i see something extra like this at the end, I assume something is 'wrong' - so maybe 'fails' is not very accurate (sorry), which is why I also try to cut paste the errors or if appriopriate point to a log file.

         
        • David Kastrup

          David Kastrup - 2020-03-07

          Well, yes, something is wrong. It's just that my script is now flagging it as wrong whereas we previously did not bother checking that the documentation links worked. At any rate, I need to rebase the patch right now since I now need to cater for the Portuguese translation as well (it has just been merged in). There will still be a wagonload of those messages at the end. I definitely hope that you'll not be the only one annoyed by them. As they are getting fixed, so will the documentation links start working. At least that is the idea.

           
          • David Kastrup

            David Kastrup - 2020-03-08

            Ok, I was squeamish about this issue because I did not know to what degree it works and what is missing. Here are the problems to expect:

            a) outdated size estimates for unreachable links in the offline HTML documentation will disappear
            b) outdated size estimates on the website may disappear irrespective of reachability: I just have no idea how the website generation works. Reinstating basic support is likely a one-liner in some script or setup file

            Problem a) is a straightforward improvement making it much easier for translators to diagnose our current problems with translated links.

            Problem b) is a moderate regression not causing functional damage to the live web pages and being a good reference point for making the size links work in the webpage.

            It is a bit unfortunate that so far nobody could be bothered about what I might have been missing about the expected problem b) and where it needs to get addressed. Letting the proverbial silt hit the fan may be what it takes, and the damage is confined. People will no longer get warned about the notation manual being 30MB in size before they download it, but then it actually is something like 6.5MB in size, so that warning is of moderate usefulness. The warnings for the big-page HTML are probably less absurdly wrong. But I doubt many people download those.

            In summary, it seems more worthwhile to go ahead rather than to refrain. Unless of course relevant input arrives on the code review that allows to improve things even before continuing.

             
  • Anonymous

    Anonymous - 2020-03-07
    • Needs: -->
    • Patch: new --> review
    • Type: -->
     
  • Anonymous

    Anonymous - 2020-03-07

    Passes make, make check and a full make doc

     
    • Anonymous

      Anonymous - 2020-03-07

      This was for patch set 2 (in case you are not sure)

       
  • Anonymous

    Anonymous - 2020-03-07
    • Needs: -->
    • Patch: new --> review
    • Type: -->
     
  • Anonymous

    Anonymous - 2020-03-07

    Passes make, make check and a full make doc (patch set 3)

     
  • Anonymous

    Anonymous - 2020-03-09
    • Patch: review --> countdown
     
  • Anonymous

    Anonymous - 2020-03-09

    Patch on coumntdown for March 11th.

     
  • Anonymous

    Anonymous - 2020-03-11
    • Patch: countdown --> push
     
  • Anonymous

    Anonymous - 2020-03-11

    Patch counted down, please push.

     
  • Jonas Hahnfeld

    Jonas Hahnfeld - 2020-03-12
    • labels: --> Fixed_2_21_0
    • status: Started --> Fixed
    • Patch: push -->
     
  • Jonas Hahnfeld

    Jonas Hahnfeld - 2020-03-12
    commit ee197383f4af552ed433c496617cb5ffe2a28dcf
    Author:     David Kastrup <dak@gnu.org>
    AuthorDate: Wed Feb 26 02:07:27 2020 +0100
    Commit:     David Kastrup <dak@gnu.org>
    CommitDate: Wed Mar 11 18:25:21 2020 +0100
    
        Issue 5822/2: Replace HTML size comments with human-readable size
    
        This calculates download sizes as the last step of HTML generation and
        puts them into the HTML files for reference.
    
        The work is done by scripts/build/fix-docsize.sh .
    
        It does that by replacing the size-requesting HTML comments in the
        files given on the command line by the respective human-readable size.
    
        scripts/build/fix-docsize.sh is run in the respective directories as
        the final step of generating the HTML versions.
    
        This script uses ls -sh in order to get a human-readable size spec.
    
    commit 1c1b076adf7458ff90efb3766ef46cba026b0af6
    Author:     David Kastrup <dak@gnu.org>
    AuthorDate: Thu Mar 5 23:47:18 2020 +0100
    Commit:     David Kastrup <dak@gnu.org>
    CommitDate: Wed Mar 11 18:25:21 2020 +0100
    
        Issue 5822/1: Replace hardwired download sizes with HTML comment
    
        Download sizes given in web page and HTML pages tended to have
        diverged a lot from reality.  This patch removes all of the download
        sizes but adds special HTML comments into the HTML files that
        previously contained size estimates.  These comments can then be
        replaced by the actual sizes with a postprocessing script.
    
        As a consequence, PDF and Info versions now no longer contain download
        size estimates.
    
     
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.