Author Topic: Programatically download the latest 64 bit version of Firefox for Windows.  (Read 4099 times)

Offline AIR

  • BASIC Developer
  • Posts: 504
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #15 on: October 28, 2018, 11:59:42 AM »
Script BASIC

Note: Doesn't meet the challenge requirement for displaying download status live. I'm not sure the user would have time to read it anyways.


How about something like this?


riveraa@nas:~/src$ ./sb64-x86_64.AppImage getFirefox.sb
Downloading Latest 64Bit Firefox (63.0) for Windows.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 42.3M  100 42.3M    0     0  71.2M      0 --:--:-- --:--:-- --:--:-- 81.3M

Firefox_Setup-63.0.exe Downloaded 44,400,072 Bytes at 74,754,643 Bytes/Second.



Add the following to your code before the final CURL::PERFORM:

Code: Script BASIC
  1.     curl::option(ch,"NOPROGRESS",0)

AIR.

Offline John

  • Forum Support / SB Dev
  • Posts: 2345
    • ScriptBasic Open Source Project
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #16 on: October 28, 2018, 12:09:08 PM »
Too cool!

The Script BASIC submission post has been updated.

I had this cURL progress display working in a example of long ago. I just couldn't remember how I did it.

Nice job AIR. This is what sets you apart from everyone else. Your curiosity, experience and being a generous contributor.

FWIW: I'm wondering if what curl::info(ch,"SIZE_DOWNLOAD") is the accumulation of all bytes downloaded since the INIT of the cURL library.
« Last Edit: October 28, 2018, 01:01:04 PM by John »

Offline John

  • Forum Support / SB Dev
  • Posts: 2345
    • ScriptBasic Open Source Project
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #17 on: October 28, 2018, 05:52:38 PM »
I download 'Firefox Setup 63.0.exe' directly from the web site and it matched (size) to what SB / cURL is saying for the download size. This solves the question I had about cURL accumulating its downloaded byte count giving a wrong value. It seems it's the cURL progress function not giving acurate file size info.

Quote
--compressed
(HTTP) Request a compressed response using one of the algorithms curl supports, and save the uncompressed document.  If this option is used and  the  server  sends  an unsupported encoding, curl will report an error.

I wonder If the progress function is seeing the compressed request byte count / rate.
« Last Edit: October 28, 2018, 07:02:11 PM by John »

Offline AIR

  • BASIC Developer
  • Posts: 504
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #18 on: October 28, 2018, 09:21:05 PM »
I download 'Firefox Setup 63.0.exe' directly from the web site and it matched (size) to what SB / cURL is saying for the download size. This solves the question I had about cURL accumulating its downloaded byte count giving a wrong value. It seems it's the cURL progress function not giving acurate file size info.

I believe it's a timing issue.  The download completes during the callback cycle, and the value may not be updated.

Quote
--compressed
(HTTP) Request a compressed response using one of the algorithms curl supports, and save the uncompressed document.  If this option is used and  the  server  sends  an unsupported encoding, curl will report an error.

I wonder If the progress function is seeing the compressed request byte count / rate.

You have to actually enable that option (CURLOPT_ACCEPT_ENCODING), does SB have that enabled in the Module by default?

AIR.

Offline John

  • Forum Support / SB Dev
  • Posts: 2345
    • ScriptBasic Open Source Project
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #19 on: October 28, 2018, 09:32:59 PM »
It doesn't look like it based on interface.c for the cURL extension module.

Quote

To aid applications not having to bother about what specific algorithms this particular libcurl build supports, libcurl allows a zero-length string to be set ("") to ask for an Accept-Encoding: header to be used that contains all built-in supported encodings.

Seems nothing equals everything.
« Last Edit: October 28, 2018, 11:34:34 PM by John »

Offline John

  • Forum Support / SB Dev
  • Posts: 2345
    • ScriptBasic Open Source Project
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #20 on: October 29, 2018, 03:44:20 PM »
Quote from: AIR
Tomaaz, I know you're out there!!!  Post it on RetroB if you want, I'm interested in seeing how you do this!

Tomaaz was a member here but deleted his own account. I think you are wasting your time on him. His only forum interest is tormenting Aurel and bitching about everything.


Offline John

  • Forum Support / SB Dev
  • Posts: 2345
    • ScriptBasic Open Source Project
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #21 on: October 29, 2018, 04:01:05 PM »
My first thought with this challenge was to use SB sockets to connect to get the Firefox download page. That was abruptly halted by Cloudfare. Not like the good old HTTP days.

Offline AIR

  • BASIC Developer
  • Posts: 504
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #22 on: October 29, 2018, 06:24:07 PM »
I think you need to send a proper user agent , as well as add SSL support to Sockets.

Bacon has the same issue with it's built in Socket support although Peter coded something that incorporates what's needed as a proof of concept.

I think that most of the older BASICS will have this problem as well.  I know that MBC does, but I'm not gonna fix that since it's essentially dead now....

AIR.


Offline John

  • Forum Support / SB Dev
  • Posts: 2345
    • ScriptBasic Open Source Project
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #23 on: October 29, 2018, 06:30:07 PM »
I wonder if James Fuller is still doing his version of BCX?

Offline John

  • Forum Support / SB Dev
  • Posts: 2345
    • ScriptBasic Open Source Project
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #24 on: October 29, 2018, 07:30:02 PM »
I think it would be possible to use wget with a SB SYSTEM or EXECUTE to get the Firefox download page and do the file download with its built in progress display.

The whole challenge could be done with bash, wget and regex.
« Last Edit: October 29, 2018, 07:33:53 PM by John »

Offline AIR

  • BASIC Developer
  • Posts: 504
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #25 on: October 29, 2018, 08:24:38 PM »
I already did a Bash version for work.....LOL.

Edit:

I originally wrote this to download the latest version for macOS; with a minor tweak it works for windows.

There was much more to this for the Mac, which I've stripped out.

Code: Bash
  1. #!/bin/bash
  2.  
  3. # ff.sh
  4. # Script to download the latest version
  5. # of Firefox 64bit for Windows
  6. #
  7. # Written by Armando I. Rivera (AIR)
  8. # Timeinc Global Technology Services
  9.  
  10. XPATH="string(//div[@id='outer-wrapper']/main/div[@id='other-platforms']/div[@class='content']/section[@class='section-other-platforms']/div[@id='download-platform-list-release']/ul[@class='download-platform-list recommended']/li[@class='os_win64']/a/@href)"
  11.  
  12. VERSION=$(curl -s https://www.mozilla.org/en-US/firefox/new/ | xmllint --html --xpath 'string(/html/@data-latest-firefox)' - 2>/dev/null)
  13.  
  14. curl -s https://www.mozilla.org/en-US/firefox/new/ | xmllint --html --xpath "$XPATH" - 2>/dev/null | xargs wget -O "Firefox Setup $VERSION.exe" -q --show-progress
  15.  
« Last Edit: October 29, 2018, 09:48:45 PM by AIR »

Offline John

  • Forum Support / SB Dev
  • Posts: 2345
    • ScriptBasic Open Source Project
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #26 on: October 29, 2018, 09:56:55 PM »
I love it when there is more comment than code.  8)

Offline AIR

  • BASIC Developer
  • Posts: 504
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #27 on: October 30, 2018, 07:06:37 AM »
I think XPATH can be replaced with:

Code: Bash
  1. XPATH="string(//div[@id='outer-wrapper']/*/*/*/*/*/*/li[@class='os_win64']/a/@href)"

 ;D ;D 8)

AIR.

Offline John

  • Forum Support / SB Dev
  • Posts: 2345
    • ScriptBasic Open Source Project
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #28 on: October 30, 2018, 09:02:00 AM »
It's amazing what can be done with Bash. Peter has a version of BaCon written in it. 8)

Offline AIR

  • BASIC Developer
  • Posts: 504
Re: Programatically download the latest 64 bit version of Firefox for Windows.
« Reply #29 on: October 30, 2018, 12:46:32 PM »
Speaking of Peter/Bacon, here's my BACON Version

I've included a zip file with the updated Curl module, as well as a custom REGEX module that I wrote just for this.

Bacon's native REGEX function doesn't return any matches as strings/text.  You have to use the return value from that function, which is where in the file the hit begins, and use the REGLEN builtin to get the length.  Then you need to feed the source, the REGEX return value, and the REGLEN variable to MID$ to retrieve the actual hit.

My function (called ReGex) returns the actual text.  Like Peter's implementation, it's based on Posix Regex, which has it's own arcane regex syntax.

Interestingly, the regex that I feed to my function doesn't work with Peter's; I haven't looked into why that is since I don't think Peter would want to modify his implementation much (if at all) since a few other Bacon functions seem to rely on it.

Anyway, here's the Bacon source (again, the required modules are in the attached zip file)

Code: Text
  1. TRAP LOCAL
  2. INCLUDE "curl.bac"
  3. INCLUDE "Regex-Plus"
  4.  
  5.  
  6. ' We store our data in this variable
  7. DECLARE data$, version_num$
  8. DECLARE regx TYPE mREGEX_type
  9.  
  10. FUNCTION progress_callback(long clientp, double dltotal, double dlnow, double ultotal, double ulnow )
  11.     LOCAL totaldotz, dotz, ii TYPE int
  12.     LOCAL fractiondownloaded TYPE double
  13.     LOCAL tmpStr$
  14.  
  15.     IF dltotal <= 0.0 THEN RETURN 0
  16.     tmpStr$ = CONCAT$((STRING)clientp,"")
  17.    
  18.     totaldotz = 40
  19.     fractiondownloaded = dlnow/dltotal
  20.     dotz = ROUND(fractiondownloaded * totaldotz)
  21.  
  22.     PRINT fractiondownloaded * 100 FORMAT "%3.0f%% [";
  23.  
  24.     WHILE ii < dotz DO
  25.         PRINT "=";
  26.         INCR ii
  27.     WEND
  28.  
  29.     WHILE ii < totaldotz DO
  30.         PRINT " ";
  31.         INCR ii
  32.     WEND
  33.  
  34.     PRINT "] \r";
  35.  
  36.     RETURN 0
  37. END FUNCTION
  38.  
  39. FUNCTION Save_Data(STRING buffer$, size_t size, size_t nmemb, void *userp)
  40.     data$ = CONCAT$(data$, buffer$)
  41.     RETURN size*nmemb
  42. END FUNCTION
  43.  
  44. SUB getData$(STRING url) TYPE STRING
  45.     LOCAL handle TYPE long
  46.     LOCAL success
  47.  
  48.     handle = curl_easy_init()
  49.     curl_easy_setopt(handle, CURLOPT_URL, url)
  50.     curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION, 1)
  51.     curl_easy_setopt(handle, CURLOPT_WRITEFUNCTION, Save_Data)
  52.    
  53.     success = curl_easy_perform(handle)
  54.     curl_easy_cleanup(handle)
  55. END SUB
  56.  
  57. SUB downloadFile(STRING url, STRING downloadFileName)
  58.     LOCAL handle TYPE long
  59.     LOCAL success
  60.  
  61.     PRINT "Downloading Latest 64Bit Firefox (",version_num$,") for Windows.\n"
  62.    
  63.     handle = curl_easy_init()    
  64.     OPEN downloadFileName FOR WRITING AS download
  65.     curl_easy_setopt(handle, CURLOPT_URL, url)
  66.     curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION,1)
  67.     curl_easy_setopt(handle, CURLOPT_WRITEDATA, download)
  68.     curl_easy_setopt(handle, CURLOPT_NOPROGRESS,0)
  69.     curl_easy_setopt(handle, CURLOPT_PROGRESSFUNCTION,progress_callback)
  70.  
  71.     success = curl_easy_perform(handle)
  72.     CLOSE FILE download
  73.     curl_easy_cleanup(handle)
  74.     PRINT "\n\nDownload Complete.\n"
  75. END SUB
  76.  
  77. CLEAR
  78.  
  79. PRINT "Firefox Download Challenge (Bacon Version) by AIR.\n"
  80.  
  81. getData$("https://www.mozilla.org/en-US/firefox/new")
  82.  
  83. regx = RegEx(data$,"data-latest-firefox=.([0-9]+\\.[0-9]+)")
  84.  
  85. IF regx.count THEN
  86.     version_num$ = regx.result$[0]
  87.     PRINT version_num$ FORMAT "Firefox Setup %s.exe" TO version$
  88. END IF
  89.  
  90. regx = RegEx(data$,".+href=\"(http.+latest-ssl&amp;os=win64&amp;lang=en-US)\"")
  91.  
  92. IF regx.count THEN
  93.     downloadLink$ = regx.result$[0]
  94. END IF
  95.  
  96. downloadFile(downloadLink$, version$)
  97. PRINT
  98.  
  99.  

AIR.

P.S.  Also like Peter's function, my RegEx function only returns a single (the first) hit.  I'll see about tweaking it so it will return multiple hits like most are used to seeing when using a library like PCRE...