Xidel Activity
Xidel is a cli webpage scraping tool supporting XPath/XQuery 3 and CSS
Brought to you by:
benibela
Activity for Xidel
-
Virgus posted a comment on discussion Help
Thank you so much for your detailed replies ! I was just passing by and I'm looking forward to read all your info attentively asap. Thanks for having taken the time to reply to me and talk to you soon, V.
-
Reino modified a comment on discussion Help
I'd like also to know if there are tutorials for unexperienced users like I am, I couldn't find any, except for some examples here and there... For Xidel stuff, see https://github.com/benibela/xidel/issues/67#issuecomment-770084663. For XPath/XQuery stuff, see https://github.com/benibela/xidel/issues/106#issuecomment-1627386429. An extensive wiki is still on my to-do-list.
-
I'd like also to know if there are tutorials for unexperienced users like I am, I couldn't find any, except for some examples here and there... For Xidel stuff, see https://github.com/benibela/xidel/issues/67#issuecomment-770084663. For XPath/XQuery stuff, see https://github.com/benibela/xidel/issues/106#issuecomment-1627386429.
-
Hello Virgus, Have you actually removed (or commented) @ECHO OFF to debug your script? The first extraction... %XIDEL% -s %SRCURL% -e "//a[@class='downloadLink']/@href" ...doesn't only extract the first "apk-release-url". It extracts... xidel -s "https://www.apkmirror.com/apk/teamviewer/teamviewer-host/"^ -e "count(//a[@class='downloadLink']/@href)" 50 ...of them. And because of the FOR-loop only the last one will be assigned to %DLHREF%. If you only want the first one, use -e "(//a[@class='downloadLink'])[1]/@href"....
-
Anonymous posted a comment on discussion Help
I finally found a way: wget was just missing the --user-agent parameter and further steps were necessary to get both the final url and a meaningful file name. I didn't make it in avoiding the first wget that is used to get the file download "proxy" page. I wish I could do it via Xidel avoiding a creation of a temporary file. But I failed with xidel getting the content of "https://www.apkmirror.com/apk/teamviewer/teamviewer-host/teamviewer-host-15-43-203-release/teamviewer-host-15-43-203-android-apk-download/download/?key=1d371836842c115a39fe79d1413e29a5ea3619bb"...
-
Hello, I'm new to Xidel and I'm trying to learn how to use it with lots of trial and errors. I managed up to now to script download updates from sourceforge, github and "normal" websites but now I'm facing a new challenge and I might need advise from expert users. I took as an example teamviewer latest apk. Package base url is: https://www.apkmirror.com/apk/teamviewer/teamviewer-host/ To get to the download link I needed to call xidel three times: To get latest release url: https://www.apkmirror.com/apk/teamviewer/teamviewer-host/teamviewer-host-15-43-203-release/...
-
Hello, I'm new to Xidel and I'm trying to learn how to use it with lots of trial and errors. I managed up to now to script download updates from sourceforge, github and "normal" websites but now I'm facing a new challenge and I might need advise from expert users. I took as an example teamviewer latest apk. Package base url is: https://www.apkmirror.com/apk/teamviewer/teamviewer-host/ To get to the download link I needed to call xidel three times: To get latest release url: https://www.apkmirror.com/apk/teamviewer/teamviewer-host/teamviewer-host-15-43-203-release/...
-
Anonymous posted a comment on discussion Help
Hello Reino, thank you for another elaborated reply. - I'll be back when I read it attentively.
-
Hello Michael, sort() is for individual items, like sort(//div[@class='d-flex flex-column flex-md-row my-5 flex-justify-center']//@datetime) for instance. This won't work for sorting nodes. You'll have to use the XQuery "Order By Clause" in a FLWOR expression: xidel -s "https://github.com/Hibbiki/chromium-win32/releases" -e "for $x in //section order by $x//@datetime descending return $x/join((.//@datetime,h2),x:cps(9))" xidel -s "https://github.com/Hibbiki/chromium-win32/releases" -e ^"^ for $x...
-
Anonymous posted a comment on discussion Help
And this is the final result (for now): set "xtr= $srt/join(" set "xtr=%xtr% (.//@datetime, " set "xtr=%xtr% substring(concat(normalize-space(.//span[@class='ml-1 wb-break-all']), ' '), 1, 23 ), " set "xtr=%xtr% div[@class='col-md-9']//h1)" set "xtr=%xtr% , x:cps(9))" set "xtr=%xtr%" for /f "delims=" %%l in ('xidel.exe -s "%fil%" --output-format=cmd --extract-exclude=srt --extract "srt:=reverse(sort(//div[@class='d-flex flex-column flex-md-row my-5 flex-justify-center'], (), function($div){$div//relative-time/@datetime}))...
-
That's a creative way to assign the (prettified) extraction-query to a variable. The outer parenthesis ( ) aren't necessary btw. Alternatively you could of course insert the extraction-query directly (with the necessary escape-characters): FOR /F "delims=" %A IN (' xidel -s "%fil%" -e ^" vrs:^=//div[@class^='d-flex flex-column flex-md-row my-5 flex-justify-center']/join^( ^( ^(.//@datetime^)[1]^, normalize-space^(.//span[@class^='ml-1 wb-break-all']^)^, div[@class^='col-md-9']//h1 ^)^,x:cps^(9^)...
-
Anonymous posted a comment on discussion Help
Sorry I'm a bit delayed - my systems power supply had given up. Reino, I thank you for this piece of code. With it's concreteness and it's clearness it is something I had long been looking for. Especially the clarification of how to combine adressing & processing helped me a lot. ... and this is what I made from it set "xtr= vrs:=(//div[@class='d-flex flex-column flex-md-row my-5 flex-justify-center']/" set "xtr=%xtr% join(" set "xtr=%xtr% (" set "xtr=%xtr% (.//@datetime)[1]," set "xtr=%xtr% normalize-space(.//span[@class='ml-1...
-
Anonymous posted a comment on discussion Help
... I'm stunned - and I have to think over. For the moment i thank you for another midnight lesson.
-
Reino, nice to meet you again ... Wish I could say "likewise", but with you being anonymous that's not so obvious to me. --extract "vrs:=($cls)//normalize-space(span[@class = 'ml-1 wb-break-all'])" That would be... --extract "vrs:=normalize-space($cls//span[@class='ml-1 wb-break-all'])" or... --extract "vrs:=$cls/normalize-space(.//span[@class='ml-1 wb-break-all'])" I'd forget about the use of external variables if I were you, because that's something very difficult to accomplish in cmd, if possible...
-
Anonymous posted a comment on discussion Help
Reino, nice to meet you again ... I'm trying to build 'one liners' like 2022-04-04T14:53:27Z gs9561 Ghostscript/GhostPDL 9.56.1 : I tried --extract "cls:=//div[@class='col-md-2 d-flex flex-md-column flex-row flex-wrap pr-md-6 mb-2 mb-md-0 flex-items-start pt-md-4']" --extract "vrs:=normalize-space($cls//span[@class = 'ml-1 wb-break-all'])" giving SET vrs=gs9561 gs1000rc2 gs10.0.0rc1 gs9560 gs9550 gs9560rc2 gs9560rc1 gs9540 gs9550rc1 gpdf_beta1 --extract "vrs:=($cls)(normalize-space(//span[@class...
-
All these are arrays. Sequences. I get everything - syntax error, endless loop and a string of all entries - but no array of normalized entries. What exactly have you tried? And what is the end-result you're actually looking for?
-
Anonymous posted a comment on discussion Help
I'm working on https://github.com/ArtifexSoftware/ghostpdl-downloads/releases and I'm able to extract the title lines --extract "txt:=//div[@class='col-md-9']//h1" or --extract "txt:=//div[@class='flex-1']/h1" the released versions --extract "cls:=//div[@class='col-md-2 d-flex flex-md-column flex-row flex-wrap pr-md-6 mb-2 mb-md-0 flex-items-start pt-md-4']" --extract "vrs:=$cls//span[@class = 'ml-1 wb-break-all']" the acc. date --extract "cls:=//div[@class='col-md-2 d-flex flex-md-column flex-row...
-
Anonymous posted a comment on discussion Help
... since " double-quotes are no problem anymore :-)
-
Anonymous posted a comment on discussion Help
I used @onclick because I thought, @onclick references the 'true' UUID. When I realized, that UUID is consistantly used all over the site, I switched to @id in the way you mentioned. ... sInce " double-quotes are no problem anymore :-) Thank you very much, Reino
-
No problem. One last thing I forgot to mention: substring-before(@id,'_link') is another option instead of extract(@onclick,'\"(.*)\"',1). Probably easier because there are no double-quotes to deal with.
-
Anonymous posted a comment on discussion Help
What an enlightenment! Every line! ... and so many things I should have known. Reino, I thank you this late night lesson. - I won't forget this one.
-
Multiple things are going wrong here. The only way to correctly export xidel variables with --output-format=cmd in a for-loop is by using "delims=" instead of "tokens=*". See https://stackoverflow.com/a/61972422. https://sourceforge.net/p/xidel/discussion/help/thread/3797db1dc8/ could also be informative. Within a for-loop you shouldn't end a line with a ^. Escape-character-hell! It's not about escaping the \", it's about (.*) between the second pair of double-quotes. In a for-loop the following...
-
Anonymous posted a comment on discussion Help
The input is: https://www.catalog.update.microsoft.com/Search.aspx?q=KB5016676%20windows%207%20-embedded The command is: for /f "tokens=*" %%l in ( 'xidel --output-format=cmd ^ --extract "x86:=//a[contains(text(), 'x86')]" ^ --extract "x64:=//a[contains(text(), 'x64')]" ^ --extract "x86.id:=$x86/extract(@onclick, '\"(.*)\"', 1)" ^ --extract "x86.txt:=$x86/normalize-space(text())" ^ --extract "x64.id:=$x64/extract(@onclick, '\((.*)\)', 1)" ^ --extract "x64.txt:=$x64/normalize-space(text())" ^ "%fil%"')...
-
Could you share the input/source you're processing as well as the exact cmd command and the error-message?
-
Anonymous posted a comment on discussion Help
thx Reino for your instant response on a sunday afternoon. May I have another question ... ??? I try to use this in a FOR-loop - and I can't find a way to get ... extract(@onclick, '.\"(.)\"', 1), ... e.g. extract(@onclick, '.\^"(.)\^"', 1), working because of (DOS) syntax errors. Is there a way to let XIDEL read it's commands from a file ? thx again
-
You could use --extract-exclude: xidel -s "%fil%" --extract-exclude=x86,x64 ^ -e "x86:=//a[contains(text(), 'x86')]" ^ -e "x64:=//a[contains(text(), 'x64')]" ^ -e "x86.id:=$x86/extract(@onclick, '.\"(.*)\"', 1)" ^ -e "x86.txt:=$x86/normalize-space(text())" ^ -e "x64.id:=$x64/extract(@onclick, '.\"(.*)\"', 1)" ^ -e "x64.txt:=$x64/normalize-space(text())" ^ --output-format=cmd There is however no need for so many extraction-queries. One is enough: xidel -s "%fil%" -e "//a[contains(text(), 'x86')]/(x86.id:=extract(@onclick,...
-
Anonymous posted a comment on discussion Help
I'm working on xidel --silent --output-format=cmd ^ --extract "x86:=//a[contains(text(), 'x86')]" ^ --extract "x64:=//a[contains(text(), 'x64')]" ^ --extract "x86.id:=$x86/extract(@onclick, '.\"(.*)\"', 1)" ^ --extract "x86.txt:=$x86/normalize-space(text())" ^ --extract "x64.id:=$x64/extract(@onclick, '.\"(.*)\"', 1)" ^ --extract "x64.txt:=$x64/normalize-space(text())" ^ "%fil%"' which works so far .... How can I avoid output of 'temporary' variables x86 + x64 ? thx for your attention & thx for your...
-
Anonymous posted a comment on discussion Help
VERIFIED thx
-
XPath notation: ECHO {"bitrate":1603000} | xidel -se "$json/decimal(bitrate)" ECHO {"bitrate":1603000} | xidel -se "$json/integer(bitrate)" ECHO {"bitrate":1603000} | xidel -se "$json/int(bitrate)" ECHO {"bitrate":1603000} | xidel -se "$json/bitrate cast as decimal" ECHO {"bitrate":1603000} | xidel -se "$json/bitrate cast as integer" ECHO {"bitrate":1603000} | xidel -se "$json/bitrate cast as int" dot notation, what you're using: ECHO {"bitrate":1603000} | xidel -se "($json).bitrate cast as decimal"...
-
XPath notation: ECHO {"bitrate":1603000} | xidel -se "$json/decimal(bitrate)" ECHO {"bitrate":1603000} | xidel -se "$json/integer(bitrate)" ECHO {"bitrate":1603000} | xidel -se "$json/int(bitrate)" ECHO {"bitrate":1603000} | xidel -se "$json/bitrate cast as decimal" ECHO {"bitrate":1603000} | xidel -se "$json/bitrate cast as integer" ECHO {"bitrate":1603000} | xidel -se "$json/bitrate cast as int" dot notation, what you're using: ECHO {"bitrate":1603000} | xidel -se "($json).bitrate cast as decimal"...
-
Anonymous posted a comment on discussion Help
echo { "bitrate":1603000 } | xidel.exe - --silent --extract "($json).bitrate gives 1.603E6 ... how can I avoid conversion to obtain '1603000' Thx in advance
-
Benito van der Zander posted a comment on discussion Help
That command does not check if the file is JSON It checks if the file is XPath/XQuery with extensions. It would calculate the sum in {"a": 1+2+3} and not allow strings with \" You need the command from the other thread https://sourceforge.net/p/xidel/discussion/help/thread/8312ddcf43/
-
Anonymous posted a comment on discussion Help
Ahhhh !!!!! Notepad++ and Json Viewer/JSON Format plug-in .... that automatically changes "small errors" to the correct code. My code generated the following JSON "reduced_income": { "description": "BL-INK-REDUCERAD - reduced income amount", "type": "integer", "maximum": 999999999, "minimum": 0 }, <---------- note the extra comma } } }, "case_id_crypt": { "description": "ID-ARENDE-HASH - assignment number hash", "type": "string", "maxLength": 70, "minLength": 70 }, and once I ran the JSON format,...
-
Michael Simpson posted a comment on discussion Help
I am running Xidel to check the validity of generated Swagger/schema files and Xidel is complaining with the message above for both the Swagger and the schema response files. I'll attach the schema response file plus the actual BAT file as well as the input file to XIDEL I'm running. Note that the schema response file formats with no errors using https://www.jsonschemavalidator.net/ and the actual swagger formats fine using https://editor.swagger.io/ Any suggestions as to what might be wrong would...
-
Anonymous posted a comment on discussion Help
... wrx
-
Anonymous posted a comment on discussion Help
thx
-
Anonymous posted a comment on discussion Help
The old way was a bug, now it should be correct You need to write (//a[ends-with(@href, '.zip')]/@href)[2]
-
Anonymous posted a comment on discussion Help
test.data <a href="https://file-x64.zip" >https://file-x64.zip</a> <a href="https://file.zip" >https://file.zip</a> xidel-0.9.9.20220102.8308.be361bf9a08e.win32.zip xidel.exe --silent --xpath "//a[ends-with(@href, '.zip')]/@href[1]" test.data https://file-x64.zip ... as expected xidel.exe --silent --xpath "//a[ends-with(@href, '.zip')]/@href[2]" test.data https://file.zip ... as expected xidel-0.9.9.20220131.8335.4e6fcea4d02e.win32.zip xidel.exe --silent --xpath "//a[ends-with(@href, '.zip')]/@href[1]"...
-
Anonymous posted a comment on discussion Help
Thank you. I implemented your second suggestion.
-
Anonymous posted a comment on discussion Help
More like this: "H:\Xidel\Xidel.exe" --input-format json-strict -s "H:\ICA\PRIV.JSON\JSON_FORMATTED\JPB113_SCHEMA_REQUEST.JSON" -e "$json" 1>NUL 2>"H:\A_MS_exe\json_format_results.txt" And this would be faster: "H:\Xidel\Xidel.exe" --input-format json-strict -s "H:\ICA\PRIV.JSON\JSON_FORMATTED\JPB113_SCHEMA_REQUEST.JSON" -e "count($json)" 1>NUL 2>"H:\A_MS_exe\json_format_results.txt"
-
Anonymous posted a comment on discussion Help
Thank you for your suggestion. Just so we're on the same page, is this what you meant? @ECHO OFF del "H:\A_MS_exe\valbuddy.json" /q 2>NUL "H:\Xidel\Xidel.exe" -s "H:\ICA\PRIV.JSON\JSON_FORMATTED\JPB113_SCHEMA_REQUEST.JSON" -e input-format json-strict "$json" 1>NUL 2>"H:\A_MS_exe\json_format_results.txt"
-
It does not really work. If you have an invalid JSON like {"a": 1, } or {"a": 0001 }, it is also loaded You need the --input-format json-strict option to actually check the validity
-
Anonymous posted a comment on discussion Help
Reino. Sorry to have been so slow in acknowledging your answer (which did the trick for me). I think I had a blank after the 2> (either that or 2 >), can'r remember. Be that as it may, the final line I ran was xidel_exe' -s input_file_to_check -e "$json" 1>NUL 2>dummy_output_file If dummy_output_file is empty, then the input file was well-.formed. Thanks again (and apologies again) Michael
-
Hello Michael, This is more of a cmd question than a question about xidel, but no matter. You're probably looking for -e "$json" 2>NUL.
-
First of all, I have to say that the readme.txt that came with Xidel zip file was ....... extensive. So extensive that I couldn't find what I was actually after. I'm running Xidel under Windows, and would like to call it in a simple way so that it checks whether the JSON code in the file is valid or not. I tried the following "h:\xidel\xidel.exe" -s jpb172_swagger.json -e "$json" > xidel_output.json which worked fine inasmuch as xidel_output.json was "pretty formatted" However, when I deliberately...
-
Anonymous posted a comment on discussion Help
I have fixed it: https://sourceforge.net/p/videlibri/code/ci/e8883e74cd6724db0cb157cd4857712cf1ba7216/ All options became undefined when reordering arguments
-
Anonymous posted a comment on discussion Help
I confirm that the original problem is solved by xidel t:\test.json -s --extract "akt.url:=(($json//flyers)()[books/shortName='...' and contains(pdfUrl,'/....')]/pdfUrl)[1]" Why the users specify the file last ... because it's a common convention to specify input files last - exspecially with Xidel-like commands win: find(str) unx: awk, grep, sed because Xidel-Help lacks a 'Global-Syntax'-line like xidel <input> <options> ... like (git)grep [OPTION]... PATTERN [FILE]...</options> ... and because...
-
I also get the same error message with the normal Windows binary, BUT not with the OpenSSL binary. How very peculiar! However... xidel t:\json --extract "($json)/version" With the input first, followed by the extraction query next (as it should), this works just fine. Btw, the parentheses are only needed when you use the dot-notation (-e "($json).version"). Otherwise -e "$json/version" will do just fine. And as of r7880 the stdin dash (-) isn't needed anymore, so ECHO {"version":1} | xidel -e "$json/version"...
-
Anonymous posted a comment on discussion Help
what about this ... xidel --version Xidel 0.9.9 (20210720.7961.2e0374a546a6) echo {"version":1} | xidel - --extract "($json).version" * Processing: stdin:/// * 1 echo {"version":1} | xidel - --extract "($json)/version * Processing: stdin:/// * 1 echo {"version":1}> t:\json xidel --extract "($json).version" t:\json * Retrieving: t:\json * Processing: t:\json 1 xidel --extract "($json)/version" t:\json * Retrieving: t:\json * Processing: t:\json Error: pxp:JSON: PXP Json extensions are disabled Possible...
-
Anonymous posted a comment on discussion Help
... under investigation
-
Anonymous posted a comment on discussion Help
But if I call it like that, it works with --extract (see screenshot) Perhaps --json-mode default or --json-mode unified or --json-mode deprecated helps map:find($json, "books") is the standard equivalent to //books
-
Anonymous posted a comment on discussion Help
... but I have not been able to translate --extract "akt.url:=(($json//books)()[regions/shortName='...' and contains(pdfUrl,'/....')]/pdfUrl)[1]" accordingly
-
Anonymous posted a comment on discussion Help
no, sorry, it's --extract, not --xpath ... and the only other option I use is: --output-format=cmd Another problem I was able to sort out by replacing --extract "fml.pct:=($json/brochures)/pageCount" with --extract "fml.pct:=($json).brochures().pageCount
-
I have disabled that for --xpath because it is not an official W3C-standarized XPath syntax But with --extract it should work If you have some other options (--extract-kind xpath), remove them
-
Anonymous posted a comment on discussion Help
--extract "akt.url:=(($json//books)()[regions/shortName='...' and contains(pdfUrl,'/....')]/pdfUrl)[1]" was fine with xidel-0.9.9.20210708.7941.da5d96270c0e.win32.zip ... xidel-0.9.9.20210720.7961.2e0374a546a6.win32.zip gives Error: pxp:JSON: PXP Json extensions are disabled Possible backtrace: $0049EB44 TXQTERMPATH__EVALUATE, line 3268 of /home/benito/hg/components/pascal/data/xquery_terms.inc: perhaps TXQTermBinaryOp + 3716 ? but unlikely $0049425B TXQTERMSEQUENCE__EVALUATE, line 1323 of /home/benito/hg/components/pascal/data/xquery_terms.inc:...
-
Anonymous posted a comment on discussion Help
Got it; thanks.
-
You can run a function over all nodes with the transform function (which I need to be rename eventually, since xpath 3.1 adds a transform function which is completely unrelated to my old function. might call it for-each-node) https://github.com/benibela/xidel/issues/45 https://sourceforge.net/p/videlibri/mailman/videlibri-xidel/thread/08529395-19ff-a8f9-7865-7b2cc88f9b76%40xs4all.nl/#msg36903285 https://en.wikibooks.org/wiki/XQuery/Filtering_Nodes Someone made a library module with functions for...
-
Anonymous posted a comment on discussion Help
Given a particular node, is there a way to change its name, change or add an attribute, and change or add an attibute value? So, for example, move from <old_name>something</old_name> to <new_name some_attr="some_value">something</new_name> Thanks.
-
Anonymous posted a comment on discussion Help
Got it; thanks.
-
That is not implemented You can run a function over all nodes with the transform function (which I need to be rename eventually, since xpath 3.1 adds a transform function which is completely unrelated to my old function) https://github.com/benibela/xidel/issues/45 https://sourceforge.net/p/videlibri/mailman/videlibri-xidel/thread/08529395-19ff-a8f9-7865-7b2cc88f9b76%40xs4all.nl/#msg36903285 https://en.wikibooks.org/wiki/XQuery/Filtering_Nodes
-
Anonymous posted a comment on discussion Help
Using the xquery update facility (as implemented in BaseX, for example), I can do this: copy $target := <my>titles <your> All Yours </your> </my> modify ( delete node $target//your, insert node <mine>All Mine</mine> as last into $target ) return $target And get this output: <my>titles <mine>All Mine</mine> </my> Is it possible to something like that with xidel?
-
Anonymous posted a comment on discussion Help
It looks like we have to stick to your original answer :D. For xidel stand alone, your new version (with or without --output-declaration in Windows 10) does produce valid xm. But when wrapped inside a FOR/DO it fails, since my_var1 is the last node tag, not the whole input file. The outcome is the same even if, instead of -se "//div/*", you use -se "join(//div/*)" or -se "join(//div/*/outer-xml(.))". So, your original still stands... By way of background (I didn't want to clutter the original question...
-
That's great, but I think it's worth noting however that if you want to output valid xml, then you'd better do... xidel input.xml -se "//div/*" --output-format=xml --output-declaration="<?xml version=\"1.0\" encoding=\"utf-8\"?>" <?xml version="1.0" encoding="utf-8"?> <xml> <p>Hello</p> <span>World</span> </xml> On Win10 you could try without --output-declaration, but as my cmd's code-page defaults to 437 I'm getting <?xml version="1.0" encoding="ibm437"?>
-
Anonymous posted a comment on discussion Help
Nothing is ever simple, but it worked: FOR /F "delims=" %A IN ('xidel input.xml -se "join(//div/*/outer-xml(.))"') DO SET my_var1=%A Thank you very much! Learned something new today.
-
In cmd you can't assign new lines (\r\n) to a variable. The first method only works for 1 single line output, so you'd have to string-join the text-nodes with xidel if you want "Hello World" as output: C:\>FOR /F "delims=" %A IN ('xidel input.xml -se "join(//div/*)"') DO SET my_var1=%A C:\>SET my_var1=Hello World But the tags are stripped. That's what xidel does. If you really want the tags preserved, you'd have to use outer-xml(): xidel input.xml -se "//div/*/outer-xml(.)" <p>Hello</p> <span>Wo...
-
Anonymous posted a comment on discussion Help
The 'direct' method doesn't really work, as it creates a variable only from the text node of the last element. To illustrate, if input.xml is <div> <p>Hello</p> <span>World</span> </div> and I use FOR /F "delims=" %A IN ('xidel "input.xml" -e "//div"') DO SET my_var1=%A then echo %my_var1% outputs World The indirect method FOR /F "delims=" %A IN ('xidel "input.xml" -e "my_var2:=//div" --output-format^=cmd') DO %A works a little better: echo %my_var2% Hello World and gets both nodes. But the tags...
-
Either directly... FOR /F "delims=" %A IN ('xidel "in.xml" -e "//my_node"') DO SET my_var=%A ...or let Xidel do it... FOR /F "delims=" %A IN ('xidel "in.xml" -e "my_var:=//my_node" --output-format^=cmd') DO %A %A is for the command-line. In a batch-file use %%A.
-
Anonymous posted a comment on discussion Help
I'm aware of this instruction in the documentation FOR /F "delims=" %%A IN ('xidel http://site -e "title:=//title" -e "links:=//a/@href" --output-format cmd') DO %%A and of these answers on Stack Overflow but I still get errors (Windows 10) mostly along the lines of %%A was unexpected at this time. So: if I have file "in.xml" and I want to select node <my_node> and export it to a cmd variable "my_var" - what exactly do I need to do? Thanks.
-
Anonymous posted a comment on discussion Help
Thanks
-
Anonymous posted a comment on discussion Help
https://www.msys2.org/ Windows MSYS2: $ xidel https://google.com -e //title **** Retrieving (GET): https://google.com **** **** Processing: https://www.google.com/ **** Windows cmd: > xidel https://google.com -e //title **** Retrieving (GET): https://google.com **** **** Processing: https://www.google.com/ **** Google I can run almsost any command in MSYS2 except xidei on Windows
-
This discussion was already finished, so I suggest you start a new discussion, instead of hijacking this one.
-
Anonymous posted a comment on discussion Help
xidel 0.9.8 does not work with MSYS2 on Windows, not results return, but works with Windows cmd: Windows MSYS2: $ xidel https://google.com -e //title **** Retrieving (GET): https://google.com **** **** Processing: https://www.google.com/ **** Windows cmd: > xidel https://google.com -e //title **** Retrieving (GET): https://google.com **** **** Processing: https://www.google.com/ **** Google I can run almsost any command in MSYS2 except xidei on Windows
-
That's good to hear! You're welcome.
-
tholeunseen posted a comment on discussion Help
It worked perfectly, Thanks a lot!
-
Hello Reino, Thank a million for your suggestion. I am speachless. I will try it today
-
There are different ways to accomplish this, but this is how I would do it. I'm going to assume 'input.csv': urltoparse,tokentoparse http://intranet/readingarea/pages/service/personalpages/D41952.aspx,dappl http://intranet/readingarea/pages/service/personalpages/D41959.aspx,dappl I'm also going to assume the urls with the tokens are inside a <img src="*.jpg"> node. To ignore the first line in 'input.csv' I'd create a sequence of every new line and select line #2 and onward: xidel -s input.csv -e...
-
Sure, the file is a csv file that looks like this: urltoparse,tokentoparse http://intranet/readingarea/pages/service/personalpages/D41952.aspx,dappl http://intranet/readingarea/pages/service/personalpages/D41959.aspx,dappl I was hoping that Xidel could connect and parse the source code of each webpage in the "urltoparse" column and in that webpage find all url references that contains the string "tokentoparse" and output the entire url. I can also use a solution where the file only contains url:...
-
Could you post an example of this text-file?
-
Hello, I have stubled upon this very interesting tool while trying to solve a problem. Basically I habe a text file containing 2 columns, column (1) contains a url, column (2) a token to parse. I would like to parse the source the webpage specified in column (1) for url that contain the tokem specified at columne (2) and if found, output the url to a file. Is this possible with this tool? I could not see an option to read the list of webpages from a file. To make it more simple, I could skip parsing...
-
why is it still so complex? what is wrong with the "except" and the "not"|"!=" solutions which, for my understanding are semanticaly equivalent (though the "transform" way offers much more possibilities). when you select an element with a normal xpath query, you get the entire element - with all its descendants. * except e can distinguish between <a>..</a> and <e>..</e>, it never looks at the decendants. <a><e>..</e></a> is an a-element, so you get all of it. * except //e does the same. though, as...
-
Anonymous posted a comment on discussion Help
Exactly what i wanted to avoid and you've proven me wrong...Arghhhh, it hurts my little pr'I'de. I also scratched the surface of xslt and the templates, just to turn my head, kind of scared by what was hiding in there. Some more work and courage could have made me dig out some thing like your solution. But again, why is it still so complex? what is wrong with the "except" and the "not"|"!=" solutions which, for my understanding are semanticaly equivalent (though the "transform" way offers much more...
-
See http://videlibri.sourceforge.net/xidel.html#examples #13. I believe this query should work: $ xidel -s --xml b.xml -e 'transform(/,function($x){$x/(if (name()="e") then () else .)}) ' <?xml version="1.0" encoding="ibm437"?> <d/> <c> <d> </d> </c> <d/> <c> </c> <d/> encoding="ibm437"? My shell's code page is CP437. At the moment I don't know how I can have it put out encoding="utf-8"?. Benito would have to answer that.
-
See http://videlibri.sourceforge.net/xidel.html#examples #13. I believe this query should work: $ xidel -s --xml b.xml -e 'transform(/,function($x){$x/(if (name()="e") then remove($x,1) else .)}) ' <?xml version="1.0" encoding="ibm437"?> <d/> <c> <d> </d> </c> <d/> <c> </c> <d/> encoding="ibm437"? My shell's code page is CP437. At the moment I don't know how I can have it put out encoding="utf-8"?. Benito would have to answer that.
-
Zui posted a comment on discussion Help
just in case you needed more concrete case, i made this : OK i build an example of an xml content of which i want to remove all the eleent with certain carateristics... say, all the <e> elements :</e> $ cat b.xml <d></d> <c> <d> <e> </e> </d> </c> <d></d> <c> <e></e> </c> <d></d> these two commands leaves all <e> in place $ xidel b.xml --xml -e '[not(self::e)]' $ xidel b.xml --xml -e ' except e'</e> again, thank you for your help, zui 201908062 Hello, I've been trying different ways to remove all...
-
Hello, I've been trying different ways to remove all the Paths elements from the settings file of doublecmd : doublecmd.xml (i use it in linux : https://sourceforge.net/p/doublecmd/wiki/Download/) a reason to want to do this is that when backing up to move around i don't want to carry non-pertinent data So i've used except in : $ xidel doublecmd.xml -e '* except Paths' and several variations prepending paths like // and * with <paths> never gone</paths> I also applied : -e '*[not(self::Paths)]' with...
-
Seeing you're on Windows that's because using \" inside double quotes will get you into lots of trouble. Use single quotes: FOR /F "delims=" %%A IN ('xidel -s "page.html" -e "name:=json(substring-after (//script,'myvar ='))/name" -e "lastname:=json(substring-after(//script,'myva r ='))/lastname" --output-format^=cmd') DO %%A You can use 1 query to export both variables btw: FOR /F "delims=" %%A IN ('xidel -s "page.html" -e "json(substring-after(//scr ipt,'myvar ='))/(name:=name,lastname:=lastname)"...
-
Seeing you're on Windows that's because using \" inside double quotes will get you into lots of trouble. Use single quotes: FOR /F "delims=" %%A IN ('xidel -s "page.html" -e "name:=json(substring-after(//script,'myvar ='))/name" -e "lastname:=json(substring-after(//script,'myvar ='))/lastname" --output-format^=cmd') DO %%A You can use 1 query to export both variables btw: FOR /F "delims=" %%A IN ('xidel -s "page.html" -e "json(substring-after(//script,'myvar ='))/(name:=name,lastname:=lastname)"...
-
Anonymous posted a comment on discussion Help
It works! Thank you!
-
probably. the linux arm version might work: https://sourceforge.net/projects/videlibri/files/Xidel/Xidel%20development/ Or you could try to install freepascal on the raspberry and compile xidel on it
-
Anonymous posted a comment on discussion Help
Hello, Is there a way to install Xidel on a Raspberry Pi 3? Thank you
-
lemale posted a comment on discussion Help
Thanks Benito, it works perfectly! The command works fine, but when there is a script tag before it doesn't work. How do I specify the script tag I want to get? I also want to save the data in variables, but I seem to be doing something wrong. I'm using this command: for /f "delims=" %%a in ('xidel "page.html" -e "name:=json(substring-after(//script, \"myvar =\"))/(name)" -e "lastname:=json(substring-after(//script, \"myvar =\"))/(lastname)" --output-format cmd') do %%a
-
yes, but the script is treated as plaintext and you need to get the text with the object first. e.g. json(substring-after(//script, "myvar ="))/(name,lastname) or json(extract(//script, "myvar *= *(\{.*\})", 1, "s"))/(name,lastname) or in xidel 0.9.9: json(substring-after(//script, "myvar ="))?*
-
yes, but the script is treated as plaintext and you need to get the text with the object first. e.g. json(substring-after(//script, "myvar ="))/(name,lastname) or json(extract(//script, "myvar *= *(\{.*\})", 1, "s"))/(name,lastname)
-
I want to extract data from a json that is defined in a javascript variable. This looks like this: ...HTML CODE... <script> var myvar = { name: "John", lastname: "Doe" } </script> ...HTML CODE... I need to get: John Doe Can I do it using Xidel?
-
Thanks! I didn't know that parameter. Unfortunately the documentation is very little.
-
There is the --proxy option
-
I don't normally use CSS expressions (Benito would have to answer that), but an XPath expression on Linux would be: xidel "https://www.fanfiction.net/s/10572453/57/Alias" -f '//span/button[starts-with(.,"Next")]/extract(@onClick,"'\''(.+)'\''",1)' -e '$url' **** Retrieving (GET): https://www.fanfiction.net/s/10572453/57/Alias **** **** Processing: https://www.fanfiction.net/s/10572453/57/Alias **** **** Retrieving (): https://www.fanfiction.net/s/10572453/58/Alias **** **** Processing: https://www.fanfiction.net/s/10572453/58/Alias...
-
Hi, I want to request a URL through a proxy. Is it possible to use Xidel with a proxy with a syntax similar to CURL? curl -x host:port -U username:pass
-
JGF posted a comment on discussion General Discussion
Not sure if it's some third party library causing this. But thought you should have a heads up : https://www.virustotal.com/#/file/96854c2be1e3755f56fabb8f00d1fe567108461b9fab139039219a1b7c17e382/detection (Baidu and Cylance detect issues)
1 >
