Activity for Open Source Data Quality and Profiling
-
arrah posted a comment on discussion Help
Hi, First you can decide what data quality dimension you are looking for Like Completeness , Accuracy, Validity, Uniqueness Then follow do the quality rules to do that Like For Completeness -- Check Null and Empty column For Uniqueness - Check de-Dedup Hope it helps On Wed, Feb 1, 2023 at 3:26 AM Mazen Saie mazen-saie@users.sourceforge.net wrote: Hope you are doing well, Im in process of implementing a data quality project, and one of the current phase's deliverable is data quality dimensions, the...
-
Mazen Saie posted a comment on discussion Help
Hope you are doing well, Im in process of implementing a data quality project, and one of the current phase's deliverable is data quality dimensions, the problem is the only discovery made is data profiling which is a high level analysis, Any suggestion on how to use this profiling to calculate the dimensions, no examples needed just direction or suggestions Thank you
-
Here is link of video to explain some features of osDQ. Also is attached documents to for UDF and Scheduling https://www.youtube.com/watch?v=_Allh9Uraoo&t=59s https://www.youtube.com/watch?v=_lvcaj8rPb8&t=12s https://www.youtube.com/watch?v=MQMoZeUjJRw&t=56s https://www.youtube.com/watch?v=q4PvY3ty880&t=2s
-
Open Source Data Quality and Profiling released /dataquality/Version6.3.3/ProfileV6.3.3.zip
-
-
-
-
8 GB looks to less for bigger file because it needs other components also like SWING and AWT and local variable. You need bigger RAM. There is core component of osDQ at https://github.com/arrahtech/osdq-core Though you will have to write your owb driver. regards. ++++++ I bumped up the RAM to 8 GB (Xmx, Xms) (see below). I have a file that's around 5.6 GB. On loading the file, I get the following error. Any ideas on how to handle files upto 8 GB? Is it possible to invoke the profile at the command...
-
Need to keep the password safe
-
closing it
-
in release 6.3.1 password is masked with *
-
-
-
-
-
https://stackoverflow.com/questions/1565388/increase-heap-size-in-java make changes in runprofiler.[sh][bat]
-
Julia Przibylla posted a comment on discussion Help
i realy have no expierence with increase heap space in java. where/how do I set it up?
-
loading is slow because snowflake is cloud database. You have to be co-located to have faster access. There is some URL which talks about above error https://support.snowflake.net/s/question/0D50Z00008TSnlZSAT/jdbc-fetching-query-result-failed-with-the-target-server-failed-to-respond https://discourse.metabase.com/t/metabase-failing-to-fetch-large-dataset/5789/2 Seems like a large data so you need to increase heap space in "C:\Users\przibylla\Desktop\ProfilerV6.2.9\ProfilerV6.2.9>java -Xmx4096M -Xms4096M...
-
Hello again, can s.o. help me to figure out the problem BASH: C:\Users\przibylla\Desktop\ProfilerV6.2.9\ProfilerV6.2.9>java -Xmx4096M -Xms4096M -classpath osdq-desktop-6.2.9-SNAPSHOT.jar;lib\rowset.jar;lib\jdbcjar*;lib\hivejar*;lib\hive2jar*;lib*;lib\saxon* org.arrah.gui.swing.Profiler Mõr 27, 2020 12:20:39 PM net.snowflake.client.jdbc.internal.apache.http.impl.execchain.RetryExec execute INFORMATION: I/O exception (net.snowflake.client.jdbc.internal.apache.http.NoHttpResponseException) caught when...
-
Thank you very much, it does work, but it seems like it takes a long time to load the database ...
-
if you face any issue share the screen shots
-
1.) download the jar (snowflake-jdbc-3.9.2.jar )file from https://repo1.maven.org/maven2/net/snowflake/snowflake-jdbc/3.9.2/ 2.) copy this file to ProfilerV6.2.9/lib/jdbcjar 3.) With Otherthing DB Driver put the driver name net.snowflake.client.jdbc.SnowflakeDriver 4.) Put the connect string as per snowflake URL and click on connect
-
Connection Failed. Driver Class Not Found. Look into System DSN if using jdbc:odbc bridge Make sure Driver class is in classpath if using native System Message:net.snowflake.client.jdbc.SnowflakeDriver
-
"...along with other drivers"?
-
I will try it, nut do i need to specify path?
-
https://docs.snowflake.com/en/user-guide/jdbc-configure.html 1.) Download and save the jdbc driver in under jdbc directory along with other drivers 2.) In connection dialog UI choose others ( JDBC) 3.) In Driver name "net.snowflake.client.jdbc.SnowflakeDriver" 4.) Put the connect string and test connection it should work
-
Connection Failed. Driver Value Not Found - Check DB Driver field
-
what error message you are getting on command line and in connection dialog ? On Thu, Mar 26, 2020 at 5:23 AM Julia Przibylla juliaprz@users.sourceforge.net wrote: Hello, Can someone help me to set the configuration correctly?! Connection to Snowflake (JDBC Driver) is failing https://sourceforge.net/p/dataquality/discussion/602692/thread/9103ffc18a/?limit=25#c4d8 Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/dataquality/discussion/602692/ To unsubscribe from...
-
Hello, Can someone help me to set the configuration correctly?!
-
No. You can create any number of rules. There is no limit on 5 rules.
-
Dipali Nanware posted a comment on discussion Help
I have created building rules, but after 5 rules i cant create any more ... so is there any limitation to build an buisness rules.
-
thanks for your comments. 1.) It shows DSN name for now. Future releases it will show user name also. 2.) In future release, both option will be given - show password or * 3. ) How to uncomment connection parameter in configFile.txt original /* Postgres Native JConnector DRIVER Database_Type="POSTGRES" Database_Driver="org.postgresql.Driver" Database_Protocol="jdbc:postgresql" Database_DSN="//???/????" Database_User="????" Database_Passwd="????" Database_Catalog="\"?????\"" Database_SchemaPattern="????"...
-
Bart Jonk modified a comment on ticket #80
Hi Arrah, four observations & a new question: When starting from configfile.txt 1) when starting the program from configfile.txt, osDQ will prompt for a password, but there's no hint for which user: [image: image.png] (I tried a different dsn that possibly solves that, see 3 below) 2) When copying the password from my password manager, the password is printed in plaintext (instead of being masked ****): [image: afbeelding.png] 3) with following config nothing happens (no proper connection) (Note:...
-
Hi Arrah, four observations & a new question: When starting from configfile.txt 1) when starting the program from configfile.txt, osDQ will prompt for a password, but there's no hint for which user: [image: image.png] (I tried a different dsn that possibly solves that, see 3 below) 2) When copying the password from my password manager, the password is printed in plaintext (instead of being masked ****): [image: afbeelding.png] 3) with following config nothing happens (no proper connection) (Note:...
-
6.2.9 of osDQ releasd
-
Hi , If you want to remove added connection open the DBConnections.xml file in XML editor. Select the node 'connection' and delete/remove - it will remove the node and it all elements. PS * while adding connection you can choose to keep password or give at runtime.
-
Need to keep the password safe
-
This feature has been added in 6.2.9 . If password is empty, null or "****" it would prompt for password from confileFile and DBConnections.xml. Please confirm.
-
-
-
Thanks Paula for using osDQ Hope this link helps you https://sourceforge.net/p/dataquality/discussion/602692/thread/2677c28c/ In brief, osDQ make catalog.table call. Regards, Vivek On Mon, Oct 28, 2019 at 3:35 PM Paula DiTallo plditallo@users.sourceforge.net wrote: Gurus-- This may be a configuration issue, but here is what I see in the attached image when I have tables that are under a schema other than dbo. Is there a setting somewhere to include these tables? SQL Server schema issue on reporting...
-
Paula DiTallo posted a comment on discussion Help
Gurus-- This may be a configuration issue, but here is what I see in the attached image when I have tables that are under a schema other than dbo. Is there a setting somewhere to include these tables?
-
thanks. Typically community release takes 4-5 months. regards. On Thu, Oct 24, 2019 at 10:29 AM Bart Jonk bartjonk@users.sourceforge.net wrote: Hi Arrah, That's really nice of you. I added request 80. How often do you release? cheers, Bart. On Thu, 24 Oct 2019 at 15:01, arrah arrah@users.sourceforge.net wrote: Thanks Bart of using osDQ. Please file a feature request for this. In coming releases it will be taken care of. However, if you needed urgently, we can custom build for you out of turn. regards,...
-
Hi Arrah, That's really nice of you. I added request 80. How often do you release? cheers, Bart. On Thu, 24 Oct 2019 at 15:01, arrah arrah@users.sourceforge.net wrote: Thanks Bart of using osDQ. Please file a feature request for this. In coming releases it will be taken care of. However, if you needed urgently, we can custom build for you out of turn. regards, On Thu, Oct 24, 2019 at 4:26 AM Bart Jonk bartjonk@users.sourceforge.net wrote: Hi, I edited the DBConnections.xml to add two Postgress connections....
-
Need to keep the password safe
-
Thanks Bart of using osDQ. Please file a feature request for this. In coming releases it will be taken care of. However, if you needed urgently, we can custom build for you out of turn. regards, On Thu, Oct 24, 2019 at 4:26 AM Bart Jonk bartjonk@users.sourceforge.net wrote: Hi, I edited the DBConnections.xml to add two Postgress connections. This kind of works, since I can now select them in the Connection Dialog Window and connections will work eefectively. However, the selected values do not appear...
-
Hi, I edited the DBConnections.xml to add two Postgress connections. This kind of works, since I can now select them in the Connection Dialog Window and connections will work eefectively. However, the selected values do not appear in the fields. I would have hoped they did, since it would allow me to leave the passowrd te be filled in in the window, in stead of leaving it in the config file. Can this be fixed? Or is there another way to keep my password safe? cheers, Bart
-
XLSX and XML saving and reading error
-
There is old saxon jar file with conflicts with XLSX open format. If you are facing XLSX saving problem, remove saxon directory. However remvoing saxon directory will create problem in saving as XML format. We are looking to update saxon jar file
-
Mpume modified a comment on discussion Help
This is a great piece of software! I know when many other Java products you can specify the look and feel. Personally I am not about looks but when you want to show a colleague of convince someone to take a look at the product the look and feel can have an impact on take up of the product. My question is how do I specify look and feel to be Windows native for this product? I know in other products its a matter of specifying the look and feel to use com.sun.java.swing.plaf.windows.WindowsLookAndFeel...
-
In case anybody else runs into this. I added the schema to the DB Catalog instead of the DB Schema Pattern. This resolved the issue I was having. On Sep 12, 2019, at 12:30 PM, Support support@arrahtech.com wrote: https://sourceforge.net/p/dataquality/discussion/602692/thread/5b1857a3/?limit=25#dec3 https://sourceforge.net/p/dataquality/discussion/602692/thread/bea09f66/ Hope it works for you. Hi, I’ve got the profiler tool and was attempting to use it on a postgres system. The postgres is in AWS....
-
Thanks Mpume for evaluating osDQ. This software will also have same process for specifying window look and feel though I have not tested it. If someone is interested, we can make the changes and test it,
-
This is a great piece of software! I know when many other Java products you can specify the look and feel. Personally I am not about looks but when you want to show a colleague of convince someone to take a look at the product the look and feel can have an impact on take up of the product. My question is how do I specify look and feel to be Windows native for this product? I know in other products its a matter of specifying the look and feel to use com.sun.java.swing.plaf.windows.WindowsLookAndF...
-
From: Support support@arrahtech.com Sent: Tuesday, July 16, 2019 10:52 AM To: **** Subject: Re: MySql Connection Please download jconnector for SQL 8 https://dev.mysql.com/downloads/connector/j/ put that jar file into lib/jdbcjar and remove old mysql-connector-java-5.1.26-bin.jar ( this is for version 5 ) . I think then it will work. Please let me know ---- On Tue, 16 Jul 2019 20:04:26 +0530 ** wrote ---- I do note in your example you didn't include the port number. I tried both ways and got nothing....
-
-
Hi Bhavana, Please visit this site - http://www.arrahtec.com/ look in bottom right - product live demo However, we also do paid training. regards, Vivek On Wed, May 22, 2019 at 5:52 PM K.Naga Bhavana bhavana@users.sourceforge.net wrote: Hii, I am new to OSDQ. I need complete video tutorial of osdq can you please share me. Thanks Sent from sourceforge.net because you indicated interest in https://sourceforge.net/p/dataquality/wiki/Home/ To unsubscribe from further messages, please visit https://sourceforge.net/auth/subscriptions/...
-
K.Naga Bhavana posted a comment on a wiki page
Hii, I am new to OSDQ. I need complete video tutorial of osdq can you please share me. Thanks
-
Hi Andy, You can look at code at : https://github.com/arrahtech/osdq-core On Fri, May 17, 2019 at 6:18 PM Andy mrspaceman@users.sourceforge.net wrote: I am interested in looking through the code, but I can't find any commits newer than 2015. There is a relase this year, so I must have missed something. can anybody suggest what I am doing wrong ? Code Commits https://sourceforge.net/p/dataquality/discussion/602692/thread/b80d967c7f/?limit=25#3477 Sent from sourceforge.net because you indicated interest...
-
Andy posted a comment on discussion Help
I am interested in looking through the code, but I can't find any commits newer than 2015. There is a relase this year, so I must have missed something. can anybody suggest what I am doing wrong ?
-
thanks for using osDQ. In information panel it show only top 20 patterns and combined others into "others". However you can go to DataQuality --> Duplicate and select the table/column you want to see pattern. It will show all. Hope it helps.
-
Dennes G Skaros posted a comment on discussion Help
I click on a field in a Database and the Information window opens up and shows me the Pattern chard. But it only shows me a limited number of Pattern information, the morjority of the Pattern Information is grouped together as "Others". How can I see what the "Others" breakdwon is?
-
-
-
Thanks for the prompt and helpful response. It worked using your suggested method. I also managed to get the configFile.txt to work by escaping the slashes. I am running on Windows 7. Config below worked (the \/ is back and forward slash not letter V): Database_Type="Sqlite" Database_Driver="org.sqlite.JDBC" Database_Protocol="jdbc:sqlite" Database_DSN="C:\/path\/to\/my\/database\/databasefile.db" Database_User="" Database_Passwd="" Database_Catalog="" Database_SchemaPattern="" Database_TablePattern=""...
-
Thanks for the prompt and helpful response. It worked using your suggested method. I also managed to get the configFile.txt to work by escaping the slashes. I am running on Windows 7. Config below worked : Database_Type="Sqlite" Database_Driver="org.sqlite.JDBC" Database_Protocol="jdbc:sqlite" Database_DSN="C:\/path\/to\/my\/database\/databasefile.db" Database_User="" Database_Passwd="" Database_Catalog="" Database_SchemaPattern="" Database_TablePattern="" Database_ColumnPattern="" Database_Tabl...
-
Hi, Thanks for using osDQ. You should first run SQLLite form connection Dialog which is attached. If it is successful, then use the same parameters and Database_Type="OTHERS" as sqlLite is not pre-configured. Make sure the jdbc jar file for SQLLite is under lib/jdbc Make us know if you still face any problem. On Fri, Jan 11, 2019 at 3:20 PM Mpume msimanga@users.sourceforge.net wrote: I am trying to use Sqlite. I have placed the Sqlite JDBC driver in the classpath and in the configFile.txt I have...
-
I am trying to use Sqlite. I have placed the Sqlite JDBC driver in the classpath and in the configFile.txt I have the following entry: Database_Type="Sqlite" Database_Driver="org.sqlite.JDBC" Database_Protocol="jdbc:sqlite" Database_DSN="C:\path\to\my\database\databasefile.db" Database_User="" Database_Passwd="" Database_Catalog="" Database_SchemaPattern="" Database_TablePattern="" Database_ColumnPattern="" Database_TableType="" The console window does not bring back any of the meta data for the...
-
org/arrah/framework/xml/XmlReader this class should be in osdq-core-6.2.6.jar file which should be in lib directory. If you do runprofiler.bat it will set classpath where it will pick up. Can you please share how you are running profiler and screen shot from where you are getting error regards,
-
Harris posted a comment on discussion Help
I am trying to run the Profiler from Windows and get below err.. Appreciate any help Exception in thread "main" java.lang.NoClassDefFoundError: org/arrah/framework/xml/XmlReader at org.arrah.gui.swing.TestConnectionDialog.createGUI(TestConnectionDialog.java:113) at org.arrah.gui.swing.Profiler.main(Profiler.java:877) Caused by: java.lang.ClassNotFoundException: org.arrah.framework.xml.XmlReader at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source)...
-
Hi Naveen, I am not able to see attachment. If may happen if some of the columns are binary type or stream type. Please share more information.
-
Hi Naveen, thanks for using osDQ. Business Rules are saved ( if scheduled) as shcedudedoutput.csv ( if it is not a qualified csv then change name to txt and load into osDQ. You can load using Tools --> Open File --> Open CSV or TXT format. Once the file is loaded in Option -> Load To DB will take information about which coulmn you want want to update ( assuming you have edit persmission) Some databases do not allow primary key to to be changed Let me know if you face any issues
-
P.Naveen Kumar posted a comment on discussion Help
I have created a business rule and executed it. On Execution window the data is profiling properly but if we close the execute business rule window the changes are not reflecting in the table data. I have removed N from the word 'NAVEEN' its is executing my business rule but how do you save this updated data through osdq tool.And it should be reflected in my oracle table aslo.
-
When i am loading a table from oracle database the table is getting loaded but the data is not getting loaded. I have attached the screenshot for reference.
-
Null replacement using regression
-
closed. It is working
-
In "Business Rule" section first step is 'add DB'. It is open up a connection dialog ( like the first time you run profiler) - make sure to give a name to connection, add connection parameters, validate then add connection. It is give a message "connection saved" You can use this connection to create Business Rules. Select Connection, it will show tables, columns of that connection. Make sure to give name to business rules , create a sql based business rule ( at present only SQL based business rules...
-
Hi Dennes, In "Business Rule" section first step is 'add DB'. It is open up a connection dialog ( like the first time you run profiler) - make sure to give a name to connection, add connection parameters, validate then add connection. It is give a message "connection saved" You can use this connection to create Business Rules. Select Connection, it will show tables, columns of that connection. Make sure to give name to business rules , create a sql based business rule ( at present only SQL based...
-
---- On Thu, 27 Sep 2018 17:36:39 +0530 Support support@arrahtech.com wrote ---- Thanks for spending time with tool. Yes, DB2 treats catalogue as database itself. Good that You are able to connect and analyze. 1.) There are multiple ways you can delete regular expression - you can go to Data Quality --> Standardization Regex ( create a key value pair of file if not already there - key can be regex and values can be replace value ) -choose the column and file - it will do. You can save it back to...
-
Hai Arrah, Really impressed by the tool and i am starting using it. Can you please provide me any videos or documents on how to use the tool and function of each and every tab in that tool.I have started exploring it but finding it little difficult to browse each and every option. Thanks and Regards, <reply> thanks for using osDQ tool. You can see some video at http://www.arrahtec.com/ -- bottom right side Product live demo. Also in download there is user guide though it is little old but it contains...
-
-
-
-
-
Thanks Shazzle for using osDQ. 1.) Masking -- Tools — >Open file —> Column —> Mask Column ) PII functionality looks all the available columns and try to see if the name and data matches the PII information. In resource/piisearch.txt has mapping to PII columns i.e CreditCard="Creditcard,CC,Credit_card" DoB="DateOfBirth,DoB,BirthDay" Email="Electronic Mail,MailID,email" PhoneNumber="Cell,Cell Number,Landline,phone" SocialSecurity="SSN,Social,SocialSecurity" IP="ipv4,ipv6,inet,hostname,ipaddress,host,inetaddress"...
-
shazzle posted a comment on discussion Help
How can I make use of the masking capabilities claimed on the feature list? Also what does the Tool --> Personally Identifiable Information item do functionally? Thank you very much - this is a very interesting tool set!
-
Please put the limiting schema or catalog in catalog field also. Internally it build cat.table and filter metadata in catalog On 19 Mar 2018 9:03 pm, "Joseph Harvat" jharvat@users.sourceforge.net wrote: Hi - I specified a target schema but when I open the tool, it seems as though it is trying to load metadata for every table in the entire PostgreSQL environment. There are thousands of tables and it's taking forever to load. What do I need to do to limit the number of target tables to only those in...
-
Joseph Harvat modified a comment on discussion Help
-
-
-
Hi - I specified a target schema but when I open the tool, it seems as though it is trying to load metadata for every table in the entire PostgreSQL environment. There are thousands of tables and it's taking forever to load. What do I need to do to limit the number of target tables to only those in the schema I am interested in. Thanks!
-
Hi Asad, Thanks for using osDQ. Which Operating System you are using ? just go to command line ( console) and type > java -version java 8 should be in your path. In case java is not here you can download java 8 Here is a link if java is not your path http://www.tomsguide.com/faq/id-1761312/fix-create-java-virtual-machine-issue.html On Thu, Mar 15, 2018 at 2:14 PM, Asad Ahmed Tarin dirtyminer@users.sourceforge.net wrote: Hi i am trying to isntall profilerv6.2.4 and get the error on command line "...
-
Asad Ahmed Tarin modified a comment on discussion Help
Hi i am trying to install data profilerv6.2.4 and get the error on command line " Could not create the Java Virtual Machine". Could anyone help on this thanks.
-
Hi i am trying to isntall profilerv6.2.4 and get the error on command line " Could not create the Java Virtual Machine". Could anyone help on this thanks.
-
Functions for System and User
-
added
-
MinVal MaxVal Mean added
-
More info in Data Summary Metadata
-
More info in Data Summary Metadata
-
Functions for System and User
-
Null replacement using regression