I do believe... it's finally complete! I think I changed the majority of the code over to what was less error-prone code. Then consolidated all functions into the working code. wow what a cleanup. Only thing left to do is test it with random data. and Garbage data etc. There's probably a few variables I could clean up. I used the "less memory more cpu" approach with the data so you can stuff as much data into the ram as possible. The cpu intensive tasks are column related, as you can imagine, because the data is stored as rows. That means every column action has to DSV import/export. I cut slow corners as much as possible, but if something turns out to be extra slow, I can take another look at it.
so to recap:
UI cleaned up to wxphoenix standards, program code all cleaned up to work properly
"cleaned" really means rewritten. Finally a program worthy of use.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ohhhhh no! I forgot about keeping the selection of rows/cols. It forgets them because I had forgotten they were used again if you open the find grid it should be able to select the old rows/cols that are visible, you had selected before you clicked on close. When I deleted the "filter rows/cols" menu I removed that as well. Better go add it. Also I will make a menu selection to erase all matches and selections.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I added a selections menu for cols and rows. Also... I noticed lots of invalid characters from when I was editing in windows. So I got rid of them. Always wondered why it would freeze up microsofts code editor. I might be able to consolidate the code for generating the rows/cols view/editor. Since I created the method to view the data based on which ones had been selected, it's very close to the code that views them from a built up range. Ohh and I also noticed the find data searches every row of data not just the visible ones. Probably wasting cpu cycles. I will add the code to clear selections on the grid and unmark data cells as well. The features I wanted to add are going in nicely. you can execute the script with './CSVOMATIC.py mary.csv' on linux systems because now I can chmod u+x on it with all the invalid windows characters removed.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
there's a few rogue spaces in there still somewhere according to vscode. Yikes! The important part is, I've completed most of the extreme wishlist. row/col transform is the remainder, and I've been hatching an idea. There are a few edge cases left to test in there. Overall though the latest source is really performing well. Haven't generated an invalid CSV file in a long time!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
now there has been lots of testing and bug fixing. Most of the bugs resided in the row adding and column deleting code. There was a distinct lack of updating on the row/col indexes which threw the entire thing into disarray. So, I wrote 2 methods which handle adding a sequence to another sequence and deleting a sequence from another sequence. hah hah that sounds funny, but the index from the row you are seeing that connects to the real row in the data has to be updated if there are changes to the underlying data, of course. And the index is an arbitrary number. But has to be in numerical order. etc. Some other bugs remain, like refreshing the column labels if you add a new blank row for the headers. Oh yah and the activation of the deselect menu in the find grid. I'll be getting to all these bugs now that the code is featured up. I'll post when I think they are all debugged.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
added the menu deactivation. Now you can see if there are row/col selections waiting or not. Sometimes rows are "selected" but not shown as selected in the grid. I suppose I could work on making sure the selected rows are always selected when the grid is displayed. Probably a good idea. Then I can get rid of some of the menu options for find. I debugged the code pretty good. I'm going to post another release of it.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
debugging is going great! I have detected and eliminated quite a few related to inserting the wrong row/col in the grid. It's just some index references I forgot to make. It works great... unless you have col 0/row 0 missing/showing. The indexes point to the right grid row in relation to the data row and vice versa. Some places I forgot to use the index when changing the grid. I'll hunt them all down eventually. I also got the custom delimiter working. Replaced all hard-coded \t... I wonder why I had pasted them in there to begin with. Everything worked great as long as you only ever used \t as delimiter. I am considering breaking the script into a couple parts, for easier scrolling through for editing purposes. I'll have to go with zipfile releases then. I'll consider it. There is also a bunch of redundant code for viewing rows/cols. I wrote better functions, just haven't used them there.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ahh it is quite amazing how well it works. I am even loading random data, to see what happens. It'll probably take me way longer to find new bugs! Hopefully after a couple more days I will have eliminated them all.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am focusing on linting right now. Already broke the script into two, and might divide the code into another part. the main parts are the dialogs, grid and main window. I know 1600 lines of code is too much for one file.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
divided the code into another file containing the find grid. now the dialogs and frames are seperate! Also I am adding docstrings to the functions. Phew I still had the ancient oldschool comments above the methods.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
great news! The linting is complete, and it helped out in several spots. I completed all docstrings as well. I know the linting loves spaces and hates tabs heh heh I suppose that's why spaces are taking over in the indent world. But it couldn't agree on the spacing needed for continued lines. Anyways, I hope it's more legible now. In a day or two, I will make the executable installer and release it. Yes, it works properly! I've been trying to get some crashing going on but it seems solid now. So it works and it's hard to crash, good times!
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
well, when I created a tar bz of the compiled program CSVOMATIC it's 156 megabytes. I know... that includes every library and the interpreter.... but, wow. That's nuts. and it's all wx core... well at least 99 megabytes is. I guess I should spend some more time optimizing the install. I'll go and try it under windows 64 bit as well.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ok I made a 64 bit windows build that's about 8 megabytes. Enjoy! It contains the fix I needed in wxpython_phoenix. Linux users... well I think I might need to strip debugging symbols, I know I can get it down under 150 megabytes! I used inno setup for the installer.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
cruuuuud under windows the find dialog is....ahhh...kind of messed up. I didn't use a panel. The dialog is based on old-school wx examples. I will upgrade them to more modern code! The dialog works, you just can't see what you typed in for some reason.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
just a slight change there to the dialog. I added staticbox and a panel, then updated the text control so you can hit enter and it does the search. EndModal was the lesson I really learned, because I was using ancient code with SetReturnCode... It never actually really sets a return code that you can control under certain situations. That combined with the text enter calling self.Close.... oh my! It would NEVER work properly. I figured that out when I recycled the dialog on a different project. Had to hit the documentation hard.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
completion status
I do believe... it's finally complete! I think I changed the majority of the code over to what was less error-prone code. Then consolidated all functions into the working code. wow what a cleanup. Only thing left to do is test it with random data. and Garbage data etc. There's probably a few variables I could clean up. I used the "less memory more cpu" approach with the data so you can stuff as much data into the ram as possible. The cpu intensive tasks are column related, as you can imagine, because the data is stored as rows. That means every column action has to DSV import/export. I cut slow corners as much as possible, but if something turns out to be extra slow, I can take another look at it.
so to recap:
UI cleaned up to wxphoenix standards, program code all cleaned up to work properly
"cleaned" really means rewritten. Finally a program worthy of use.
ohhhhh no! I forgot about keeping the selection of rows/cols. It forgets them because I had forgotten they were used again if you open the find grid it should be able to select the old rows/cols that are visible, you had selected before you clicked on close. When I deleted the "filter rows/cols" menu I removed that as well. Better go add it. Also I will make a menu selection to erase all matches and selections.
I added a selections menu for cols and rows. Also... I noticed lots of invalid characters from when I was editing in windows. So I got rid of them. Always wondered why it would freeze up microsofts code editor. I might be able to consolidate the code for generating the rows/cols view/editor. Since I created the method to view the data based on which ones had been selected, it's very close to the code that views them from a built up range. Ohh and I also noticed the find data searches every row of data not just the visible ones. Probably wasting cpu cycles. I will add the code to clear selections on the grid and unmark data cells as well. The features I wanted to add are going in nicely. you can execute the script with './CSVOMATIC.py mary.csv' on linux systems because now I can chmod u+x on it with all the invalid windows characters removed.
there's a few rogue spaces in there still somewhere according to vscode. Yikes! The important part is, I've completed most of the extreme wishlist. row/col transform is the remainder, and I've been hatching an idea. There are a few edge cases left to test in there. Overall though the latest source is really performing well. Haven't generated an invalid CSV file in a long time!
just testing now and fixing some minor bugs. If you paste in a row longer than the other data, that works. Haven't tested longer columns yet :)
now there has been lots of testing and bug fixing. Most of the bugs resided in the row adding and column deleting code. There was a distinct lack of updating on the row/col indexes which threw the entire thing into disarray. So, I wrote 2 methods which handle adding a sequence to another sequence and deleting a sequence from another sequence. hah hah that sounds funny, but the index from the row you are seeing that connects to the real row in the data has to be updated if there are changes to the underlying data, of course. And the index is an arbitrary number. But has to be in numerical order. etc. Some other bugs remain, like refreshing the column labels if you add a new blank row for the headers. Oh yah and the activation of the deselect menu in the find grid. I'll be getting to all these bugs now that the code is featured up. I'll post when I think they are all debugged.
added the menu deactivation. Now you can see if there are row/col selections waiting or not. Sometimes rows are "selected" but not shown as selected in the grid. I suppose I could work on making sure the selected rows are always selected when the grid is displayed. Probably a good idea. Then I can get rid of some of the menu options for find. I debugged the code pretty good. I'm going to post another release of it.
debugging is going great! I have detected and eliminated quite a few related to inserting the wrong row/col in the grid. It's just some index references I forgot to make. It works great... unless you have col 0/row 0 missing/showing. The indexes point to the right grid row in relation to the data row and vice versa. Some places I forgot to use the index when changing the grid. I'll hunt them all down eventually. I also got the custom delimiter working. Replaced all hard-coded \t... I wonder why I had pasted them in there to begin with. Everything worked great as long as you only ever used \t as delimiter. I am considering breaking the script into a couple parts, for easier scrolling through for editing purposes. I'll have to go with zipfile releases then. I'll consider it. There is also a bunch of redundant code for viewing rows/cols. I wrote better functions, just haven't used them there.
just mopping up a few minor bugs. They are getting hard to find!
ahh it is quite amazing how well it works. I am even loading random data, to see what happens. It'll probably take me way longer to find new bugs! Hopefully after a couple more days I will have eliminated them all.
I am focusing on linting right now. Already broke the script into two, and might divide the code into another part. the main parts are the dialogs, grid and main window. I know 1600 lines of code is too much for one file.
divided the code into another file containing the find grid. now the dialogs and frames are seperate! Also I am adding docstrings to the functions. Phew I still had the ancient oldschool comments above the methods.
when I am done linting and debugging, I will make an exe and windows installer. then... I'll release it on sourceforge...joy!
great news! The linting is complete, and it helped out in several spots. I completed all docstrings as well. I know the linting loves spaces and hates tabs heh heh I suppose that's why spaces are taking over in the indent world. But it couldn't agree on the spacing needed for continued lines. Anyways, I hope it's more legible now. In a day or two, I will make the executable installer and release it. Yes, it works properly! I've been trying to get some crashing going on but it seems solid now. So it works and it's hard to crash, good times!
well, when I created a tar bz of the compiled program CSVOMATIC it's 156 megabytes. I know... that includes every library and the interpreter.... but, wow. That's nuts. and it's all wx core... well at least 99 megabytes is. I guess I should spend some more time optimizing the install. I'll go and try it under windows 64 bit as well.
ok I made a 64 bit windows build that's about 8 megabytes. Enjoy! It contains the fix I needed in wxpython_phoenix. Linux users... well I think I might need to strip debugging symbols, I know I can get it down under 150 megabytes! I used inno setup for the installer.
cruuuuud under windows the find dialog is....ahhh...kind of messed up. I didn't use a panel. The dialog is based on old-school wx examples. I will upgrade them to more modern code! The dialog works, you just can't see what you typed in for some reason.
just a slight change there to the dialog. I added staticbox and a panel, then updated the text control so you can hit enter and it does the search. EndModal was the lesson I really learned, because I was using ancient code with SetReturnCode... It never actually really sets a return code that you can control under certain situations. That combined with the text enter calling self.Close.... oh my! It would NEVER work properly. I figured that out when I recycled the dialog on a different project. Had to hit the documentation hard.