Activity for Mark Michalek
-
Mark Michalek posted a comment on discussion Help
Jeff, that is an excellent find. Sorry you had to struggle with that. I opened an issue in the Symmetric issue tracker so this can get fixed. https://www.symmetricds.org/issues/view.php?id=4211 Thanks, Mark
-
Hoarau, SymmetricDS will write the sym tables to the default database for your JDBC URL/user combination. You might be able to specify the public schema on your URL. Or you could create a symmetric database user whose database search path / default database is public. Mark
-
Hello. I like your idea #1. Can you describe in more detail what a "reinit" is for you? Thanks, Mark
-
Steven - did you try the sym_trigger SOURCE_SCHEMA_NAME maybe? If your triggers are appearing on the wrong table, the problem is most likely in the sym_trigger configuration which is what controls the physical trigger creation. Mark
-
Hi Steven, here's basically how this works: 1) You configure your engine.properties file with a database URL and database user. That DB connection will have a default database that it connects to. 2) The sym tables will be created on the default database from the properties file. 3) When it comes to your triggers, if you leave off the catalog/schema, then it the triggers would attempt to create on the default database from the properties file. 4) If your application tables are in a different database...
-
Hello userx, Can you describe the problem you are trying to solve a little more? It seems like you are trying to get a message back to central, but what is the nature of that message? My guess is it woould be better to implement a seperate status table and set that up for sync as well. The setSqlMessage is really reserved for SQL errors generated during Symmetric's internal processing, so I would not recommend modying that field. Does that make sense? Thanks, Mark
-
Stian, good job digging into it and thank you for posting your solution.
-
Nicolas, I think you are on the right track. Good work and thanks for posting your solution.
-
Nicolas, I think you are on teh right track. Good work and thanks for posting your solution.
-
plgmail - maybe it would help to think through how the sync works and you could see what point the process breaks down from what you expect. When a corp side Productos row changes, the Productos_corp trigger should fire and capture a row into sym_data representing those changes. When the routing process runs, the corp_2_one_client router should create a row in sym_outgoing_batch where the node_id is the node you expect to receive the change. If you see the node_id as "-1", that means "unrouted" and...
-
plgmail, you can initiate an initial load from the client with this SQL: update sym_node_security set initial_load_enabled = 1, initial_load_time = NULL where node_id = ?
-
David, maybe this article would help: https://medium.com/data-weekly/character-sets-and-symmetricds-c8f4c3ac6d4
-
SebiH, when it ranomdly shuts down, did you see the visual indication still that it was a foreground service? Maybe you could try adding some logging for the Android service lifecycle events. E.g. override fun onTaskRemoved(rootIntent: Intent?) { super.onTaskRemoved(rootIntent) Log.d(TAG, "onTaskRemoved") } override fun onDestroy() { super.onDestroy() Log.d(TAG, "onDestroy") } override fun onTrimMemory(level: Int) { super.onTrimMemory(level) Log.d(TAG, "onTrimMemory") }
-
Garth, are you hitting the same problem with the StagingManager as mentioned in the other post? I think you could set the staging manager from code within Android Studio without having to recompile the libraries. Mark
-
SebiH, thanks for reaching out to the forums. My understanding here is that the foreground service would be less likely to be killed, but will display the notification all the time to the user. From the docs: "if the service is declared to run in the foreground, it's rarely killed." (https://developer.android.com/guide/components/services). Maybe you could try to determine if services is killed by Andriod for resources or if there is some sort of error occurring. Logically I think Symmetric fits...
-
Neo, the data is in base64 in the batch file which is also a standard textual representation of binary data. You could select your RAW data as base 64 in order to compare. Symmetric installs a function called sym_blob2clob(column) that you could try. Otherwise, check https://stackoverflow.com/questions/3804279/base64-encoding-and-decoding-in-oracle.
-
Sankar, I assume that is happening when Symmetric tries to decrypt the database password. Try just supplying your database password in plain text (remove the enc: prefix) and see if you can start up that way. If that works, you can look at re-enrypting your database password. See: http://www.jumpmind.com/downloads/symmetricds/doc/3.10/html/user-guide.html#_encrypted_passwords
-
Eric, a couple of ideas about the location of the sqlite file: 1) Could the current working directory be different when you run from command line and when run symmetric? 2) Are you using a different OS user when you run from the command line vs. run the software? Also, usually Symmetric should just create it's sym tables automatically when you startup the software. So using the symadmin create-sym-tables command is not strictly required. Mark
-
Red, You are correct, the latest versions of SymmetricDS reference java.nio.file.DirectoryStream. which while part of Java 7, is not available on android until API 26 (Android 8). See: https://developer.android.com/reference/java/nio/file/DirectoryStream It's possible you could patch this with a small code change but I am not sure if this would work. You could replace AndroidSymmetricEngine.createStagingManager() with a stubbed out IStagingManager. You would also want to set: stream.to.file.enabled=false...
-
Yes, for sure, SymmetricDS was originally designed for that scenario. The stores will PUSH up to the central, static IP, and they will PULL down from that central, static IP. Mark
-
Yes, for sure, SymmetricDS was originally designed for that scenario. The stores will PUSH up to the central, static IP, and they will PULL down from that central, static IP. Mark
-
Andy, what version are you on currently? I assume your 6 servers are in a cluster (since they are pointed at the same datasource). You would need to update those 6 at the same time since they will update the database schema for the newer version. You can update the clients later; the protocol is background compatible. I recommend updating to 3.9.18 which we have lots of users running on which are very stable. Also, for stablity, it's imperitive to make sure cluser.locked.enabled=true on all 6 of...
-
Konrad, correct, it's file.sync.enable=true. And you are correct you would nee the start.file.sync.tracker.job, and start.file.sync.push.job=true, start.file.sync.pull.job=true
-
Pavithra - it looks you need to allow (open) registration for your node on your registration server. See http://www.symmetricds.org/doc/3.9/html/user-guide.html#_registration You could also set auto.registration=true
-
Ali, Here's what I would recommend: 1) Don't use the router to change the table name (so leave sym_router.target_table_name NULL). 2) You're entry in sym_transform_table should do the work of changing the table name during EXTRACT. For your sym_transform_table to get used, the catalog/schema/table would need to match exactly what is coming out the sym_router. 3) With sym_transform_table column_policy = IMPLIED like you have, columns with the same name will be copied over. So you should only need...
-
Marco, That exception should be logged at DEBUG level, and can be ignored on Android. I recommend adjusting the logging to INFO level if you can. Mark
-
Linh, the same rules still basically apply. Truncates are not usually considered normal DML and are not captured by the triggers. Whatever code you are using to perform the truncate on the source database could insert a SQL "SEND" ('S') event into Symmetric's sym_data to perform the truncate on the other side. Mark
-
Patrick, if you know the remoteAddress upfront, you could also manually open registration by inserting the sym_node_security row yourself, with registration_enabled=1. Mark
-
Patrick, if you know the remoteAddress upfront, you could also manually open registration by inserting the sym_node_security row yourself, with registration_enabled=1. Mark
-
Ike, can you give a higher level summary of what you need to accomplish? Are you trying to concatenate multiple columns into a single PK column? Mark
-
Ike, That example shows how to generate a manual "reload" (R) event in SymmetricDS. That reload approach starts with inserting an 'R' row in sym_data. The RouterService will then route that 'R' entry and turn it into a batch (see sym_outgoing_batch). The batch will start out with the NE (New) status and get sent to the target and move to OK status. It should only get processed once. Mark
-
See my post here: https://sourceforge.net/p/symmetricds/discussion/739236/thread/615f0a9f0d/?limit=25
-
See my post here: https://sourceforge.net/p/symmetricds/discussion/739236/thread/615f0a9f0d/?limit=25
-
Ike - you could look up en engine like this. ISymmetricEngine engine = AbstractSymmetricEngine.findEngineByName(engineName);
-
Aldiney, thanks for your question. Are you getting an error message on SQL lite? Can you be more specific about how it fails? Does the row file to load? Or is the column blank, etc? Thanks, Mark
-
Hi jmdiazlr, the extracted data will also get put through your router (sym_router) - so it's possible the router is reducing the number of rows. Also it's possible to reduce the number of rows through a transform (sym_trasnform) so also check that.
-
Thanks for posting your solution!
-
Thank you for posting your solution!
-
I would guess this is related to the older server having "-Dfile.encoding=windows-1252" and the newer server having "-Dfile.encoding=utf-8"
-
Hi Bob, maybe I'm not fully understanding your use case. Here's how Symmetric is typcailly used: There is one engines.properities file per database. There is often also one SymmetricDS software install per database. Each SymmetricDS install should have a LAN connection to the local database. * In your case, I would guess would have 1000+ local Symmetric installs along side of your inventory management softare installation, all registratering to a central server. Given that, can you describe your...
-
Roberto, DB2 will invalidate triggers when procedures or table structures change. Normally SymmetircDS should also detect these structural changes and rebuild the triggers as needed. If there is a case where the trigger got invlalidated (or disabled on other platforms), Symmetric would not recognize this. We allow for disabled triggers so users can temporarily pause their data capture. If you recognize a pattern to this issue, please let us know what you observe and maybe we can help come up with...
-
Surendra, see Mark's post over here: https://sourceforge.net/p/symmetricds/discussion/739235/thread/eb8a8f13/ dbsql will send the SQL command directly to the database and does not use the mssql.bulk.load.unc.path parameter. The mssql.bulk.load.unc.path parameter is only used by the code in the MS SQL Bulk loader.
-
Radovan, all the requirements you mentioned sound like a very doable use case with SymmetricDS. For point #1, the "routers" in Symmetric control which nodes get which data. It is usually best if there is some sort of field within the data itself that indicates where it needs to go (it could be the node id itself or the store number in a retail example. That would be a column match router http://www.symmetricds.org/doc/3.9/html/user-guide.html#_routers). For a good intro, you could go through the...
-
Patrick, that message is incorrectly logged in 3.9.13. It will be fixed in 3.9.14 and you can ignore it in the meantime. Mark
-
John glad you solved this. You might also be interested in the create.table.without.foreign.keys parameter.
-
Peter, are these batches configured to push or pull to the target? Is the push/pull job running properly? Are there any errors in your logs on either side?
-
Ken, you're understanding is correct. Each SymmetricDS node connects to one database. When you cluster, you are standing up multiple SymmetricDS servers which act as one node and connect to the same database instance. Users who need more HA usually cluster at the database level using things like Oracle RAC.
-
Yeah, I think that select needs to just be the where clause. What is the advantage or ordering in your initial load select, if you are selecting all?
-
Nick, what version are you running with?
-
Punith, sometimes we this when there are mismatching FK constraints on the source and target database. In order to provide more input, we would need to see the sym_data, sym_data_event, and sym_outgoing_batch rows associated with the error, plus logs from the source and target syste.
-
Piotr, that is an interesting case. If you could provide an issue with use case DDL that would help a lot. You can submit an issue here: http://www.symmetricds.org/issues/view_all_bug_page.php And if you could provide a patch or pull request, even better! Mark
-
Bob, Symmetric was actually created for this kind of use case, but the typical use case is for a small Symmetric engine to be run on each of the 1000 computers. We would then expect just one engine running on your server which would service all of your 1000 clients.
-
Bangassou, each server should only have its own engine file. They communicate with each other only through the registration.url and sync.url properties.
-
Efraim, can you verify this happens with the latest (3.9.12) and provide a test case to recreate?
-
Aleksey, the batch data should be ordered by sym_data.data_id, so the create_time of sym_data is not considered for ordering with default settings. Can you help us by capturing a concrete case with details of your batch: 1) Export of sym_data, sym_data_event, and sym_outgoing_batch rows for the bad batch. 2) symmetric.log from the timeframe that the data was caprued and routed. 3) Batch CVS files from incoming and outgoing (see under <symds>/tmp/<nodeId>/outgoing on the source and <symds>/tmp/<nodeId>/incoming...
-
Martin, I think the main problem to focus on is this: "Protocol error. Session setup failed." That sounds like a Postgres connection problem. See https://stackoverflow.com/questions/31091471/org-postgresql-util-psqlexception-protocol-error-session-setup-failed There is reason for the 11th parameter is that is a conditional INSERT statement. Note that the values for the insert statemnt come from a SELECT statement, which also has a WHERE clause. The number of SQL Parameters should match up with the...
-
Re: the ServerEndpoint message - that will be completely muted in 3.9.10. Per http://www.symmetricds.org/issues/view.php?id=3630.
-
void2020 , SymmetricDS 2.7 is end of life. I strongly recommend going with the latest (currently SymmetricDS 3.9.9). There are lots of fixes and improvements that will make your setup much easier.
-
Hello Mahesh, It is possible to accomplish this with a single transform. I tried this scenario with the "item" table that comes with the SymmetricDS sample setup. I added a "REPLICATION_FLAG" table to the item table to test your scenario. Basically, it is possible in the transform to write conditional logic for the "UPDATE_ACTION". In the below transform, the script in the UPDATE_ACTION converts the UPDATE to a DELETE if the FLAG value does not equal 3. INSERT INTO "SYM_TRANSFORM_TABLE" ("TRANSFORM_ID",...
-
Hi. Each SymmetricDS node connects to one database. So in the example you gave, you might have a server database (with SymmetricDS installed) and a database on your Java client (also with SymmetricDS installed.) The two installations of SymmetricDS will communicate with each other over http and keep the data in sync. It is possible to have SymmetricDS create your schema (see: https://www.symmetricds.org/docs/how-to/sync-schema-ddl-changes)
-
mjay, That error appears to be a bug with the MyISAM storage engine (https://bugs.mysql.com/bug.php?id=4541). Please use the InnoDB storage engine for the Symmetric tables. See more here: http://www.symmetricds.org/doc/3.9/html/user-guide.html#_mysql Thanks, Mark
-
There are parameters which allow you to run SQL before and after your initial load which may be simpler for your configuration here. See: initial.load.before.sql and initial.load.after.sql
-
Roger, you can set an environment variable called SYM_HOME to specify the location where SymmetricDS can write the instance.uuid file.
-
Guillaume, check the sym_node table on both sides. There is a sync_url column that is being used here.
-
Susant, are you running these nodes in a cluster? (ie, more than one SymmetricDS server pointed at the same DB/schema?) If you do have a cluster, you'll need to run with this parmaeter on all participants in the cluster: cluster.lock.enabled=true. Otherwise, I recomend upgrading to the latest 3.9.x to fix this and some other issues you mentioned. See: https://sourceforge.net/projects/symmetricds/files/symmetricds/symmetricds-3.9/symmetric-server-3.9.8.zip/download Also, take a look over the change...
-
Francisco, as long as your client can access your server over HTTP (or HTTPS) you could use SymmetricDS to connect the 2 databases. You could make the Amazon server your registration server. You could accomplish bi-directional sync, with one way HTTP communication by having 2 node group links: client pushes to server server waits for pull from client Hope this helps. Mark
-
Francisco, as long as your client can access your server over HTTP (or HTTPS) you could use SymmetricDS to connect the 2 databases. You could make the Amazon server your registration server. You could accomplish bi-directional sync, with one way HTTP communication by having 2 node group links: client pushes to server server waits for pull from client Hope this helps. Mark
-
Ali, SymmetricDS will look for new data in sym_data using the next "gap" from sym_data_gap. First it will check what channels are available in that "gap", and then order those channels according to the processing order. Specifically "order by c.processing_order asc, c.channel_id". So the channel processing order should be roughly respected. Routing is a serial process though. Your best bet is to look at ways to speed up your routing process, which is going to depend on how quickly SymmetricDS can...
-
Jerry, this sounds like a limitation of dbexport as it looks like all query results get loaded into memory around this line org.jumpmind.db.sql.AbstractSqlTemplate.query(AbstractSqlTemplate.java:207). You have a few options: 1) Crank that wrapper.java.maxmemory as high as your system can handle (4g or 8g perhaps) 2) Figure out a query that would allow you to segment the data a bit. That we you could do multiple extract passes and combin the file after the fact. 3) Submit a pull request that handles...
-
Grzegorz, for large loads, if performance becomes a problem, we sometimes recommend users do the following procedure: 1) Disable the triggers during the load (ALTER TABLE <TABLE> DISABLE TRIGGERS ALL;) 2) Load the data 3) Issue a table reload in SymmetricDS. My preference is to use the 'R' event_type in sym_data to issue a reload of your table. See http://www.symmetricds.org/doc/3.9/html/user-guide.html#_send 4) Reenable your table triggers. (ALTER TABLE <TABLE> ENABLE TRIGGERS ALL;)
-
You assesment of the cause is accurate. You should be albe to specify columns to use for the primary key in two ways: 1) Common seperated list of sync_key_names on sym_trigger. 2) Specifying primary key columns in a sym_transform/sym_transform_column.
-
Tushar, I think if SymmetricDS is truly embedded in your application, you should be able to use embedded Derby with multiple connections. The key is that SymmetricDS and your application have to be running in the same JVM. From the Derby docs: "Embedded Derby supports multiple users in one JVM Derby supports multiple connections to a given database. An example of an embedded application that manages concurrent users is a Tomcat or Geronimo application server that embeds Derby. Any number of users...
-
Brian, I am glad you found it. There are some fixes related to that timeout in 3.9.4 if you are not already on that version. Not sure if this will help with your situation but might be worth a shot. http://www.symmetricds.org/issues/view.php?id=3406 Under normal circumstances, there is an HTTP keep-alive that is supposed to sent between the nodes to avoid that idle timeout, which is why it is set so low out of the box.
-
Guys, feel free to raise an issue(s) in the tracker with specific fixes for the Rest status codes, along the lines of https://www.symmetricds.org/issues/view.php?id=3279
-
Brian, check the sym_data_gap table before and after you run the routing job. That able represents what data_id's SymmetricDS will look for to route in sym_data. The last row in sym_data_gap is referred to as the "last gap" and should have the next expected data_id on the low end, and a much higher data_id on the high end. You could also look for patterns as to which sym_data events route and which do not. You could look for patterns on the channel_id as well as trigger_hist_id.
-
Nitesh, You should have 1 properties file for each logical database. Say you have 1 MS-SQL db, and 2 PostgresSQL servers - acting as a cluster, so acting like 1 database. In that case, you would have 1 property file for the MSSQL, and 1 property file for the Postgres cluster. Hope that helps and let me know if I misunderstood. Mark
-
Hi Ray, For the first case you saw, there are certain batch statuses that trigger the batch to get re-extracted, which might explain how the situation self-healed. It is odd that got another error the second time you tested, and that it had to do with a different field. I wonder if your outgoing batches are getting corrupted somehow. Some things to consider: 1) Try another test, where you first clear your <symds>/tmp directory (on the outgoing node) 2) While the error is occuring, compare the batch...
-
Ray, 3 things: 1) Your first log mentions batches 116, 127, 128 needing to be re-sent. You could try clearing just those out of tmp, and resetting the status of those to NE on sym_outgoing_batch. The Load id of 128 in the other log snippet is not the same as batch id 128 - that log snippet mentions extracting batch 5531 (for load id 128). 2) When you clear tmp during an initiat load (with default paramters), the entire initial load will get re-extracted which can be time consuming. 3) With certain...
-
Guys, that problem can also happen if there is a slow query to sym_outgoing_batch and the client times out while the server is still processing the request. There will be some improvement for this in 3.9.4, see: https://www.symmetricds.org/issues/view.php?id=3406
-
Guys, that problem can also happen if there is a slow query to sym_outgoing_batch and the client times out while the server is still processing the request. There will be some improvement for this in 3.9.4, see: https://www.symmetricds.org/issues/view.php?id=3406
-
https://www.symmetricds.org/issues/view.php?id=3408
-
Thank you Lukas, I missed that. Your PR has been been merged now and will be in 3.9.4 - thank you for your contribution.
-
Frank, we are evaluating the use of jsch based on your post: https://www.symmetricds.org/issues/view.php?id=3395 kxml2 is used by DatabaseXmlUtil class. It probably would be possible to port to another, more up-to-date XML provider, but would take code changes. Pull requests are always welcome. bsh - is beanshell, and is pretty central to SymmetricDS' extension mechanism. I don't see a newer version available. Let us know if you are aware of a newer version. Mark
-
We will look into recreating this and let you know what we find.
-
Nixon, Here is the general flow of the data from source to target: 1. Data change is captured to sym_data 2. Data change is extracted to a tmp file under <symds>/tmp 3. Data is transmitted over HTTP to the target. 4. Data is written to tmp file on the target, under <symds>/incoming 5. Data is loaded into the target over JDBC. Can you pin down at which step the encoding gets curropted? It's probably 1,2 or 4,5 where it happens. Mark
-
Hmm.. It does sound like something got locked up in the DB. Can you share a log file from when it happened?
-
Luiz, RT = means that a batch is in "Routing" status. Routing is the process where SymmetricDS reads through sym_data and puts the data into batches (ie, create sym_data_event and sym_outgoing_batch.) If a table is getting locked in your DB during routing, that would explain why the routing gets stuck. Some questions: does this happen at a certain time of day? what DB platform are you running? I would try to isolate which DB tables are locked for you, and where the contention is. We had some users...
-
Mahesh, Unfortunately that is not supported in SymmetricDS. You have a choice betwen: 1) Transactional: where you would keep everything on the same channel. 2) Non-transactional, but keep more data flowing on seperate channels, as you have done. You could also consider writing a IDatabaseWriterErrorHandler extension. That would give you the ability to quarentine the problematic data for later manual review. That way you could keep these on the same channel and also let batches progress while stashing...
-
Matthias, it will be in 3.9. If all goes well that will be released today. Mark
-
Jonas, check out this example that is a little more recent. https://www.jumpmind.com/downloads/symmetricds/extras/Android-FileSync-Demo.zip Feel free to tweak that and contribute those changes as a better starting point for new users. Thanks for your interest.
-
Tim, Thanks for your continued feedback on Android. The stuff about instanceId appears to be a legitimate bug on Android. The placement of those methods is correct, but the instance.uuid is supposed to be written under the Symmetric install in a desktop installation, but the path comes out invalid on Android. We'll review that logic for Android - thank you for pointing that out. We'll also look into the general sync problems you are having. We recently fixed some bugs that we thought would fix this,...
-
Matthias, if I understand correctly, you want to sync the 2 most recent events on a schedule, whether they have changed or not. If this understanding is correct, it would be possible to periodically insert a reload event for those 2 rows using Symmetric. In SymmetricDS 3.9 (to be released very soon), you could do a custom SQL job in the sym_job table. In SymmetricDS 3.8, you could do a heartbeat listenter extension, so your sync would run in conjunction with the periodic heartbeat. The SQL to queue...
-
Matthias, if I understand correctly, you want to sync the 2 most recent events on a schedule, whether they have changed or not. If this understanding is correct, it would be possible to periodically insert a reload event for those 2 rows using Symmetric. In SymmetricDS 3.9 (to be released very soon), you could do a custom SQL job in the sym_job table. In SymmetricDS 3.8, you could do a heartbeat listenter extension, so your sync would run in conjunction with the periodic heartbeat. The SQL to queue...
-
Matthias, if I understand correctly, you want to sync the 2 most recent events on a schedule, whether they have changed or not. If this understanding is correct, it would be possible to periodically insert a reload event for those 2 rows using Symmetric. In SymmetricDS 3.9 (to be released very soon), you could do a custom SQL job in the sym_job table. In SymmetricDS 3.8, you could do a heartbeat listenter extension, so your sync would run in conjunction with the periodic heartbeat. The SQL to queue...
-
A note on the creation of outgoing batches... Outgoing batches are created by the "Routing" job. The Routing job looks at the sym_data_gap table to determine which data to select from sym_data. So given: sym_data_gap start_id=1, end_id=5000001 and sym_data that has data_id's, of 1,2,3: Routing will run and create sym_data_events for 1,2,3, with corresponding rows in sym_outgoing_batch. Sometimes the sym_data_gap can get out of range with the data_id's in sym_data. In that case you can safely stop...
-
Wayne, can you re-test with 3.8.32 and let us know? I wonder if this will help: http://www.symmetricds.org/issues/view.php?id=3318
-
Guys, thanks for the offers of help. I think we got to the bottom of this one and it is fixed in 3.8.32. https://sourceforge.net/projects/symmetricds/files/symmetricds/symmetricds-3.8/ Mark
1 >