Activity for Josh Hicks
-
Josh Hicks posted a comment on discussion Help
Please submit a pull request for the change you want and we can review it to part of the working branch.
-
Bulk loaders are only available in the professional edition.
-
Routing controls the data gaps so as data is routed it will adjust the gaps. Is routing running to completion without failure? There is a parameter that controls data gap expiration as well if you want to be adjusted. It will default to 1 day. purge.expired.data.gap.retention.minutes=1440
-
Sym data captures an event type when the change is made.
-
There is a DB Compare tool in the pro version. Otherewise you need to make sure there are not any outstanding batches, no data to route, and no batches in error.
-
You can achieve this with one central node(engine). Each site property file would have a property to identify its database on the central. For example : central.db.name Then in the site to central router put the target catalog name to the variable $(central.db.name).
-
The transform pk settings are used in a where clause so it would be applied to updates and deletes but not inserts. So if the database does not prevent duplicates (via a composite key) you would need to check for duplicates before every insert of data. This could be done with a transform or load filter but would slow down inserts as they would need to perform the additional check before deciding to insert.
-
David we work with ISV vendors all the time and to be sure you are not in violation of the GPL the open source has in place you should review. Here is a link with some more details. https://www.jumpmind.com/products/symmetricds/isv
-
You can just remove the channel and the purge will clean up old references to it where needed. Any pending changes will be lost on this channel.
-
It might be a table with large objects but it also could be an already resolved issue fixed in 3.13.
-
Columns should be upper case, a .equals() on a string, and semi colons to complete statements. Try this: if (REGISTRATIONSTATUS.equals("Pending Check")){ // row will not be loaded return false; } else { // row will be loaded return true; }
-
You need transforms to change the table name. The same trigger will not work as it has different names at each location if you are setting up bi-directionally.
-
SymmetricDS replicates only changes as they occur. It can be used to load existing data between nodes. However it sounds like you are looking for a feature that will analyze the current state of each database and get them in sync. There is a dbcompare feature that can be used to achieve this use case if the nodes/engines are all running on a single instance of SymmetricDS. You can find dbcompare in the bin folder of SymmetricDS and in the documentation.
-
NGINX is likely blocking. You would need all the HTTP methods available including, head, put, post, get.
-
Try setting the full path to java in the conf/sym_service.conf
-
Set these in the engine file so that they are not started when SymmetricDS starts up. Then you can call them on demand when they will not interfere with your application. start.push.job=false start.pull.job=false start.router.job=false start.purge.outgoing.job=false start.purge.incoming.job=false start.heartbeat.job=false start.monitor.job=false
-
This appears to be a firewall or network issue between the client and the registration node. The client is trying to pull but is receiving a connection reset message off the network stack. Can you verify there are not any firewalls in place or shut them down temporarily to verify? Suppressed: java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(Unknown Source) at java.net.SocketOutputStream.write(Unknown...
-
We would need the original SQL Exception message in the logs. The logging suppresses the messages so that it does not fill the logs too quickly. The full stack trace should be at top or the first occurrence of the error. Can you send that over to determine the root cause.
-
I do not think the csv route is the proper way to go. I agree with Phil that you should just replicate the underlying tables regardless how complicated the view is in place. Otherwise the view(s) might not potentially work with only partial data.
-
That is fine to restore the back up with the sym tables. It will pick up where it left off prior to backup and restore. What issues are you having?
-
Shutting SymmetricDS down will not cause any loss of capture. Since it is trigger based the changes will continue to capture into sym_data. You will want to copy the following items from the old server to the new server - The security folder in SymmetricDS - The engine file(s) - The sync.url property in the master server engine file to represent the new server name - The sync url column in the master servers database sym_node table to represent the new server name - You may need to update the registration.url...
-
What version of SymmetricDS are you running? Can you send your db.url and db.driver values?
-
Yes there are custom jobs you can configure. https://www.symmetricds.org/doc/3.12/html/user-guide.html#_jobs
-
Using open source - All replication in the open source product is based on triggers for change data capture. You could setup an extract only (same as load only) for your source node so that the sym tables are created in H2. Then when you create the trigger entries in sym_trigger you need to turn off the sync_on_update, sync_on_insert, and sync_on_delete flags. This would prevent triggers from being installed on the database but still allow for initial loads. Then you would need to crate an initial...
-
There are two sets of files. There are bat and bash files.
-
Here is a blog that should help explain it. https://www.jumpmind.com/blog/symmetricds-transform-pipeline
-
Transforms can be confusing. Basically the source values must match the upstream values set on the target values. Here is a blog that helps explain. https://www.jumpmind.com/blog/symmetricds-transform-pipeline
-
This is working as designed, all configuration is sent to all nodes. Each node will only apply the configuration that it needs. We use the same change capture techniques to replication configuration so that it is in sync at all nodes.
-
Do you have an Oracle driver jar in the lib folder of SymmetricDS? Can you remove that and retry if one is present. Have you tried with a lower version of Java Im not sure we have tried something that recent with the Sybase driver in place.
-
Alessio The channels do not block other nodes from syncing. The channels are used to separate tables not nodes. So a batch in error will only effect the node that is in error not other nodes. You can certainly put all client nodes in a single group if they sync the same way (regardless of data subsetting). You can subset data during routing to determine which client it should be sent to. Here is a blog on subsetting data with routers. https://www.jumpmind.com/blog/subsetting-data-during-replication...
-
Is the server that is running SymmetricDS running other applications that can use up memory as well? Do you have any special networking setup that would cause each client interaction to appear with a new session id?
-
What data types are on the table VGCBS.DATA_VALIDATION_MASTER? Are you using Longs?
-
You need to increase the timeouts on the Oracle queries. You can adjust db.properties and set all these timeouts higher. oracle.net.CONNECT_TIMEOUT=60000;oracle.net.READ_TIMEOUT=360000;oracle.jdbc.ReadTimeout=360000;SetBigStringTryClob=true You probably should increase the SymmetricDS query time out as well to match. db.sql.query.timeout.seconds
-
You would install SymmetricDS on the cloud and on your local pc. This will ensure a faster database connection and SymmetricDS will replicate over HTTP/s on the sync url provided.
-
https://www.symmetricds.org/doc/3.12/html/user-guide.html#_running_as_a_service
-
You have Out of Memory errors in the log you may check conf/sym_service.conf to adjust the memory. wrapper.java.initmemory=1024 wrapper.java.maxmemory=1024
-
It has been added and should be supported in latest version.
-
It should not matter if temporal tables are on in MariaDB are you having issues?
-
What if you run the following (insert the PID that is failing at the time if it is not 15145) ls /proc/15145/cmdline cat /proc/15145/cmdline
-
Change all your group links to either P or W
-
https://www.symmetricds.org/doc/3.12/html/user-guide.html#_running_as_a_service
-
Felipe I tried reproducing with a similar setup and was not able to do so. Both the wildcard and non wildcard cases worked. Are there any more details you can provide?
-
That sounds like a viable solution thanks for sharing. If you enter an issue we can add it to a future roadmap.
-
What version of Oracle are you running? The application is calling this utility dbms_utility.db_version and receiving question marks (???) when we expect integers that would represent the database version.
-
SymmetricDS can replicate across different database platforms such as MSSQL and MySQL so you are set there. It can also replicate between databases that do not match on each side. This however requires more configuration to setup transforms based on your business requirements. Here is some documentation for the reverse scenario but you can probably apply the concepts to what you need. https://www.symmetricds.org/docs/how-to/transform-multiple-tables-to-single-table
-
If you populate the sym_node_security.initial_load_end_time this should resolve the issue for this client.
-
This has been resolved in 3.12.7. Let us know if you are still running into the issue
-
In a single server mode it is started using the sym command with the -p argument to specify a property file to use. This would run in single server mode. If this is not provided it will use the engines directory as a default location and load all .properties files in this location. So it seems like you may have been starting SymmetricDS with different options.
-
The source engine is never starting. Are you using a system user account for this engine? Look in the source engine file db.user=... This must be an actual Oracle user and not a system account. The dest node will not register until the source node starts up successfully.
-
Teo, You could set up two triggers that use a wildcard as their table name. This will ensure any new tables would be added to replication. The corp triggers though need an additional flag turned on at the trigger configuration, sync on incoming. This will recapture changes by SymmetricDS and send them out to the other stores. The store configuration should not have this turned on. This will allow changes to sync to other stores without looping. We have not worked with CouchDB before so I am not sure...
-
If it gets stuck again can you post the logs from each side to see what was happening at the time it stopped.
-
Can you add some logging to each of your conditions to determine which path it actually went down.
-
Do you have a PK on the target table? With 40k updates an no PK it will use all columns as part of the PK and the update will run slower naturally on the target without a PK.
-
Batches in NE status represent data that has been routed and placed into batches. These batches are awaiting the push or pull job (depending on your setup) to transfer these batches to the target. So based on the state you described it sounds like the push or pull jobs are not occurring through your embedded application implementation of SymmetricDS. You may want to use your config in a standalone setup to verify how it would work in a standalone situation. Then port that over to your embedded setup....
-
The jobs you mention are responsible for generate the files used during offline. The node.offline property is what drives the node being offline or not. The jobs are just running on a schedule regardless of if they have work to do or not.
-
We added a feature request to track this as a new job type. https://www.symmetricds.org/issues/view.php?id=4664
-
During an initial load the FK and indexes are removed from the target tables while the data is loading. Once the data is fully loaded they will be added back to the tables. Did you cancel a load in the middle by chance which would interrupt this process?
-
Could you add an issue to the issue tracker with the steps to reproduce the error and the table definitions you have and the databases used.
-
Do you have any patches on your classpath?
-
Remove the entry from sym_trigger_router for these tables
-
During an initial load you can have Symmetri create target tables for you.
-
I would start with a single default channel for all tables first. You should not need a channel per table. Sometimes in advanced setups users will create a channel for several related tables based on a business unit/domain. Too many channels could slow down routing and may be leading to memory error.
-
The open source version does not have a web console. You would need to run the professional version.
-
You should upgrade to 3.11 there were some routing improvements made around this exact use case. This would produce 200 times less data when routed. Here is a blog around the new common batch improvements made in 3.11. https://www.jumpmind.com/blog/common-batches-311
-
1.) Is this a default router? 2.) Is SymmetricDS running near your database (in the cloud as well) otherwise the JDBC connection to the cloud will slow things down. 3.) How many channels are configured? 4.) Upgrading to 3.11 or highter should put these in common batches now if the same batch is going to all 200 clients. This would prevent so many sym_data_event rows. Right now it is doing 200 times the amount of rows in sym_data_event that it would in the new common routing. Here is a blog explaining...
-
Upgrading to 3.12 will have a lot of performance improvements around this use case.
-
The trigger defines a source table of : new_table_rach. This needs to be the source table name in the transform as well. However I see : New-rach as the source table name in the tranform. Change this value in the transform to be new_table_rach.
-
In the DB19C.properties leave the registration.url blank as it is the registration server. registration.url=
-
This issue has been fixed in 3.12 releae.
-
Make sure your target is empty when using bulk. You can just set it to 'bulk' and it will determine the proper bulk loader to use.
-
Do you have your SymmetricDS installation running near the databases? If you have databases in different networks (WAN, cloud to on premise etc) you would need to install SymmetricDS at each location.
-
You can run the ./gradlew clean build from the symmetric-assemble folder
-
If you perform a full initial load it will order the tables based on foreign key relationships. So that the parent tables are loaded first. You can also specify to truncate or delete first to ensure the target is empty before loading the data. If something fails you would need to address why it is failing (maybe target schema or relationships are not the same as the source) and adjust configuration and cancel the load and send again.
-
You can use any port number you like. You may want to stay over 1024 though.
-
Yes I would run routing every 5s. Depending on your system but it is the more resource intensive that the push pull jobs.
-
Alessio SymmetricDS is designed for asynchronous replication so it will never truly be real time but instead "near real time". This prevents holding onto a user transaction while replication occurs. If there was an issue with the transmission of the change or in loading the data this would effect the end users interaction as they would not be able to commit their data. With the asynchronous approach the end user is not affected. The trade off is there is the potential gap in replication. You could...
-
It looks like a firewall or some other process might be blocking communication. I noticed your port value was 1434 for Symmetric while using HTTP. This is different than the database port and usually defaults to a higher port value (31415). Sometimes lower port values are reserved or blocked. Have you tried the default ports for your sync urls?
-
Yes it is supported in 3.11 but needs to be added to the documentation.
-
Are you clustered at central?
-
MSSQL log based replication is on the roadmap for 3.12 SymmetricDS professional.
-
SymmetricDS utilizes JDBC which does not perform well over wide area connection. So we recommend installing SymmetricDS close to your database so that the JDBC calls between the SymmetricDS and the underlying database are reliable and local. With that being said we would recommend installing SymmetricDS in the cloud when utilizing a cloud database. If you install SymmetricDS on-prem and connect it to a cloud database it will not perform as well with the JDBC calls being made over that connection...
-
You could try using an extension that would check the first url and if it is not available return your fallback ur. Here is some sample code where you would implement your check in the resolveUrl method. Then add it as an extension. http://www.symmetricds.org/doc/3.11/html/user-guide.html#_extensions import java.net.URI; import org.jumpmind.symmetric.ISymmetricEngine; import org.jumpmind.symmetric.transport.ISyncUrlExtension; import org.jumpmind.symmetric.ext.ISymmetricEngineAware; import org.slf4j.Logger;...
-
I believe we may have talked on the chat as well but you may also want to add the parameter auto.reload=true so that an initial load will be sent when the node registers. So you could turn this on and register the node again. The other option is to add an entry into the sym_table_reload request to send an on demand node. In here there is a column to also create tables if needed. http://www.symmetricds.org/doc/3.11/html/user-guide.html#_load_data
-
What is the error you are getting? What version are you on? Is your SymmetricDS instance installed on the same netowrk as your database server. If it is over a WAN the JDBC connection might be the issue and you may want to install SDS closer to the database server.
-
If you are able to upgrade there have been a lot of fixes since 3.5 with the purge job. We have had customers recently upgrade from 3.3 to 3.11 with success. The purge job is responsible for purging the sym_data, sym_outgoing_batch, sym_data_event, and sym_incoming_batch tables. If these tables are rather large its likely purge is not running or not completing successfully. You could increase the timeout on your databse connection if it is timing out.
-
Add this to your db.url to MySQL ,useSSL=false
-
You need to make sure there are triggers and trigger routers setup between S-001 and M-000 and also M-000 and S-002. It sounds like you want a change to sync from slave up to master and back down to the other slaves. Is this correct? If so you will need two sets of triggers. You may want to prefix one with "master" for example. This allows you to set the sync on incoming flag for the master triggers only. This flag will ensure a change coming in as part of replication will be recaptured and synced...
-
We can change the message to debug level for a future release. Is it installing the triggers now though?
-
The best way to subset data to the targets in SymmetricDS is through the use of various routers. You were on the right path with a column match router for the use case you described where the column that contains the node id to route to is directly on the table. This should be a varchar type column. Here is a blog that describes in more details the various ways to subset data in SymmetricDS as well as pros and cons of each. https://www.jumpmind.com/blog/subsetting-data-during-replication
-
Different channels are not necessary for this use case. You need to verify the source catalog and schema on your trigger definition for the ccc table. This table should have a source schema of "symm2". After a restart it should run the sync triggers job and apply the trigger on the proper source table (assuming you do have access to alter tables in that schema). Once the trigger is all setup you should be ready to go. The router from corp to store though would need to specify the target schema "symm"....
-
I apologize this was addressed and fixed in 3.8.31. http://www.symmetricds.org/issues/view.php?id=3298
-
This is not supported currently on the ASE dialect. You would need your connection for the node to be the same database as your source tables. You could submit an issue request and we can add it to the road map for a future release.
-
What is your overall use case? To sync tables in different schemas from source to target? If so you can still use a single user to connect into your source database. Then provide a source catalog and source schema on the triggers to replicate tables from a variety of locations. Your SymmetricDS user would need to have access to all these schemas. If you need to adjust the target schemas they replicate to you can provide in the target catalog and target schemas of the routers.
-
This is interesting in the logging do you know what the tab char might be? Does your username have a special character on the end? userid=MySymSchemaName{tabChar}
-
Yes you can set it in the target_schema of the router or using transforms. If all tables will be the same schema you can do it in a router. Otherwise you can do it at the table level with transforms.
-
No the client node should have both registration url (back to the registration server) and sync url (the url this client node will sync on). Can you verify that both the server and client have sym_node_group_link entries that match? Essentially you need a group link and a router that allows data to be sent from server to client. You could allows the row in sym_node_identity for your client database to force the client to re-register and pull down configuration. This assumes though that the configuration...
-
Can you verify both these parameters are not being adjusted in your engine? Otherwise at startup you should see logging around running the sync triggers job. The sync triggers job would be responsible for detecting the missing triggers and rebuild them. Also check you entries in sym_trigger, sym_router, and sym_trigger_router that would match the source node for these tables otherwise they will not be installed. Specifically check sym_trigger_router to ensure the entries are enabled as well. auto.sync.triggers=true...
-
The way most of our clients solve this issue is to add a composite key at the server which includes a location id and the id column. Then a transform is used (variable type on the location id) to provide the client external id when the data is sent from the client. This preserves all the auto increment ids for all 3 tables. This also prevents collisions now at the server because the same id on A could appear many times as the new composite key at the server will include location id as well. Also...
1 >