#1629 All replicas ignored for data synchro in clean iTop database

[Graham Snow]

Hi,

The issue could be my understanding or a bug, but I have read all the documentation and cannot find the answer to my issue. I am running a data synchronization, but all the replicas are being 'ignored':

Screenshot

I am trying to import 10 records for a new CI I have created called DatabaseSchema. To prove that the issue is not a custom CI, I also tried to import some contacts and I get the same problem.

The documentation says that "Ignored records corresponds to objects no longer existing in iTop." However, this is a clean database, none of these CIs or contacts have existed before, so I don't understand the problem.

A sample of the data:

id	primary_key	name	org_id	business_criticity
3	APEX_040200	APEX_040200	3	high
4	APPQOSSYS	APPQOSSYS	3	high
5	AUDSYS	AUDSYS	3	high

I'm also not clear on what the primary_key is used for. Perhaps this is my issue? As the objects have never existed before, I am not clear on what should be in the primary_key field. I have also tried the primary_key as null but this makes no difference.

Otherwise the system is not giving me any error or warning for why the items are being ignored.

Your help with this would be very much appreciated.

Thanks, Graham

[Jeffrey Bostoen]

Could you also share the settings on your data source and the reconciliation part?

primary_key seems to be something you can use freely (or if it comes from another source which has a primary key for it).

[Graham Snow]

I hope this is what you need to know. Thanks for your help with this:

https://imgur.com/a/s2BVwSR

https://imgur.com/a/bXuENDU

I have restricted the amount of data I am importing to just 3 fields/columns as I was trying to keep the import simple while testing. These 3 are just the mandatory fields.

[Hipska]

You need to change the "Reconciliation Policy" to "Use attributes" if you don't add the id's of iTop objects in the primary_key column.

[Graham Snow]

Hi Hipska, thanks for the quick reply. I've previously used the "Reconciliation Policy" setting of "Use the attributes" and had the same problem. To prove this, I just changed the setting to "Use the attributes" again and ran syncho_exec.php again.

As you can see from the screenshot, I'm still having the same problem:

https://imgur.com/a/cjXjDHV

I have just tried this by setting primary keys to be a unique value, and tried setting the value to null. Same problem either way :(

[Hipska]

Now this can come because the fields/rows in the synchro table have not been updated I think. Can you try again after adding 1 row?

[Graham Snow]

I've added 1 row (you can see the number of items Ignored has gone from 10 to 11):

https://imgur.com/a/Pr8R6P8

Sadly no luck

[Hipska]

And another question, why did you create "DatabaseSchema" as this already exists in default datamodel?

[Graham Snow]

Apologies, I got confused, DatabaseSchema was not one of mine (I have created Table, View and Cluster). I've created 15 new classes/modules so lost track of which were mine! I have redefined DatabaseSchema though, so that it links to Cluster, to View and Table.

[Pierre Goiffon]

Hello,

Maybe opening the corresponding Replica objects will give you some hints ? Also, you didn't provided the synchro_exec.php execution logs...

Viewing the DatabaseSchema object specs in iTop data model viewer, there is a mandatory attribute you didn't provide : dbserver_id... It is a good reason for object creation to fail...

[Graham Snow]

Hello Pierre,

Thank you for your reply. Looking at the Replica object, there are no errors or warnings.

Firstly, where do I find or enable the synchro_exec.php logs? There is nothing in the log directory.

The reason I didn't include dbserver_id is because I have changed the data model so that Database Schema now connects to Database Service instead of Database Server. I have included this in the table:

https://imgur.com/a/rolZ5bY

To prove that the issue is nothing to do with my data model changes, I have the same problem with Contact, which I have not redefined.

https://imgur.com/a/TBeT144

Any thoughts?

Thanks in advance for your help!

[Pierre Goiffon]

What I meant by synchro_exec.php log is what the page sends to stdout.

Isn't there is a mandatory external key to Database Service now ? Can you provide a screenshot of the class fields in the data model viewer (admin tools / data model) ?

[Graham Snow]

This is the page output:

Replicas: 1
Replicas touched since last synchro: 0
Objects deleted: 0
Objects deletion errors: 0
Objects obsoleted: 0
Objects obsolescence errors: 0
Objects created: 0 (0 warnings)
Objects creation errors: 0
Objects updated: 0 (0 warnings)
Objects update errors: 0
Objects reconciled (updated): 0 (0 warnings)
Objects reconciled (unchanged): 0 (0 warnings)
Objects reconciliation errors: 0
Replica disappeared, no action taken: 0

There is a mandatory external key to Database Service. This is the one circled in the screenshot, and as you can see has been supplied in the table. I have checked and the ID is a valid Database Service:

https://imgur.com/a/QyVhKfr

This is the data model for DatabaseSchema:

https://imgur.com/a/TWwYOSj

This is the data model for Contact (which I also cannot import):

https://imgur.com/a/MEDQOqz

[Thomas Radszuweit]

Hi everyone,

I know this Ticket is quiet old but I have a question which might get in the right direction of this Problem too. I stumbled upon it as i tried to import Software from another Host and then tried to instantiate and link them to the corresponding Host Object.

Can someone confirm that Destination Type on this Image https://imgur.com/a/QyVhKfr
posted by Graham Snow is correct?
When i try to import into e. G. "DB Server" i get the same problem with Destination Type. When i check this setting on the Software Import-Datasource there stands Software as i would expect.

Thank you

[Hipska]

That can happen when you removed replicas before and did other stuff to you DB manually.

[Thomas Radszuweit]

Yeah i thought about this too. So before i was writing my comment above, i tried to do this on a plain install without any additional data.
So there was no Demo Data or any other changes except one Server Object and one sync User. Both were created via Web Gui.

Then i made four Import DataSources.
One to import Software running into DBs: synchro_data_software_1
And three for

• WebServer: synchro_data_webserver_2
• DB Server: synchro_data_dbserver_3
• OtherSoftware: synchro_data_othersoftware_4

The result is except in the Software Data Source (which runs flawlessly) the same. All three DataSources had the Destination Type Organization.

[Hipska]

Could you show an example of data you are entering into the webserver synchro? And also the configuration of that synchro.

[Thomas Radszuweit]

Anyone else can check this behaviour on a plain installation? I'm pretty sure this one is a bug.

[Thomas Radszuweit]

Ok i made a db export of software and webserver synchro and attached them. This is the configuration. All standard nothing changed on attribute tab.

Synch Overview

Software Import properties
Software Import status
Software Import attributes
Software Import replica history

Webserver Import properties
Webserver Import status
Webserver Import attributes
Webserver Import replica history

[Hipska]

It doesn't help much if they are in German, but I see no problems. The webservers already existed so they are not created. The replica then indeed has no object ID and also does have a default value for the class type in that case. That is not a problem, but can be ignored.

What if you change one of the webservers and then do a synchro, so the synchro must change a field?

[Thomas Radszuweit]

Ok i changed the language to English and re-added the Screenshots in the above post.

As advised by Hipska i also changed one of the webservers name and executed another synch. This one ran successfully. I don't understand why this one is now running through.

Webserver Import replica history after successful run

The destination type is now also as i would expect it.

Thank you very much Hipska.
Can you explain why this problem exists?

[Pierre Goiffon]

Hello,
Thomas I really don't understand what is the problem you're pointing ?
Can you share a very precise step by step guide to reproduce, using a fresh iTop install with sample data for example ?

[Thomas Radszuweit]

Ok i will try to describe what i'm trying to archive.

We have several Linux based Servers and VMs which i'm going to inventory into iTop. To archive this i'm running a Python-Script provisioned by Ansible on to every system.

This script has the following steps:

1. It collects all software packages and stores only some key components like webserver, database-server, GIT and so on.
2. After collecting the data the script will check if the software does already exist in iTop. If not it will be transfered into synchro_data_software_1. In this step there will also be a data import triggerd by a simple post call to synchro_exec.php
3. In the next step it will check if the previously collected software packages are already instanciated in iTop for this specific host. (API Call to class SoftwareInstance). If not, it will push that data into one of the following data-tables: synchro_data_webserver_2, synchro_data_dbserver_3 or synchro_data_othersoftware_4. If instances are found it will do nothing to prevent duplicated entrys.
4. In this last step there will be aditional post triggers like in the second step calling synchro_exec.php. This time to all other synchro jobs (synchro_data_webserver_2, synchro_data_dbserver_3 and synchro_data_othersoftware_4).

Steps to reproduce:

Installed the following software on CentOS 7.9.2009 (Core)

• MariaDB 10.3.28
• HTTPD 2.4.6-97
• PHP 7.1.33
• php-cli-7.1.33-12
• php-mbstring-7.1.33-12
• php-pecl-zip-1.19.2-1
• php-common-7.1.33-12
• php-mysqlnd-7.1.33-12
• php-mcrypt-7.1.33-12
• php-xml-7.1.33-12
• php-ldap-7.1.33-12
• php-json-7.1.33-12
• php-pdo-7.1.33-12
• php-soap-7.1.33-12
• php-gd-7.1.33-12

Downloaded and installed iTop version 2.6.3-5092

After a successful plain installation (no demo data), i do the following tasks:

• Create an example organization.
• Create a sync user with password and the following two profiles: REST Services User & Administrator (Administrator is needed otherwise the sync user is not allowed to create software instances. There may be a better profile than Administrator. But currently i don't know which one to take and it shouldn't be relevant at all.)
• Create a test server in iTop like testvm.example.org

• Create the following synchronisation data sources: software & webserver (The two other data sources for dbserver and othersoftware are not needed in this example)

• Import Data from the above thread posted by myself on Friday the 5th of March:

• synchro_data_software_1.csv into db-table: synchro_data_software_1
• synchro_data_webserver_2.csv into db-table: synchro_data_webserver_2
• Call the data import manually via cli
• php -q /var/www/html/synchro/synchro_exec.php --auth_user=synch --auth_pwd=<user_password> --data_sources=1</user_password>
• php -q /var/www/html/synchro/synchro_exec.php --auth_user=synch --auth_pwd=<user_password> --data_sources=2</user_password>

The first call to synchro_exec.php with data_source 1 runs successfully. If you check the software catalog you should see the imported items.
The second call to synchro_exec.php will not throw any errors on the cli. But you won't get any webserver instances on testvm.example.org. A look into the webserver replica history you will see a faulty destination type.

Hope this helps and is clarifying some things.