GitLab

Bitrix24 CRM Migrate Data

Comprehensive implementation of the tasks set:

https://bitrix.vidaselect.com/company/personal/user/322/tasks/task/view/575915/

https://bitrix.vidaselect.com/company/personal/user/322/tasks/task/view/624444/

First of all, it should be noted that when creating custom fields in the CRM system interface, absolutely non-informative field names are created.

UF_CRM_1234567890 <- unix date

There is another way, and it is better to ask to create programmer fields at the level of the administrative panel with readable names.

But at the moment we have a huge number of such fields created.

In the \Dev\Constants\ namespace, I created classes where I manually specified the correspondence of these fields with understandable keys:

[WARNING] For the centralized connection of the MSSQL server, data was added to the file: /bitrix/.settings.php In case of changing accesses - change only here.

All classes are located here: /local/lib/Migrations

To configure the exchange of new data, you need to add create two files and add settings to the observer:

1. /local/lib/Migrations/ORM/ [support external tables for Bitrix24 ORM]
2. /local/lib/Migrations/Sources [Basic settings for data exchange. Source CPM or ORM classes. List of fields. Possible field mapping for other types (if necessary)]

Conditions for filtering data are set in the method ::getDbConditions():

[WARNING] These two files must have the same names and end with %YOUR_NAME%Table (as Bitrix24 wishes).

3. /local/lib/Migrations/Observer.php [::options and getSimilar options] Processing event names from a file: local/modules/dev.tools/lib/Handlers.php

At the moment, the following events are being processed:

1. Create a match
2. Match updates [The row is completely overwritten with actual data]
3. Deleting a match [in the external table, the data is not deleted, but simply marked]

Event processing can be disabled in the Bitrix24 admin panel [dev.tools module]: https://bitrix.vidaselect.com/bitrix/admin/settings.php?mid=dev.tools&lang=en

Agent Mode allows you to take the load off the client and delay sending data to the MSSQL server.

A new table is created automatically in the MSSQL server. As well as new added fields. It is not possible to automatically change the type of a field, but you can change the type of information that is sent. For example, convert to text.

The working version has the abbreviation _prod.

There are several important things to note. All submissions and their results are recorded in a special logging table. In the event of an error sending the entire data, an attempt will be made to sequentially send all existing columns of data. This will help to avoid the problem with data that is not allowed to be written to the table.

It is in the erroneous cell that there will be no data, but this will not affect the entire row as a whole.

In the current version, you can perform a full upload of data using the Symfony Console in terminal: php /home/bitrix/www/local/console migrate

New table classes will appear there automatically. After the data upload is completed, you will be prompted to run a test. The test is a full repetition of creating, updating and deleting a CRM entity.

To avoid unexpected situations, a limit of 100,000 items has been set.

An example of independent launch of the exchange, including for specified elements: local/tests/Dev/Migrations/MigratorTest.php

This tool is designed to transfer a variety of Bitrix24 data and should not require complex settings.

If necessary, sufficiently detailed logs can be created. All of them have names similar to the classes and methods used in this solution: /local/log