数据更新基础配置

scheme.load配置项 类型 说明
type jdbckafkaservicerabbitmq 定义 目标数据源类型
batch_size(batch-size) int load批量操作的数据量, type=servicerabbitmq时才有效
batch_skip_error(batch-skip-error) boolean 批量load时是否跳过错误, type=service时才有效

注意:scheme.load.type=rabbitmq时,表示数据同步的目标数据库是RabbitMQ。每一种目标数据源都有对应的扩展配置,下面讲讲数据同步到RabbitMQ数据源的扩展配置

数据同步到RabbitMQ扩展配置

load.type=rabbitmq的扩展配置项 类型 说明
datasource rabbitmq@{datasourceName}(Datasource引用) 必须引用rabbitmq数据源
care List<String> 过滤数据,只发送关心的字段到MQ
compress String 消息发送到MQ中使用的压缩算法,可选snappy
exchange String 设置RabbitMQ交换器名称,默认: amq.default
route_key(route-key) String exchange 和 queueName 对应的 route key,默认: queueName配置值
queue_name(queue-name) String 队列名称,用于保存非删除类型的数据
delete_route_key(delete-route-key) String exchange 和 deleteQueueName 对应的 route key,默认: deleteQueueName配置值(当scheme.extract.type=hermes时生效)
delete_queue_name(delete-queue-name) String 队列名称,用于保存删除类型的数据(当scheme.extract.type=hermes时生效)

同步新增或更新的数据

使用route_keyqueue_name配置定义新增或者更新的数据同步到哪个消息队列

同步增量删除的数据

使用delete_route_keydelete_queue_name配置定义删除的数据同步到哪个消息队列

使用Demo

scheme:
  LoadToRabbitMQTest-01:
    enable: true
    extract:
      - type: hermes
        datasource: rabbitmq@b2b_third_prod
        table: b2b_third.tb_package_main
        queue_prefix: lizw-test-
    load:
      - type: rabbitmq
        datasource: rabbitmq@b2b_third_prod
        # care: ['id', 'package_id', 'store_no', 'order_code', 'erp_order_code']
        # compress: snappy
        exchange: lizw-test-load-exchange
        route_key: lizw-test-load
        queue_name: lizw-test-load-queue
        delete_route_key: lizw-test-load
        delete_queue_name: lizw-test-load-queue

关键日志

2020-03-19  11:05:45.125 [pool-20-thread-4] DEBUG com.jzt.data.sync.datax.job.support.JobLogger - [scheme@LoadToRabbitMQTest-01[0]-437972000456048640] - [HermesExtract] 触发增量任务: message=[{"size":2,"data":[{"database":"b2b_third","type":"UPDATE","cols":[{"before":"1107986424536772609","name":"id","index":0,"after":"1107986424536772609"},{"before":"101","name":"package_id","index":1,"after":"101"},{"before":"1089704947936186369","name":"store_no","index":2,"after":"1089704947936186369"},{"before":"hubei0XS00000037","name":"order_code","index":3,"after":"hubei0XS00000037"},{"before":"","name":"erp_order_code","index":4,"after":""},{"before":"3","name":"distribution_mode","index":5,"after":"3"},{"before":{"isNull":true},"name":"out_stock_id","index":6,"after":{"isNull":true}},{"before":{"isNull":true},"name":"out_comment","index":7,"after":{"isNull":true}},{"before":"2019-06-25 11:35:46.000","name":"out_time","index":8,"after":"2019-06-25 11:35:46.000"},{"before":{"isNull":true},"name":"transport_code","index":9,"after":{"isNull":true}},{"before":"","name":"courier_company","index":10,"after":""},{"before":"1","name":"receive_status","index":11,"after":"1"},{"before":"系统2","name":"operator","index":12,"after":"系统1"},{"before":"2019-03-19 20:45:23.000","name":"create_at","index":13,"after":"2019-03-19 20:45:23.000"},{"before":"2020-03-19 11:05:14.340","name":"update_at","index":14,"after":"2020-03-19 11:05:19.733"},{"before":"0","name":"is_edit","index":15,"after":"0"},{"before":"0","name":"courier_other","index":16,"after":"0"},{"before":{"isNull":true},"name":"last_syn_time","index":17,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state","index":18,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json","index":19,"after":{"isNull":true}},{"before":{"isNull":true},"name":"shipper_code","index":20,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_error_json","index":21,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code2","index":22,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json2","index":23,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state2","index":24,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code3","index":25,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json3","index":26,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state3","index":27,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code4","index":28,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json4","index":29,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state4","index":30,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code5","index":31,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json5","index":32,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state5","index":33,"after":{"isNull":true}}],"table":"b2b_third.tb_package_main"},{"database":"b2b_third","type":"UPDATE","cols":[{"before":"1107986424536772609","name":"id","index":0,"after":"1107986424536772609"},{"before":"101","name":"package_id","index":1,"after":"101"},{"before":"1089704947936186369","name":"store_no","index":2,"after":"1089704947936186369"},{"before":"hubei0XS00000037","name":"order_code","index":3,"after":"hubei0XS00000037"},{"before":"","name":"erp_order_code","index":4,"after":""},{"before":"3","name":"distribution_mode","index":5,"after":"3"},{"before":{"isNull":true},"name":"out_stock_id","index":6,"after":{"isNull":true}},{"before":{"isNull":true},"name":"out_comment","index":7,"after":{"isNull":true}},{"before":"2019-06-25 11:35:46.000","name":"out_time","index":8,"after":"2019-06-25 11:35:46.000"},{"before":{"isNull":true},"name":"transport_code","index":9,"after":{"isNull":true}},{"before":"","name":"courier_company","index":10,"after":""},{"before":"1","name":"receive_status","index":11,"after":"1"},{"before":"系统1","name":"operator","index":12,"after":"系统2"},{"before":"2019-03-19 20:45:23.000","name":"create_at","index":13,"after":"2019-03-19 20:45:23.000"},{"before":"2020-03-19 11:05:19.733","name":"update_at","index":14,"after":"2020-03-19 11:05:19.753"},{"before":"0","name":"is_edit","index":15,"after":"0"},{"before":"0","name":"courier_other","index":16,"after":"0"},{"before":{"isNull":true},"name":"last_syn_time","index":17,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state","index":18,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json","index":19,"after":{"isNull":true}},{"before":{"isNull":true},"name":"shipper_code","index":20,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_error_json","index":21,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code2","index":22,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json2","index":23,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state2","index":24,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code3","index":25,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json3","index":26,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state3","index":27,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code4","index":28,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json4","index":29,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state4","index":30,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code5","index":31,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json5","index":32,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state5","index":33,"after":{"isNull":true}}],"table":"b2b_third.tb_package_main"}]}]
2020-03-19  11:05:45.129 [pool-20-thread-4] INFO  com.jzt.data.sync.datax.job.support.JobLogger - [scheme@LoadToRabbitMQTest-01[0]-437972000456048640] - [HermesExtract] 增量数据提取完成,数据量: 2
2020-03-19  11:05:45.130 [pool-20-thread-4] DEBUG org.springframework.amqp.rabbit.core.RabbitTemplate - Executing callback RabbitTemplate$$Lambda$187/686510857 on RabbitMQ Channel: Cached Rabbit Channel: PublisherCallbackChannelImpl: AMQChannel(amqp://admin@10.4.9.175:5672/,1), conn: Proxy@1a8b3d7d Shared Rabbit Connection: SimpleConnection@760e29c9 [delegate=amqp://admin@10.4.9.175:5672/, localPort= 61949]
2020-03-19  11:05:45.130 [pool-20-thread-4] DEBUG org.springframework.amqp.rabbit.core.RabbitTemplate - Publishing message (Body:'[{"transport_code3":null,"transport_code4":null,"transport_code5":null,"hermes_dml_operation":"UPDATE","store_no":"1089704947936186369","syn_res_json3":null,"syn_res_json2":null,"syn_res_json5":null,"package_id":"101","syn_res_json4":null,"operator":"系统1","courier_company":"","syn_logistic_state":null,"hermes_table":"b2b_third.tb_package_main","distribution_mode":"3","syn_logistic_state4":null,"syn_logistic_state3":null,"syn_res_error_json":null,"syn_logistic_state2":null,"id":"1107986424536772609","shipper_code":null,"out_comment":null,"transport_code":null,"transport_code2":null,"syn_logistic_state5":null,"out_time":"2019-06-25 11:35:46.000","syn_res_json":null,"erp_order_code":"","is_edit":"0","receive_status":"1","order_code":"hubei0XS00000037","last_syn_time":null,"out_stock_id":null,"courier_other":"0","update_at":"2020-03-19 11:05:19.733","create_at":"2019-03-19 20:45:23.000"},{"transport_code3":null,"transport_code4":null,"transport_code5":null,"hermes_dml_operation":"UPDATE","store_no":"1089704947936186369","syn_res_json3":null,"syn_res_json2":null,"syn_res_json5":null,"package_id":"101","syn_res_json4":null,"operator":"系统2","courier_company":"","syn_logistic_state":null,"hermes_table":"b2b_third.tb_package_main","distribution_mode":"3","syn_logistic_state4":null,"syn_logistic_state3":null,"syn_res_error_json":null,"syn_logistic_state2":null,"id":"1107986424536772609","shipper_code":null,"out_comment":null,"transport_code":null,"transport_code2":null,"syn_logistic_state5":null,"out_time":"2019-06-25 11:35:46.000","syn_res_json":null,"erp_order_code":"","is_edit":"0","receive_status":"1","order_code":"hubei0XS00000037","last_syn_time":null,"out_stock_id":null,"courier_other":"0","update_at":"2020-03-19 11:05:19.753","create_at":"2019-03-19 20:45:23.000"}]' MessageProperties [headers={__TypeId__=java.lang.String}, contentType=text/plain, contentEncoding=UTF-8, contentLength=0, deliveryMode=PERSISTENT, priority=0, deliveryTag=0])on exchange [lizw-test-load-exchange], routingKey = [lizw-test-load]
2020-03-19  11:05:45.130 [pool-20-thread-4] INFO  com.jzt.data.sync.datax.job.support.JobLogger - [scheme@LoadToRabbitMQTest-01[0]-437972000456048640] - [RabbitMQLoad] 发送LoadData到MQ成功,数据量: [2]
2020-03-19  11:05:45.130 [pool-20-thread-4] INFO  com.jzt.data.sync.datax.job.support.JobLogger - [scheme@LoadToRabbitMQTest-01[0]-437972000456048640] - [RabbitMQLoad] 发送LoadData到MQ完成,数据量: [2]
2020-03-19  11:05:45.130 [pool-20-thread-4] INFO  com.jzt.data.sync.datax.job.support.JobLogger - [scheme@LoadToRabbitMQTest-01[0]-437972000456048640] - [HermesExtract] 增量任务执行结果: accept, message=[{"size":2,"data":[{"database":"b2b_third","type":"UPDATE","cols":[{"before":"1107986424536772609","name":"id","index":0,"after":"1107986424536772609"},{"before":"101","name":"package_id","index":1,"after":"101"},{"before":"1089704947936186369","name":"store_no","index":2,"after":"1089704947936186369"},{"before":"hubei0XS00000037","name":"order_code","index":3,"after":"hubei0XS00000037"},{"before":"","name":"erp_order_code","index":4,"after":""},{"before":"3","name":"distribution_mode","index":5,"after":"3"},{"before":{"isNull":true},"name":"out_stock_id","index":6,"after":{"isNull":true}},{"before":{"isNull":true},"name":"out_comment","index":7,"after":{"isNull":true}},{"before":"2019-06-25 11:35:46.000","name":"out_time","index":8,"after":"2019-06-25 11:35:46.000"},{"before":{"isNull":true},"name":"transport_code","index":9,"after":{"isNull":true}},{"before":"","name":"courier_company","index":10,"after":""},{"before":"1","name":"receive_status","index":11,"after":"1"},{"before":"系统2","name":"operator","index":12,"after":"系统1"},{"before":"2019-03-19 20:45:23.000","name":"create_at","index":13,"after":"2019-03-19 20:45:23.000"},{"before":"2020-03-19 11:05:14.340","name":"update_at","index":14,"after":"2020-03-19 11:05:19.733"},{"before":"0","name":"is_edit","index":15,"after":"0"},{"before":"0","name":"courier_other","index":16,"after":"0"},{"before":{"isNull":true},"name":"last_syn_time","index":17,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state","index":18,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json","index":19,"after":{"isNull":true}},{"before":{"isNull":true},"name":"shipper_code","index":20,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_error_json","index":21,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code2","index":22,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json2","index":23,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state2","index":24,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code3","index":25,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json3","index":26,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state3","index":27,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code4","index":28,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json4","index":29,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state4","index":30,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code5","index":31,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json5","index":32,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state5","index":33,"after":{"isNull":true}}],"table":"b2b_third.tb_package_main"},{"database":"b2b_third","type":"UPDATE","cols":[{"before":"1107986424536772609","name":"id","index":0,"after":"1107986424536772609"},{"before":"101","name":"package_id","index":1,"after":"101"},{"before":"1089704947936186369","name":"store_no","index":2,"after":"1089704947936186369"},{"before":"hubei0XS00000037","name":"order_code","index":3,"after":"hubei0XS00000037"},{"before":"","name":"erp_order_code","index":4,"after":""},{"before":"3","name":"distribution_mode","index":5,"after":"3"},{"before":{"isNull":true},"name":"out_stock_id","index":6,"after":{"isNull":true}},{"before":{"isNull":true},"name":"out_comment","index":7,"after":{"isNull":true}},{"before":"2019-06-25 11:35:46.000","name":"out_time","index":8,"after":"2019-06-25 11:35:46.000"},{"before":{"isNull":true},"name":"transport_code","index":9,"after":{"isNull":true}},{"before":"","name":"courier_company","index":10,"after":""},{"before":"1","name":"receive_status","index":11,"after":"1"},{"before":"系统1","name":"operator","index":12,"after":"系统2"},{"before":"2019-03-19 20:45:23.000","name":"create_at","index":13,"after":"2019-03-19 20:45:23.000"},{"before":"2020-03-19 11:05:19.733","name":"update_at","index":14,"after":"2020-03-19 11:05:19.753"},{"before":"0","name":"is_edit","index":15,"after":"0"},{"before":"0","name":"courier_other","index":16,"after":"0"},{"before":{"isNull":true},"name":"last_syn_time","index":17,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state","index":18,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json","index":19,"after":{"isNull":true}},{"before":{"isNull":true},"name":"shipper_code","index":20,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_error_json","index":21,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code2","index":22,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json2","index":23,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state2","index":24,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code3","index":25,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json3","index":26,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state3","index":27,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code4","index":28,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json4","index":29,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state4","index":30,"after":{"isNull":true}},{"before":{"isNull":true},"name":"transport_code5","index":31,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_res_json5","index":32,"after":{"isNull":true}},{"before":{"isNull":true},"name":"syn_logistic_state5","index":33,"after":{"isNull":true}}],"table":"b2b_third.tb_package_main"}]}]
文档更新时间: 2020-03-19 11:13   作者:lizw