Cross-Device

source of information =

https://wiki.extranet.weborama.com/xdevice/start

Open

The Weborama XDevice solution has two important parts

1. xdevice data ingestion

2. upload support

In order to upload data like idfa instead appnexus id, we need add support to the existing uploader to recognize new options in the current data_transfer integration.

After this, we need xdevice data to be able to translate weborama ids to idfa and upload to the supported dsps.

For now, we collect anything from action on computer browser.

Edit

From Collect Frontend

We can collect data by ThirdParty, Exchange and WAMFactory services. Each service handle custom segments and we intercept this operation to ingest data.

In this example, we use the ThirdParty service ( d.A=tp ).

Here parameters we use :

• d.a = 123 for account_id

• g.xcrm = 123456789 for crm identifier

• g.ism = 1 if mobile

• g.did = 12345678987654321234567898765432 for device id

• g.dty = 1 for Apple device, 2 for Google device (if 0 or not present, we will try to use the User Agent to detect the device family type, however the request may be reject by xdevice daemon)

AFFICHE_W will be created by a translation of device id. It will begin by @ (because it's mobile).

This operation is not bijective (device id → AFFICHE_W).

We will push into RabbitMQ queue :

meta:device contains 3 values + 2 optional : device_type (by default : 0), device_id, user_agent, ip, 2 letter country code.

if the 2 letter country code is present, we will use it in the index “country”. if it is not present, we will use the ip to perform a geolocation query.

meta:idcrm contains 2 values : account_id, id_crm

full example

the entry will be rejected if:

1. we can't auto-detect the mobile id type from user agent in auto-detect mode 2. we can't detect the country from the ip - if it is present.

Edit

From CSV Files

We have a script to read CSV files and push data into RabbitMQ.

This script is : wdx-populate-queue-from-csvfile We can pass these parameters :

• infile *

• fields *

• account_id

• device_type

• max : number of lines pushed to queue

• separator : separator into csv file

• debug

• dry-run

• is mandatory

• File with weborama and idcrm

Push in RabbitMQ queue:

Debug mode :

And finally the result :

We will have 5 messages into RabbitMQ like this :

• File with idcrm and device_id

Push in RabbitMQ queue:

Debug mode :

And finally the result :

We will have 5 messages into RabbitMQ like this :

Edit

XDevice daemon

A daemon will pick some messages (provide by Collect or CRM File) from this queue to store into CouchBase.

• meta:idcrm

With account id, we can retrieve : account name and scope. Account name is used as a key. Scope will be used as a namespace.

• meta:device

If device type = 1 or 2, we affect to idfa or gaid the device id If device type = 0, we try to detect device type from user_agent We validate data for idfa and gaid

And finally we store data into CouchBase.

Examples :

Message 1 :

Create a new entry into CouchBase :

Message 2 :

Merge the past entry into CouchBase :

How to start this daemon ?

Edit

How to Search ?

We can use this script to search into CouchBase :

Parameters :

• p : provider

• i : identifier

• s : scope (to find the scope, you can check the accounts.xdevice_conf_id value and then the scope column in the table xdevice_confs)

It will return:

Edit

From CRM Files (first version)

[Deprecated - waiting p2p of RabbitMQ queue to be deleted]

the current version of xdevice support deterministic files with this kind of format

example: webo.axa.data

This is the most basic kind of file: a csv using | as separator with two fields: weborama and axa_idcrm, but can be any kind of data.

The header of this file contains the 'provider', and the rest of lines are 'identifiers'. For example, '–0m5@rSKJD-85' is one AFFICHE_W cookie who will be validate and coerced to one valid weborama id.

Important: we need identify the provider of the data, using some label. this label can contain one suffix _id but it is not mandatory.

For the current uploader integration, we need two specific providers: weborama and idfa. The rest is free to be named but should be unique and do not change.

example idfa.axa.dat

When we ingest this two files. will be possible link the weborama id '–0m5@rSKJD-' ( we remove the last two chars, coerce to real webo id ) to idfa AEBE52E7-03EE-455A-B3C4-E57283966239 using the axa_idcrm 991ea637768c911b372d688578cd61da as pivot.

Example of data ingestion:

To check data ingestion:

you can note the provider '4VRu5o09WHamwg4YxsEv0g', it is a hash of axa_idcrm and it is mandatory to store data in production couchbase ( located outside weborama ).

To ingest data from a different provider ( like La Poste ) we need use one extra parameters called to_hash

example:

To remember:

1. The script xdevice-load-file.pl came from the latest version of Weborama::CrossDevice and it is sensible to the WEBO_ENV var.

2. The script was made thinking in process axa data. To process general format of files we can use the command line options (see below ).

3. The option config mode is mandatory in the current version. You must specify 'compute' or the read_oly configuration will be used by default and nothing will be processed.

4. If the file does not have a header, please specify by –fields a,b,c

5. If the file has a header but it is wrong, you can specify the fields + ask to skip_first_line

6. By default we will not use the header. Please force it with –use_header option

7. The Debug option is important to understand what is happening, if we need show mode info, please open a redmine ticket

Edit

Uploader Integration

XDevice is generic. The focus now is use idfa instead appnexus id and to do this we need

1. Create one data_transfer with delivering_type 'mobile' ( the default is cookie )

2. The uploader, by default, will process only data_transfers with cookies

3. There was one special destination, XDeviceAppNexus who can process data_transfers with mobile.

4. Without any data_transfer, this destination will just skip data

5. If there is any data, the uploader will: translate segments and push to rabbitmq, in the queue mobile-mobile_appnexus_add on vhost wam the pairs “weborama_id”, “segments”

6. There is one new daemon will consume this queue, translate webo id to idfa and upload. This should run in google cloud `daemon-euw1-zb-00.cross-device.gce.out.weborama.fr`

Edit

Daemon

on google cloud, start/stop the monit conf `wdx-uploader-appnexus-add-xdevice`

this will start the daemon `wdx-uploader-daemon` with section `xdevice_appnexus`

Logs

The worker in google should write on /var/log/daemon.log

one line like this:

means we upload data to appnexus and this is the id of the job. To check te status of this job please do this ( please consider the WEBO_ENV and use the real job_id ):

The uploader should print something like this on aub-daemon-01

This means there is no data to upload.

Edit

ELK Metrics

uplaoder send elk metrics. the worker in google cloud no.

Edit

HLL Metrics

the package Weborama::DataExchange::Tools expose this script:

each successful upload in the worker will increase the all_uploaded on xdevice-appnexus, each user skipped by no idfa will increase the no_ext_id. In case of any failure, we will increase the failure counter.

total is global. unmapped is a basic calculation from total - sum of all other fields. this save space in hll metrics.

Edit

Other info

• In case of any trouble in the uploader, consider stop this specific destination in the command line

• Remove will be supported in one next version

• AlexisD is working on the monit conf of the xdevice uploader worker in the cloud

• AppNexus only allow us upload once per minute. We will store files in the cloud in /data/uploader/appnexus until we can upload ( using the distributed lock )

• The data ingestion should be done inside weborama and store in Couchbase (located in the cloud)

• Bruno should provide the La Poste files to ingest