{"id":1507,"date":"2024-09-04T01:46:23","date_gmt":"2024-09-04T01:46:23","guid":{"rendered":"https:\/\/craftcookcode.com\/?p=1507"},"modified":"2024-09-04T04:58:00","modified_gmt":"2024-09-04T04:58:00","slug":"qlik-replicate-youre-trapped-in-a-docker-container-now","status":"publish","type":"post","link":"https:\/\/craftcookcode.com\/?p=1507","title":{"rendered":"Qlik Replicate: You’re trapped in a Docker container now!"},"content":{"rendered":"\n

In Qlik Replicate we tasks unable to resume when we have nasty server failures (for instant the CrowdStrike outage in July 2024).<\/p>\n\n\n\n

This only happens in tasks that are impacted are a RDBMS to a cloud storage system like AWS S3 or GCS. <\/p>\n\n\n\n

In the task log the error message takes the form of:<\/p>\n\n\n

\n00002396: 2022-08-26T15:21:14 [AT_GLOBAL ]E: Json doesn't start with '{' [1003001] (at_cjson.c:1773)\n00002396: 2022-08-26T15:21:14 [AT_GLOBAL ]E: Cannot parse json: [1000251] (at_protobuf.c:1420)\n<\/pre><\/div>\n\n\n
This error gives us problems; I can\u2019t resume the task as the error re-appears.  I can\u2019t even start it from the stream position and must rely on restarting the QR task from a timestamp, which is extremely dangerous with the chance of missing out on data for that split of a second.<\/p>\n\n\n\n
I suspect the problem is that the \u201cstaging\u201d file on the QR server gets corrupted mid write when the server fails and when resume; QR can\u2019t parse it.<\/p>\n\n\n\n
But trying to recreate the problem in a safe environment to diagnose it is tricky.  Our DTL environment doesn\u2019t create enough traffic to trigger the issue.  Also, I don\u2019t want to be abruptly turning off our DTL QR servers and interrupting other people\u2019s testing.  As for trying to recreate the problem in production \u2013 the pain of all the red tape is not worth the effort.<\/p>\n\n\n\n
I needed a safer space to work in.  A space when I can pump through large volumes of data through QR and kick the QR service around trying to provoke the error.  Armed with my little Linux VM \u2013 docker containers was the answer.<\/p>\n\n\n\n
<\/p>\n\n\n\n
CentOS? Why CentOS?<\/h2>\n\n\n\n
My goal was to build a Docker container with Qlik Replicate and Postgres drivers so I can use it on my Linux VM.<\/p>\n\n\n\n
Under Support articles, Qlik has a guide on how to run Qlik Replicate in a Docker container<\/a>.<\/p>\n\n\n\n
Following the instructions I ran into some initial problems.  The first major problem was using the Cent OS docker image.  The issue was that I must use the packages in my company\u2019s artifactory and not external packages.  Although the company had CentOS; there was no other packages available to update and install.  Since my VM cannot reach http:\/\/vault.centos.org; the CentOS image was a lame duck.<\/p>\n\n\n\n
With CentOS off the cards, I had to use Redhat image that my company provided.  With Redhat \u2013 the artifactory had all the packages that I needed.<\/p>\n\n\n\n
The second problem was that I was wanting to use the 2023.11 image to match our environment.\u00a0 With 2023.11 there are some extra steps needed in the docker file compared to 2024.05.\u00a0 The differences is notated on Qlik’s support article.<\/p>\n\n\n\n
The Dockerfile<\/h2>\n\n\n\n
Here is the Dockerfile <\/p>\n\n\n
\nFROM my.companys.repo\/redhat\/ubi9\n\n\nENV QLIK_REPLICATE_BASE_DIR=\/opt\/attunity\/replicate\/\nENV ReplicateDataFolder=\/replicate\/data\nENV ReplicateAdminPassword=AB1gL0ngPa33w0rd\nENV ReplicateRestPort=3552\nENV LicenseFile=\/tmp\/replicate_license_exp2025-06-29_ser60038556.txt\n\n# Copy across installation packages and licenses\nADD postgresql*.rpm \/tmp\/\nADD areplicate-*.rpm \/tmp\/\nADD systemctl \/usr\/sbin\nADD replicate_license_exp2025-06-29_ser60038556.txt \/tmp\/\n\n# Update packages\nRUN dnf -y update\nRUN dnf makecache\n\n# To get ps command\nRUN dnf -y install procps-ng\nRUN dnf -y install unixODBC unzip\nRUN dnf -y install libicu.x86_64\nRUN rm -f \/etc\/odbcinst.ini\n\n# Installing posgres packages\nRUN rpm -ivh \/tmp\/postgresql13-libs-13.9-1PGDG.rhel9.x86_64.rpm\nRUN rpm -ivh \/tmp\/postgresql13-odbc-13.02.0000-2PGDG.rhel9.x86_64.rpm\nRUN rpm -ivh \/tmp\/postgresql13-13.9-1PGDG.rhel9.x86_64.rpm\n\nADD odbcinst.ini \/etc\/\n\n# Installing Qlik Replicate\nRUN systemd=no yum -y install \/tmp\/areplicate-2023.11.0-468.x86_64.rpm\nRUN yum clean all\nRUN rm -f \/tmp\/areplicate-*.rpm\n\nRUN export LD_LIBRARY_PATH=\/opt\/attunity\/replicate\/lib:\\$LD_LIBRARY_PATH\nRUN echo "export LD_LIBRARY_PATH=\/usr\/pgsql-13\/lib:\\$LD_LIBRARY_PATH" >> \/opt\/attunity\/replicate\/bin\/site_arep_login.sh\n\nADD start_replicate.sh \/opt\/attunity\/replicate\/bin\/start_replicate.sh\nRUN chmod 775 \/opt\/attunity\/replicate\/bin\/start_replicate.sh\nRUN chown attunity:attunity \/opt\/attunity\/replicate\/bin\/start_replicate.sh\nRUN source $QLIK_REPLICATE_BASE_DIR\/bin\/arep_login.sh >>~attunity\/.bash_profile\nENTRYPOINT \/opt\/attunity\/replicate\/bin\/start_replicate.sh ${ReplicateDataFolder} ${ReplicateAdminPassword} ${ReplicateRestPort} ${LicenseFile} ; tail -f \/dev\/null\n<\/pre><\/div>\n\n\n
The postgres packages can be obtained from https:\/\/download.postgresql.org\/pub\/repos\/yum\/13\/redhat\/rhel-9-x86_64\/<\/p>\n\n\n\n
Th file odbcinst.ini content is:<\/p>\n\n\n
\n[PostgreSQL]\nDescription = ODBC for PostgreSQL\nDriver      = \/usr\/lib\/psqlodbcw.so\nSetup       = \/usr\/lib\/libodbcpsqlS.so\nDriver64    = \/usr\/pgsql-13\/lib\/psqlodbcw.so\nSetup64     = \/usr\/lib64\/libodbcpsqlS.so\nFileUsage   = 1\n\n<\/pre><\/div>\n\n\n
The systemctl file is:<\/p>\n\n\n
\n# Run LS command - remove this line \nls\n<\/pre><\/div>\n\n\n
And of course you need the rpm for Qlik replicate and your license file.<\/p>\n\n\n\n
Once the Dockerfile and files are collated in a directory; build the container with:<\/p>\n\n\n
\ndocker build --no-cache -t ccc\/replicate:2023.11 .\n<\/pre><\/div>\n\n\n
If all goes well – a Docker contain will be built and ready to be used.<\/p>\n\n\n\n
Docker Compose<\/h2>\n\n\n\n
To make running the docker images easier; create a docker compose file:<\/p>\n\n\n
\nversion: '3.3'\n\nservices:\n  replicate:\n    image: docker.io\/ccc\/replicate:2023.11\n    container_name: replicate_2023_11\n    ports: \n      - "3552:3552"\n\n    environment:\n      - ReplicateRestPort=3552\n      - TZ=Australia\/Melbourne\n\n    volumes:\n      - \/dockermount\/data\/replicate\/data:\/replicate\/data\n\n    extra_hosts:\n      - host.docker.internal:host-gateway\n\nvolumes:\n  replicate:\n\n<\/pre><\/div>\n\n\n
Save the docker-compose.yml in a directory and from the directory start the container with the command:<\/p>\n\n\n
\ndocker-compose up -d\n<\/pre><\/div>\n\n\n
If everything is working – run the docker ps command to verify everything is working:<\/p>\n\n\n
\ndocker ps\n<\/pre><\/div>\n\n\n
\"\"<\/figure>\n\n\n\n
<\/div>\n\n\n\n
So far looking good.  Further conformation can be had by connecting into the container and observe the QR processes running:<\/p>\n\n\n
\ndocker exec -it qr_container_id bash\nps -aux\n<\/pre><\/div>\n\n\n
There should be two main processes; plus a process for each individual QR tasks running:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
<\/div>\n\n\n\n
With everything confirmed – QR console can be accessed from a browser.<\/p>\n\n\n
\nhttps:\/\/127.0.0.1:3552\/attunityreplicate\/\n<\/pre><\/div>\n\n\n
\"\"<\/figure>\n","protected":false},"excerpt":{"rendered":"
In Qlik Replicate we tasks unable to resume when we have nasty server failures (for instant the CrowdStrike outage in July 2024). This only happens in...<\/p>\n","protected":false},"author":1,"featured_media":1429,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[68,64,16],"tags":[65,56,36],"class_list":["post-1507","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-docker","category-postgres","category-qlik-replicate","tag-docker","tag-postgres","tag-qlikreplicate"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts\/1507","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1507"}],"version-history":[{"count":13,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts\/1507\/revisions"}],"predecessor-version":[{"id":1523,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts\/1507\/revisions\/1523"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/media\/1429"}],"wp:attachment":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1507"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1507"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1507"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}