{"id":1719,"date":"2026-02-04T23:10:05","date_gmt":"2026-02-04T23:10:05","guid":{"rendered":"https:\/\/craftcookcode.com\/?p=1719"},"modified":"2026-02-04T23:10:08","modified_gmt":"2026-02-04T23:10:08","slug":"now-qlik-replicate-and-mongodb-are-stuck-inside-containers","status":"publish","type":"post","link":"https:\/\/craftcookcode.com\/?p=1719","title":{"rendered":"Now Qlik Replicate and MongoDB are stuck inside containers"},"content":{"rendered":"\n

That message from my Manager<\/h2>\n\n\n\n

I got a message from my manager unexpectedly.<\/p>\n\n\n\n

\n

\u201cHey. A team is converting a MSSQL database to MongoDB.  Can Qlik Replicate capture deletes from the MongoDB?\u201d<\/p>\n<\/blockquote>\n\n\n\n

I thought this was an abnormal question; why wouldn\u2019t QR support deletes?<\/p>\n\n\n\n

Not wanting to make an assumption that may be false and have profound consequences later in a conversion project, I looked up the Qlik documentation. <\/p>\n\n\n\n

Nothing in the documented \u201cLimitations and considerations.\u201d<\/a><\/p>\n\n\n\n

Just as a safeguard I also did a google search and a forum search to see if anything else turned up.<\/p>\n\n\n\n

Again nothing.<\/p>\n\n\n\n

I pinged back to my manager.<\/p>\n\n\n\n

\n

\u201cQR should capture deletes simply fine.  Where did you hear that QR had problems with MongoDB deletes from?\u201d<\/p>\n<\/blockquote>\n\n\n\n

\n

\u201cOh.  The project manager heard from a Mongo guy that Qlik does not work.\u201d<\/p>\n<\/blockquote>\n\n\n\n

I bit my lip.  To me that seemed the same as getting health advice off a random guy at the pub who saw something on Telegram.<\/p>\n\n\n\n

But I realised now that even though I had official Qlik documentation backing me up; the burden of proof was back on me.<\/p>\n\n\n\n

Back into the world of containers<\/h2>\n\n\n\n

Qlik Replicate in a docker container<\/a> has been an extremely useful tool on my Linux development machine.\u00a0 It enables me to quickly test various aspects of QR without interrupting the official dev environment; or fighting with the cloud team to get a proof-of-concept source system set up.<\/p>\n\n\n\n

First, I had to set up a MongoDB docker container.\u00a0 Specifically, the MongoDB had to have a \u201creplica\u201d; a standalone wouldn\u2019t work. Since this was a POC; I didn’t need a fully fledge – multi replica – ultra secure cluster; just something simple and disposable.<\/p>\n\n\n\n

After fumbling around with a number of examples (Gemeni \u2013 how could you fail me!); I came across a page by \u201cGink\u201d titled \u201cHow to set up MongoDB with replica set via Docker Compose?<\/a>\u201d<\/p>\n\n\n\n

It was simple and had exactly what I needed.\u00a0<\/p>\n\n\n\n

Dockerfile:<\/strong><\/p>\n\n\n

\nFROM mongo:7.0\nRUN openssl rand -base64 756 > \/etc\/mongo-keyfile \nRUN chmod 400 \/etc\/mongo-keyfile \nRUN chown mongodb:mongodb \/etc\/mongo-keyfile \n<\/pre><\/div>\n\n\n
docker-compose.yml:<\/strong><\/p>\n\n\n
\n# From https:\/\/ginkcode.com\/post\/how-to-set-up-mongodb-with-replica-set-via-docker-compose\nversion: '3.8'\n \nservices:\n  mongo:\n    build:\n      context: .\n      dockerfile: Dockerfile\n    container_name: mongodb-replicaset\n    restart: always\n    environment:\n      MONGO_INITDB_ROOT_USERNAME: root\n      MONGO_INITDB_ROOT_PASSWORD: root\n    ports:\n      - 27017:27017\n    command: --replSet rs0 --keyFile \/etc\/mongo-keyfile --bind_ip_all --port 27017\n    healthcheck:\n      test: echo "try { rs.status() } catch (err) { rs.initiate({_id:'rs0',members:[{_id:0,host:'127.0.0.1:27017'}]}) }" | mongosh --port 27017 -u root -p root --authenticationDatabase admin\n      interval: 5s\n      timeout: 15s\n      start_period: 15s\n      retries: 10\n    volumes:\n      - data:\/data\/db\n \nvolumes:\n  data: {}\n<\/pre><\/div>\n\n\n
Connection string<\/strong><\/p>\n\n\n
\nmongodb:\/\/root:root@localhost:27017\/?authSource=admin\n<\/pre><\/div>\n\n\n
Please visit Gink’s page <\/p>\n\n\n\n
Connecting to Qlik Replicate<\/h3>\n\n\n\n
The great thing is that we don’t have to mess around installing extra ODBC drivers that can make the Qlik Replicate docker image complex.<\/p>\n\n\n\n
So with the MongoDB and the Qlik Replicate docker containers up and running; we can how create a new endpoint with the following settings:<\/p>\n\n\n\n
Field<\/strong><\/td>Variable<\/strong><\/td><\/tr>
MongoDB Deployment<\/td>Standard<\/td><\/tr>
Hosts<\/td>host.docker.internal<\/td><\/tr>
Authentication Method<\/td>SCRAM-SHA-256<\/td><\/tr>
Username<\/td>root<\/td><\/tr>
Password<\/td>root<\/td><\/tr>
Authentication database name<\/td>admin<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
A test connection confirmed that the connection was working as expected.<\/p>\n\n\n\n
Proving that deletes are working<\/h2>\n\n\n\n
Getting close to resolving the burden of proof.\u00a0 I got our GenAI to create a simple python script to add, update, and delete some data from the Mongo Database and set it running:<\/p>\n\n\n
\nimport pymongo\nimport random\nimport string\nimport time\n\ndef generate_random_string(length=10):\n    """Generate a random string of fixed length."""\n    letters = string.ascii_lowercase\n    return ''.join(random.choice(letters) for i in range(length))\n\ndef add_random_data(collection):\n    """Adds a new document with random data to the collection."""\n    \n    new_document = {\n        "name": generate_random_string(),\n        "age": random.randint(18, 99),\n        "email": f"{generate_random_string(5)}@example.com"\n    }\n    \n    result = collection.insert_one(new_document)\n\n    print(f"Added: {result.inserted_id}")\n\ndef update_random_data(collection):\n    """Updates a random document in the collection."""\n    if collection.count_documents({}) > 0:\n        # Get a random document using the $sample aggregation operator\n        try:\n            random_doc_cursor = collection.aggregate([{"$sample": {"size": 1}}])\n            random_doc = next(random_doc_cursor)\n            new_age = random.randint(18, 99)\n            \n            collection.update_one(\n                {"_id": random_doc["_id"]},\n                {"$set": {"age": new_age}}\n            )\n\n            print(f"Updated: {random_doc['_id']} with new age {new_age}")\n        except StopIteration:\n            print("Could not find a document to update.")\n    else:\n        print("No documents to update.")\n\ndef delete_random_data(collection):\n    """Deletes a random document from the collection."""\n    if collection.count_documents({}) > 0:\n        # Get a random document using the $sample aggregation operator\n        try:\n            random_doc_cursor = collection.aggregate([{"$sample": {"size": 1}}])\n            random_doc = next(random_doc_cursor)\n\n            collection.delete_one({"_id": random_doc["_id"]})\n            \n            print(f"Deleted: {random_doc['_id']}")\n        except StopIteration:\n            print("Could not find a document to delete.")\n    else:\n        print("No documents to delete.")\n\n\nif __name__ == "__main__":\n    print("Starting script... Press Ctrl+C to stop.")\n\n    try:\n        client = pymongo.MongoClient("mongodb:\/\/root:root@localhost:27017\/?authSource=admin")\n        db = client["random_data_db"]\n        collection = db["my_collection"]\n        # The ismaster command is cheap and does not require auth.\n        client.admin.command('ismaster')\n        print("MongoDB connection successful.")\n    except pymongo.errors.ConnectionFailure as e:\n        print(f"Could not connect to MongoDB: {e}")\n        exit()\n\n    while True:\n        try:\n            add_random_data(collection)\n            add_random_data(collection)\n            add_random_data(collection)\n\n            update_random_data(collection)\n            delete_random_data(collection)\n            \n            # Wait for five seconds before the next cycle\n            time.sleep(5)\n\n        except KeyboardInterrupt:\n            print("\\nScript stopped by user.")\n            break\n        except Exception as e:\n            print(f"An error occurred: {e}")\n            break\n\n    # Close the connection\n    client.close()\n    print("MongoDB connection closed.")\n\n<\/pre><\/div>\n\n\n
With the script running and adding, updating and deleting data to the MongoDB; a QR task can be created to read from the database “random_data_db” and collection “my_collection”<\/p>\n\n\n\n
And voil\u00e0. Data is flowing through Qlik Replicate from the MongoDB.<\/p>\n\n\n\n
And more importantly deletes are getting picked up as expected.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
 <\/p>\n\n\n
\n{\n\t"header__change_seq":"20260204052516000000000000000004085",\n\t"header__change_oper":"D","header__change_mask":"80",\n\t"header__stream_position":"6982d83c:00000005:00000000",\n\t"header__operation":"DELETE",\n\t"header__transaction_id":"4155544F434F4D4D4954000000000000",\n\t"header__timestamp":"2026-02-04 05:25:16.000000",\n\t"_id":"6982d8186b2cc5ee1a161d13",\n\t"_doc":"{\\"_id\\": {\\"$oid\\": \\"6982d8186b2cc5ee1a161d13\\"}}"\n}\n \n<\/pre><\/div>\n\n\n
Conclusion <\/h2>\n\n\n\n
And voil\u00e0. Data is flowing through Qlik Replicate from the MongoDB.<\/p>\n\n\n\n
And more importantly; deletes are getting picked up as expected.<\/p>\n\n\n\n
If you have come to my humble website after googling \u201cCan Qlik Replicate replicate deletes from a MongoDB?\u201d well, the answer is \u201cYes \u2013 Yes it can.\u201d<\/p>\n\n\n\n
I can only speculate where that rumour came from.  Maybe it was from an older version of QR or MongoDB that the person in question was referring to?  Or some very abnormal setup of a MongoDB cluster?<\/p>\n\n\n\n
Or some complete misunderstanding.<\/p>\n\n\n\n
Anyway, I have meeting in the next hour or two with the project team; let us find out.<\/p>\n","protected":false},"excerpt":{"rendered":"
That message from my Manager I got a message from my manager unexpectedly. \u201cHey. A team is converting a MSSQL database to MongoDB.  Can Qlik Replicate...<\/p>\n","protected":false},"author":1,"featured_media":1720,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[103,16],"tags":[104,36],"class_list":["post-1719","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-mongodb","category-qlik-replicate","tag-mongodb","tag-qlikreplicate"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts\/1719","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1719"}],"version-history":[{"count":7,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts\/1719\/revisions"}],"predecessor-version":[{"id":1729,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts\/1719\/revisions\/1729"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/media\/1720"}],"wp:attachment":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1719"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1719"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1719"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}