{"id":1402,"date":"2024-05-15T00:49:26","date_gmt":"2024-05-15T00:49:26","guid":{"rendered":"https:\/\/craftcookcode.com\/?p=1402"},"modified":"2024-05-15T00:49:26","modified_gmt":"2024-05-15T00:49:26","slug":"qlik-replicate-the-saga-of-replicating-to-aws-part-4-does-stream-size-matter","status":"publish","type":"post","link":"https:\/\/craftcookcode.com\/?p=1402","title":{"rendered":"Qlik Replicate \u2013 The saga of replicating to AWS Part 4 \u2013 Does stream size matter?"},"content":{"rendered":"\n

Continuing our ongoing Qlik Replicating story of trying to replicate a DB2\/zOS database to AWS RDS Postgres.<\/p>\n\n\n\n

We made small improvements; but nothing substantial to reach the TPS for which we were aiming.  I was at my experience end of what I knew and decided to reach for professional help.<\/p>\n\n\n\n

We have a support relationship with IBT<\/a>; who helped us out with the initial set up QR in our organisation.\u00a0 But recently we have been self-resolving our own problems and have not been using their help.\u00a0 Now this was suitable time to ask for their help.<\/p>\n\n\n\n

IBT has always been helpful when we have asked for assistance.  Another handy aspect with the relationship is that IBT has a quick support relationship with Qlik.  If they don\u2019t know the answer; they can get the answer easily from Qlik.<\/p>\n\n\n\n

IBT asked us to collect the usual data; diagnostic logs, source and target DB metrics and QR server core metrics.  Nothing looked under duress, so IBT dived into the nitty and gritty details of the diagnostic packs.<\/p>\n\n\n\n

Their techs noticed that our outgoing stream buffers were full.  This means that the changes were coming in faster than were getting sent out to the destination.  IBT suggested to try increasing the size of the outgoing stream.<\/p>\n\n\n\n

Without going through the details of this step; here is a Qlik knowledge base article on Increasing the outgoing stream queue<\/a>.<\/p>\n\n\n\n

“She’s breaking up – I can’t hold her”<\/h2>\n\n\n\n

We started off with:<\/p>\n\n\n

\n"stream_buffers_number" : 5,\n"stream_buffer_size" : 10,\n\n<\/pre><\/div>\n\n\n
It was a marginal improvement.  Measured in a thimble full of performance improvement.  Still nowhere near the TPS we needed. <\/p>\n\n\n\n
IBT asked us to increase the two variables in small increments of \u201cstream_buffers_number\u201d + 5 and “stream_buffer_size” + 10.  With each increase there was a minuscule improvement.<\/p>\n\n\n\n
 But more worrying with each increase; the QR task was using more memory to the point that increasing the buffer size was unsustainable with the resources on the server.  Even if increasing the buffer variables and the gained TPS was linear relationship; we would need a very beefy server to reach 300 TPS.<\/p>\n\n\n\n
So again, it was a little gain; and with all the added \u201cLittle gains\u201d over the past few fix iterations we were still no closer to our needed 300 TPS.<\/p>\n\n\n\n
Increasing the buffer variables might be helpful if you are close to your TPS and trying to get over the last hurdle.  But since we\u2019re so far behind; we had to look for another solution.<\/p>\n","protected":false},"excerpt":{"rendered":"
Continuing our ongoing Qlik Replicating story of trying to replicate a DB2\/zOS database to AWS RDS Postgres. We made small improvements; but nothing substantial to reach...<\/p>\n","protected":false},"author":1,"featured_media":1247,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[16],"tags":[36],"class_list":["post-1402","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-qlik-replicate","tag-qlikreplicate"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts\/1402","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1402"}],"version-history":[{"count":1,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts\/1402\/revisions"}],"predecessor-version":[{"id":1403,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/posts\/1402\/revisions\/1403"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=\/wp\/v2\/media\/1247"}],"wp:attachment":[{"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1402"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1402"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/craftcookcode.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1402"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}