)]}'
{"id":"lttng-tools~6821","triplet_id":"lttng-tools~master~Ied43cc8229413b25e69b40ac218adad420f18797","project":"lttng-tools","branch":"master","topic":"try-fix-relayd-destroy6","attention_set":{},"removed_from_attention_set":{"1000009":{"account":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"last_update":"2021-12-09 15:24:30.000000000","reason":"Mathieu Desnoyers replied on the change"},"1000008":{"account":{"_account_id":1000008,"name":"Jonathan Rajotte Julien","email":"jonathan.rajotte-julien@efficios.com","username":"PSRCode","avatars":[{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},"last_update":"2022-01-11 00:09:17.000000000","reason":"Change was submitted"},"1000006":{"account":{"_account_id":1000006,"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","username":"jgalar","avatars":[{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"last_update":"2022-01-11 00:09:17.000000000","reason":"Change was submitted"},"1000004":{"account":{"_account_id":1000004,"name":"Francis Deslauriers","email":"francis.deslauriers@efficios.com","username":"frdeso","avatars":[{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},"last_update":"2021-12-02 20:32:42.000000000","reason":"Francis Deslauriers replied on the change"}},"hashtags":[],"change_id":"Ied43cc8229413b25e69b40ac218adad420f18797","subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","status":"MERGED","created":"2021-11-22 19:39:25.000000000","updated":"2022-01-11 00:09:17.000000000","submitted":"2022-01-11 00:09:17.000000000","submitter":{"_account_id":1000006,"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","username":"jgalar","avatars":[{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"total_comment_count":2,"unresolved_comment_count":0,"has_review_started":true,"submission_id":"6759-try-fix-relayd-destroy6","meta_rev_id":"e477b14bad1cb950b6f06305bab0512228bf40cc","_number":6821,"virtual_id_number":6821,"owner":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"actions":{},"labels":{"Code-Review":{"all":[{"value":0,"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},{"value":0,"_account_id":1000008,"name":"Jonathan Rajotte Julien","email":"jonathan.rajotte-julien@efficios.com","username":"PSRCode","avatars":[{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"value":0,"_account_id":1000006,"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","username":"jgalar","avatars":[{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},{"value":0,"_account_id":1000004,"name":"Francis Deslauriers","email":"francis.deslauriers@efficios.com","username":"frdeso","avatars":[{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"value":0,"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]}],"values":{"-2":"This shall not be merged","-1":"I would prefer this is not merged as is"," 0":"No score","+1":"Looks good to me, but someone else must approve","+2":"Looks good to me, approved"},"description":"","default_value":0,"optional":true},"Verified":{"all":[{"value":0,"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},{"value":0,"_account_id":1000008,"name":"Jonathan Rajotte Julien","email":"jonathan.rajotte-julien@efficios.com","username":"PSRCode","avatars":[{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"value":0,"_account_id":1000006,"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","username":"jgalar","avatars":[{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},{"value":0,"_account_id":1000004,"name":"Francis Deslauriers","email":"francis.deslauriers@efficios.com","username":"frdeso","avatars":[{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"value":0,"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]}],"values":{"-2":"Failure","-1":"Not built"," 0":"No score","+1":"Unstable","+2":"Success"},"description":"CI Build results","default_value":0,"optional":true},"CI-Build":{"all":[{"value":0,"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},{"value":0,"_account_id":1000008,"name":"Jonathan Rajotte Julien","email":"jonathan.rajotte-julien@efficios.com","username":"PSRCode","avatars":[{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"value":0,"_account_id":1000006,"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","username":"jgalar","avatars":[{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},{"value":0,"_account_id":1000004,"name":"Francis Deslauriers","email":"francis.deslauriers@efficios.com","username":"frdeso","avatars":[{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"value":0,"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]}],"values":{" 0":"No score","+1":"Trigger a CI Build only","+2":"Trigger a CI Build with Fast Tests","+3":"Trigger a CI Build with Complete Tests"},"description":"Trigger CI builds","default_value":0,"optional":true},"Smoke-Build-Lvl1":{"all":[{"value":0,"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},{"value":0,"_account_id":1000008,"name":"Jonathan Rajotte Julien","email":"jonathan.rajotte-julien@efficios.com","username":"PSRCode","avatars":[{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"value":0,"_account_id":1000006,"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","username":"jgalar","avatars":[{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},{"value":0,"_account_id":1000004,"name":"Francis Deslauriers","email":"francis.deslauriers@efficios.com","username":"frdeso","avatars":[{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"value":0,"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]}],"default_value":0},"Smoke-Build-Lvl2":{"all":[{"value":0,"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},{"value":0,"_account_id":1000008,"name":"Jonathan Rajotte Julien","email":"jonathan.rajotte-julien@efficios.com","username":"PSRCode","avatars":[{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"value":0,"_account_id":1000006,"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","username":"jgalar","avatars":[{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},{"value":0,"_account_id":1000004,"name":"Francis Deslauriers","email":"francis.deslauriers@efficios.com","username":"frdeso","avatars":[{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"value":0,"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]}],"default_value":0}},"removable_reviewers":[],"reviewers":{"REVIEWER":[{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},{"_account_id":1000004,"name":"Francis Deslauriers","email":"francis.deslauriers@efficios.com","username":"frdeso","avatars":[{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"_account_id":1000006,"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","username":"jgalar","avatars":[{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},{"_account_id":1000008,"name":"Jonathan Rajotte Julien","email":"jonathan.rajotte-julien@efficios.com","username":"PSRCode","avatars":[{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]}]},"pending_reviewers":{},"reviewer_updates":[{"updated":"2021-11-22 20:05:49.000000000","updated_by":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"reviewer":{"_account_id":1000004,"name":"Francis Deslauriers","email":"francis.deslauriers@efficios.com","username":"frdeso","avatars":[{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},"state":"REVIEWER"},{"updated":"2021-11-22 20:05:49.000000000","updated_by":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"reviewer":{"_account_id":1000006,"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","username":"jgalar","avatars":[{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"state":"REVIEWER"},{"updated":"2021-11-22 20:05:57.000000000","updated_by":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"reviewer":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"state":"CC"},{"updated":"2021-11-22 20:26:29.000000000","updated_by":{"_account_id":1000008,"name":"Jonathan Rajotte Julien","email":"jonathan.rajotte-julien@efficios.com","username":"PSRCode","avatars":[{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},"reviewer":{"_account_id":1000008,"name":"Jonathan Rajotte Julien","email":"jonathan.rajotte-julien@efficios.com","username":"PSRCode","avatars":[{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},"state":"REVIEWER"},{"updated":"2021-11-22 21:06:00.000000000","updated_by":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"reviewer":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"state":"REVIEWER"}],"messages":[{"id":"252a9ec6730708608146356e2309f7a1f4aa2f0d","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-11-22 19:39:25.000000000","message":"Uploaded patch set 1.","accounts_in_message":[],"_revision_number":1},{"id":"c827eb7eea7a33ae37eac320981315b892036d3a","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-11-22 20:04:30.000000000","message":"Uploaded patch set 2: Commit message was updated.","accounts_in_message":[],"_revision_number":2},{"id":"314792b45f9da7c1d0722c4bcd09ce85470a9b34","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-11-22 20:05:49.000000000","message":"Patch Set 2: CI-Build+1","accounts_in_message":[],"_revision_number":2},{"id":"b911fe894d9ce22fedab72c92a5c32edc399fb32","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-11-22 20:05:57.000000000","message":"Patch Set 2:\n\nBuild Started https://ci.lttng.org/job/dev_gerrit_lttng-tools_rootbuild/1736/ (1/2)","accounts_in_message":[],"_revision_number":2},{"id":"707f3857dfbc608e53b44c935d946ab2a4f86d0d","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-11-22 20:05:57.000000000","message":"Patch Set 2:\n\nBuild Started https://ci.lttng.org/job/dev_gerrit_lttng-tools_build/2794/ (2/2)","accounts_in_message":[],"_revision_number":2},{"id":"b73991bef0298318b8926f1254ea003bf23976ae","author":{"_account_id":1000008,"name":"Jonathan Rajotte Julien","email":"jonathan.rajotte-julien@efficios.com","username":"PSRCode","avatars":[{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/f4101e071636faa234915893668fa03c.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},"date":"2021-11-22 20:26:29.000000000","message":"Patch Set 2: Code-Review-1\n\n(1 comment)","accounts_in_message":[],"_revision_number":2},{"id":"400a32889d161cd3c0e9cfe98c0c7fa33a61096a","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-11-22 20:57:26.000000000","message":"Uploaded patch set 3: Commit message was updated.","accounts_in_message":[],"_revision_number":3},{"id":"93e6ad1e0c70f3de46c7aad4cf7af4808062c40d","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-11-22 20:57:38.000000000","message":"Patch Set 2: CI-Build+1\n\n(1 comment)","accounts_in_message":[],"_revision_number":2},{"id":"67aa114ff59829a7fdb604faec44cf8048d17d2c","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-11-22 20:57:44.000000000","message":"Patch Set 2:\n\nBuild Started https://ci.lttng.org/job/dev_gerrit_lttng-tools_rootbuild/1738/ (1/2)","accounts_in_message":[],"_revision_number":2},{"id":"643d612276ad2d50e110e51d4c47cf40eb1e628a","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-11-22 20:57:45.000000000","message":"Patch Set 2:\n\nBuild Started https://ci.lttng.org/job/dev_gerrit_lttng-tools_build/2796/ (2/2)","accounts_in_message":[],"_revision_number":2},{"id":"473552a913921e79a23832115bdf06df9aeb1b3b","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-11-22 21:06:00.000000000","message":"Patch Set 2: Verified+2\n\nBuild Successful \n\nhttps://ci.lttng.org/job/dev_gerrit_lttng-tools_rootbuild/1736/ : SUCCESS\n\nhttps://ci.lttng.org/job/dev_gerrit_lttng-tools_build/2794/ : SUCCESS","accounts_in_message":[],"_revision_number":2},{"id":"bf64e8b62d17721cc0632d707e8438f5b862158c","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-11-22 21:14:05.000000000","message":"Uploaded patch set 4: Commit message was updated.","accounts_in_message":[],"_revision_number":4},{"id":"f0c8d0a99a94d34789c4b9ca5b14e6ec0d71c892","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-11-22 22:00:03.000000000","message":"Uploaded patch set 5: Commit message was updated.","accounts_in_message":[],"_revision_number":5},{"id":"31e39362e9bc4558e2ecb9f5071ee131a58ec2fa","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-11-22 22:04:45.000000000","message":"Patch Set 2: Verified+2\n\nBuild Successful \n\nhttps://ci.lttng.org/job/dev_gerrit_lttng-tools_rootbuild/1738/ : SUCCESS\n\nhttps://ci.lttng.org/job/dev_gerrit_lttng-tools_build/2796/ : SUCCESS","accounts_in_message":[],"_revision_number":2},{"id":"91d6ed530adc86e701e0d42d597c4dd652f424cc","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-12-01 16:43:41.000000000","message":"Uploaded patch set 6: New patch set was added with same tree, parent, and commit message as Patch Set 5.","accounts_in_message":[],"_revision_number":6},{"id":"e4e64b2acfe38a19a9683488c0ae3f9285703753","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-12-01 16:46:25.000000000","message":"Uploaded patch set 7: New patch set was added with same tree, parent, and commit message as Patch Set 6.","accounts_in_message":[],"_revision_number":7},{"id":"83c912d0e8fc2c46e5e7d9ad159df00af2d4dac6","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-12-01 16:50:23.000000000","message":"Uploaded patch set 8: New patch set was added with same tree, parent, and commit message as Patch Set 7.","accounts_in_message":[],"_revision_number":8},{"id":"46d89bf28f20a4c301aeab7b5c4ba25419a6aa3b","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-12-02 20:14:16.000000000","message":"Uploaded patch set 9: New patch set was added with same tree, parent, and commit message as Patch Set 8.","accounts_in_message":[],"_revision_number":9},{"id":"ae3d0b5140b3e924e320f4922a53f8fdfa00ac9e","author":{"_account_id":1000004,"name":"Francis Deslauriers","email":"francis.deslauriers@efficios.com","username":"frdeso","avatars":[{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},"date":"2021-12-02 20:32:42.000000000","message":"Patch Set 9: Code-Review+1","accounts_in_message":[],"_revision_number":9},{"id":"9c59acf7a64c5ac8fff6fb238599badef3e9f2cf","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-12-02 20:46:55.000000000","message":"Uploaded patch set 10: Patch Set 9 was rebased.","accounts_in_message":[],"_revision_number":10},{"id":"bc4de5531505ef1d5bdd652d7de1b27abb4278d7","author":{"_account_id":1000004,"name":"Francis Deslauriers","email":"francis.deslauriers@efficios.com","username":"frdeso","avatars":[{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/2117a78c2e91edc19dd421827dc2362e.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"inactive":true},"date":"2021-12-07 14:16:29.000000000","message":"Patch Set 10: Code-Review+1 CI-Build+1","accounts_in_message":[],"_revision_number":10},{"id":"bef711305594e03965cc8c5ce93069c5ac32b111","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-12-07 14:16:36.000000000","message":"Patch Set 10:\n\nBuild Started https://ci.lttng.org/job/dev_gerrit_lttng-tools_rootbuild/1754/ (1/2)","accounts_in_message":[],"_revision_number":10},{"id":"b63030dbeb9aad1bb9e31200fc18a3406504e245","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-12-07 14:16:36.000000000","message":"Patch Set 10:\n\nBuild Started https://ci.lttng.org/job/dev_gerrit_lttng-tools_build/2812/ (2/2)","accounts_in_message":[],"_revision_number":10},{"id":"c4809c037afeb7d7cdaf4e8d7ba4c41e0366fc87","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-12-07 17:10:17.000000000","message":"Patch Set 10: Verified-2\n\nBuild Failed \n\nhttps://ci.lttng.org/job/dev_gerrit_lttng-tools_rootbuild/1754/ : FAILURE\n\nhttps://ci.lttng.org/job/dev_gerrit_lttng-tools_build/2812/ : FAILURE","accounts_in_message":[],"_revision_number":10},{"id":"ba11eea2df56b8c44562efd8a74a6f558eda7207","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-12-07 18:30:41.000000000","message":"Uploaded patch set 11: New patch set was added with same tree, parent, and commit message as Patch Set 10.","accounts_in_message":[],"_revision_number":11},{"id":"7047042dfff9373880811ebc83b539c50c7dd653","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-12-08 19:30:23.000000000","message":"Uploaded patch set 12: Patch Set 11 was rebased.","accounts_in_message":[],"_revision_number":12},{"id":"040aeb93cf8d20edb5055a119732816a51a4cad9","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-12-08 19:31:24.000000000","message":"Patch Set 12: CI-Build+1","accounts_in_message":[],"_revision_number":12},{"id":"e62318024b81a83afcb8886ed96aa366a7917d3d","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-12-08 19:31:31.000000000","message":"Patch Set 12:\n\nBuild Started https://ci.lttng.org/job/dev_gerrit_lttng-tools_rootbuild/1759/ (1/2)","accounts_in_message":[],"_revision_number":12},{"id":"d35b48a5b7e05fab930fd8d9d643f1d9e92fd5da","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-12-08 19:31:31.000000000","message":"Patch Set 12:\n\nBuild Started https://ci.lttng.org/job/dev_gerrit_lttng-tools_build/2817/ (2/2)","accounts_in_message":[],"_revision_number":12},{"id":"b01eed3699eda4d9106b00ea4734a0aea3308941","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-12-08 21:00:04.000000000","message":"Patch Set 12: Verified-2\n\nBuild Failed \n\nhttps://ci.lttng.org/job/dev_gerrit_lttng-tools_rootbuild/1759/ : FAILURE\n\nhttps://ci.lttng.org/job/dev_gerrit_lttng-tools_build/2817/ : FAILURE","accounts_in_message":[],"_revision_number":12},{"id":"6d782a99f8750262a8511d034daa3978fe3f8a26","tag":"autogenerated:gerrit:newPatchSet","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-12-09 15:22:54.000000000","message":"Uploaded patch set 13: Patch Set 12 was rebased.","accounts_in_message":[],"_revision_number":13},{"id":"ec9bc72a0f2a02745b9bcce353dafb150d4e6076","author":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2021-12-09 15:24:30.000000000","message":"Patch Set 13: CI-Build+1","accounts_in_message":[],"_revision_number":13},{"id":"21f8df924db4c0ab39623dcd155ed116d159cfd4","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-12-09 15:24:37.000000000","message":"Patch Set 13:\n\nBuild Started https://ci.lttng.org/job/dev_gerrit_lttng-tools_build/2828/ (1/2)","accounts_in_message":[],"_revision_number":13},{"id":"f5bf23934e9464ec2c4d34863599b43018e9f95a","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-12-09 15:24:37.000000000","message":"Patch Set 13:\n\nBuild Started https://ci.lttng.org/job/dev_gerrit_lttng-tools_rootbuild/1770/ (2/2)","accounts_in_message":[],"_revision_number":13},{"id":"9d57feffeb1b0234d49a4bce0a542b4df7d7d548","tag":"autogenerated:jenkins-gerrit-trigger","author":{"_account_id":1000002,"name":"jenkins","email":"jenkins@lttng.org","username":"jenkins","avatars":[{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/e3f1da3d4191917309975c0380f40764.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}],"tags":["SERVICE_USER"]},"date":"2021-12-09 17:30:46.000000000","message":"Patch Set 13: Verified+2\n\nBuild Successful \n\nhttps://ci.lttng.org/job/dev_gerrit_lttng-tools_rootbuild/1770/ : SUCCESS\n\nhttps://ci.lttng.org/job/dev_gerrit_lttng-tools_build/2828/ : SUCCESS","accounts_in_message":[],"_revision_number":13},{"id":"e477b14bad1cb950b6f06305bab0512228bf40cc","tag":"autogenerated:gerrit:merged","author":{"_account_id":1000006,"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","username":"jgalar","avatars":[{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"date":"2022-01-11 00:09:17.000000000","message":"Change has been successfully pushed.","accounts_in_message":[],"_revision_number":14}],"current_revision_number":14,"current_revision":"6653fca60fdb6bf4ea4781bfee8304832d6a7e83","revisions":{"0fa5bb7f53eac20d2319539b862cbc4e5cd386db":{"kind":"REWORK","_number":1,"created":"2021-11-22 19:39:25.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/1","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/1","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/1 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/1 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/1 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/1 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/1","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/1 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"96c5151ea08c89348aadc7da11fbb72f0404e1b6","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d96c5151ea08c89348aadc7da11fbb72f0404e1b6"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d0fa5bb7f53eac20d2319539b862cbc4e5cd386db"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d0fa5bb7f53eac20d2319539b862cbc4e5cd386db"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"96c5151ea08c89348aadc7da11fbb72f0404e1b6","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":3,"change_status":"MERGED"}],"branch":"refs/heads/master"},"b6b38bdac506cbc7de8d289d726ff0c3a1867cd3":{"kind":"NO_CODE_CHANGE","_number":2,"created":"2021-11-22 20:04:30.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/2","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/2","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/2 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/2 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/2 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/2 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/2","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/2 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"96c5151ea08c89348aadc7da11fbb72f0404e1b6","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d96c5151ea08c89348aadc7da11fbb72f0404e1b6"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:49:29.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n1) The relayd main thread receives a live beacon for stream 994      (@ 17:26:56.907762798)\n2) The relayd main thread receives a data packet for stream 994      (@ 17:26:56.907855597)\n3) The relayd live thread informs the live viewer of the live beacon (@ 17:26:56.913457688)\n4) The relayd main thread receives an index packet for stream 994    (@ 17:26:56.914227502)\n5) The relayd live thread sends the packet to the viewer             (@ 17:26:57.742197990)\n   -\u003e the viewer observes time going backwards between the live beacon and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket can be anywhere on the network, not even received by the relay\ndaemon, when the live beacon is sampling a now empty ring buffer, and\nthus sends a live inactivity beacon to the relay daemon. Then, when the\nindex associated with the previously send data packet is sent, its\ntimestamp is in the past compared to the inactivity beacon.\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003db6b38bdac506cbc7de8d289d726ff0c3a1867cd3"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003db6b38bdac506cbc7de8d289d726ff0c3a1867cd3"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"96c5151ea08c89348aadc7da11fbb72f0404e1b6","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":3,"change_status":"MERGED"}],"branch":"refs/heads/master"},"6be02bbdcf5486b79280ab979961419433fbbccf":{"kind":"NO_CODE_CHANGE","_number":3,"created":"2021-11-22 20:57:26.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/3","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/3","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/3 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/3 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/3 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/3 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/3","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/3 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"96c5151ea08c89348aadc7da11fbb72f0404e1b6","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d96c5151ea08c89348aadc7da11fbb72f0404e1b6"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 20:41:00.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket can be anywhere on the network, not even received by the relay\ndaemon, when the live beacon is sampling a now empty ring buffer, and\nthus sends a live inactivity beacon to the relay daemon. Then, when the\nindex associated with the previously send data packet is sent, its\ntimestamp is in the past compared to the inactivity beacon.\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d6be02bbdcf5486b79280ab979961419433fbbccf"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d6be02bbdcf5486b79280ab979961419433fbbccf"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"96c5151ea08c89348aadc7da11fbb72f0404e1b6","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":3,"change_status":"MERGED"}],"branch":"refs/heads/master"},"98ddf8744be9a129894f078088b2f9167597431d":{"kind":"NO_CODE_CHANGE","_number":4,"created":"2021-11-22 21:14:05.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/4","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/4","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/4 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/4 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/4 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/4 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/4","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/4 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"96c5151ea08c89348aadc7da11fbb72f0404e1b6","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d96c5151ea08c89348aadc7da11fbb72f0404e1b6"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 21:08:20.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket sent at [3] can be anywhere on the network, not even received by\nthe relay daemon, when the live beacon is sampling a now empty ring\nbuffer at [12], and thus sends a live inactivity beacon to the relay\ndaemon. Then, when the index is sent by consumer_stream_send_index\n[16], its timestamp is in the past compared to the inactivity beacon\nsent by send_live_beacon [11].\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d98ddf8744be9a129894f078088b2f9167597431d"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d98ddf8744be9a129894f078088b2f9167597431d"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"96c5151ea08c89348aadc7da11fbb72f0404e1b6","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":3,"change_status":"MERGED"}],"branch":"refs/heads/master"},"f73aa2711f68aa41fd5d1db7f489954910973984":{"kind":"NO_CODE_CHANGE","_number":5,"created":"2021-11-22 22:00:03.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/5","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/5","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/5 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/5 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/5 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/5 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/5","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/5 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"96c5151ea08c89348aadc7da11fbb72f0404e1b6","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d96c5151ea08c89348aadc7da11fbb72f0404e1b6"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 21:52:26.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket sent at [3] can be anywhere on the network, not even received by\nthe relay daemon, when the live beacon is sampling a now empty ring\nbuffer at [12], and thus sends a live inactivity beacon to the relay\ndaemon. Then, when the index is sent by consumer_stream_send_index\n[16], its timestamp is in the past compared to the inactivity beacon\nsent by send_live_beacon [11].\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nHistory\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nThis issue is present since lttng-tools 2.7.0 (backported to 2.6.1),\nwhere lttng_ustconsumer_read_subbuffer() invokes\nconsumer_flush_ust_index() prior to call consumer_stream_write_index().\nIt was introduced by commit 288bdb302a1 (\"Fix: sessiond vs consumerd\npush/get metadata deadlock\").\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003df73aa2711f68aa41fd5d1db7f489954910973984"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003df73aa2711f68aa41fd5d1db7f489954910973984"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"96c5151ea08c89348aadc7da11fbb72f0404e1b6","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":3,"change_status":"MERGED"}],"branch":"refs/heads/master"},"005cc22fd3f2008ac0b8c07c2604caf4db8b2deb":{"kind":"NO_CHANGE","_number":6,"created":"2021-12-01 16:43:41.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/6","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/6","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/6 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/6 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/6 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/6 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/6","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/6 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"3c6d3f4b4572dc2ccd5ecdd758dbe8a0e350ad57","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d3c6d3f4b4572dc2ccd5ecdd758dbe8a0e350ad57"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-12-01 16:43:36.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket sent at [3] can be anywhere on the network, not even received by\nthe relay daemon, when the live beacon is sampling a now empty ring\nbuffer at [12], and thus sends a live inactivity beacon to the relay\ndaemon. Then, when the index is sent by consumer_stream_send_index\n[16], its timestamp is in the past compared to the inactivity beacon\nsent by send_live_beacon [11].\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nHistory\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nThis issue is present since lttng-tools 2.7.0 (backported to 2.6.1),\nwhere lttng_ustconsumer_read_subbuffer() invokes\nconsumer_flush_ust_index() prior to call consumer_stream_write_index().\nIt was introduced by commit 288bdb302a1 (\"Fix: sessiond vs consumerd\npush/get metadata deadlock\").\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d005cc22fd3f2008ac0b8c07c2604caf4db8b2deb"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d005cc22fd3f2008ac0b8c07c2604caf4db8b2deb"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"3c6d3f4b4572dc2ccd5ecdd758dbe8a0e350ad57","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":4,"change_status":"MERGED"}],"branch":"refs/heads/master"},"23e8f995d3f930a90389ed7a23c57f105f75e8d8":{"kind":"NO_CHANGE","_number":7,"created":"2021-12-01 16:46:25.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/7","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/7","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/7 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/7 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/7 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/7 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/7","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/7 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"0c3c4157013ec3c98bad9c4db4d1cae121ddde9a","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d0c3c4157013ec3c98bad9c4db4d1cae121ddde9a"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-12-01 16:46:22.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket sent at [3] can be anywhere on the network, not even received by\nthe relay daemon, when the live beacon is sampling a now empty ring\nbuffer at [12], and thus sends a live inactivity beacon to the relay\ndaemon. Then, when the index is sent by consumer_stream_send_index\n[16], its timestamp is in the past compared to the inactivity beacon\nsent by send_live_beacon [11].\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nHistory\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nThis issue is present since lttng-tools 2.7.0 (backported to 2.6.1),\nwhere lttng_ustconsumer_read_subbuffer() invokes\nconsumer_flush_ust_index() prior to call consumer_stream_write_index().\nIt was introduced by commit 288bdb302a1 (\"Fix: sessiond vs consumerd\npush/get metadata deadlock\").\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d23e8f995d3f930a90389ed7a23c57f105f75e8d8"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d23e8f995d3f930a90389ed7a23c57f105f75e8d8"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"0c3c4157013ec3c98bad9c4db4d1cae121ddde9a","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":5,"change_status":"MERGED"}],"branch":"refs/heads/master"},"f708a8f6bad291db254ae0ab64ea1ea94be80862":{"kind":"NO_CHANGE","_number":8,"created":"2021-12-01 16:50:23.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/8","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/8","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/8 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/8 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/8 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/8 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/8","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/8 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"4c0197f95f741ee6d7c8e268a994b5fa37890a22","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d4c0197f95f741ee6d7c8e268a994b5fa37890a22"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-12-01 16:49:54.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket sent at [3] can be anywhere on the network, not even received by\nthe relay daemon, when the live beacon is sampling a now empty ring\nbuffer at [12], and thus sends a live inactivity beacon to the relay\ndaemon. Then, when the index is sent by consumer_stream_send_index\n[16], its timestamp is in the past compared to the inactivity beacon\nsent by send_live_beacon [11].\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nHistory\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nThis issue is present since lttng-tools 2.7.0 (backported to 2.6.1),\nwhere lttng_ustconsumer_read_subbuffer() invokes\nconsumer_flush_ust_index() prior to call consumer_stream_write_index().\nIt was introduced by commit 288bdb302a1 (\"Fix: sessiond vs consumerd\npush/get metadata deadlock\").\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003df708a8f6bad291db254ae0ab64ea1ea94be80862"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003df708a8f6bad291db254ae0ab64ea1ea94be80862"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"4c0197f95f741ee6d7c8e268a994b5fa37890a22","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":6,"change_status":"MERGED"}],"branch":"refs/heads/master"},"8c06f2c7f6cff541da5a2b8eefbb0d72e6fa9ca1":{"kind":"NO_CHANGE","_number":9,"created":"2021-12-02 20:14:16.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/9","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/9","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/9 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/9 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/9 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/9 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/9","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/9 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"13da26edf5f1f550cf2f09f90d3fc7b0b2942c6a","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d13da26edf5f1f550cf2f09f90d3fc7b0b2942c6a"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-12-02 20:14:11.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket sent at [3] can be anywhere on the network, not even received by\nthe relay daemon, when the live beacon is sampling a now empty ring\nbuffer at [12], and thus sends a live inactivity beacon to the relay\ndaemon. Then, when the index is sent by consumer_stream_send_index\n[16], its timestamp is in the past compared to the inactivity beacon\nsent by send_live_beacon [11].\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nHistory\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nThis issue is present since lttng-tools 2.7.0 (backported to 2.6.1),\nwhere lttng_ustconsumer_read_subbuffer() invokes\nconsumer_flush_ust_index() prior to call consumer_stream_write_index().\nIt was introduced by commit 288bdb302a1 (\"Fix: sessiond vs consumerd\npush/get metadata deadlock\").\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d8c06f2c7f6cff541da5a2b8eefbb0d72e6fa9ca1"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d8c06f2c7f6cff541da5a2b8eefbb0d72e6fa9ca1"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"13da26edf5f1f550cf2f09f90d3fc7b0b2942c6a","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":7,"change_status":"MERGED"}],"branch":"refs/heads/master"},"7904039545b199ad4f6566a47337ad7e400be9bc":{"kind":"TRIVIAL_REBASE","_number":10,"created":"2021-12-02 20:46:55.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/10","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/10","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/10 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/10 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/10 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/10 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/10","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/10 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"f6ffb1b164d79c4af20abd3321a98fe738128074","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003df6ffb1b164d79c4af20abd3321a98fe738128074"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-12-02 20:46:35.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket sent at [3] can be anywhere on the network, not even received by\nthe relay daemon, when the live beacon is sampling a now empty ring\nbuffer at [12], and thus sends a live inactivity beacon to the relay\ndaemon. Then, when the index is sent by consumer_stream_send_index\n[16], its timestamp is in the past compared to the inactivity beacon\nsent by send_live_beacon [11].\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nHistory\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nThis issue is present since lttng-tools 2.7.0 (backported to 2.6.1),\nwhere lttng_ustconsumer_read_subbuffer() invokes\nconsumer_flush_ust_index() prior to call consumer_stream_write_index().\nIt was introduced by commit 288bdb302a1 (\"Fix: sessiond vs consumerd\npush/get metadata deadlock\").\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d7904039545b199ad4f6566a47337ad7e400be9bc"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d7904039545b199ad4f6566a47337ad7e400be9bc"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"f6ffb1b164d79c4af20abd3321a98fe738128074","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":8,"change_status":"MERGED"}],"branch":"refs/heads/master"},"477072b8e08b6f03ec5a51b6118545ee9e0d4afe":{"kind":"NO_CHANGE","_number":11,"created":"2021-12-07 18:30:41.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/11","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/11","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/11 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/11 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/11 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/11 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/11","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/11 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"0ad7cc497b48cc83c37fc5f6624b126d551925f5","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d0ad7cc497b48cc83c37fc5f6624b126d551925f5"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-12-07 18:30:17.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket sent at [3] can be anywhere on the network, not even received by\nthe relay daemon, when the live beacon is sampling a now empty ring\nbuffer at [12], and thus sends a live inactivity beacon to the relay\ndaemon. Then, when the index is sent by consumer_stream_send_index\n[16], its timestamp is in the past compared to the inactivity beacon\nsent by send_live_beacon [11].\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nHistory\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nThis issue is present since lttng-tools 2.7.0 (backported to 2.6.1),\nwhere lttng_ustconsumer_read_subbuffer() invokes\nconsumer_flush_ust_index() prior to call consumer_stream_write_index().\nIt was introduced by commit 288bdb302a1 (\"Fix: sessiond vs consumerd\npush/get metadata deadlock\").\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d477072b8e08b6f03ec5a51b6118545ee9e0d4afe"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d477072b8e08b6f03ec5a51b6118545ee9e0d4afe"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"0ad7cc497b48cc83c37fc5f6624b126d551925f5","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":9,"change_status":"MERGED"}],"branch":"refs/heads/master"},"df0c07a6049dca668e35b59b7c23a74f7f36dced":{"kind":"TRIVIAL_REBASE","_number":12,"created":"2021-12-08 19:30:23.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/12","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/12","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/12 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/12 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/12 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/12 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/12","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/12 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"241861e695f7e9d58d813d7c130080845f1e4e72","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d241861e695f7e9d58d813d7c130080845f1e4e72"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-12-08 19:26:18.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket sent at [3] can be anywhere on the network, not even received by\nthe relay daemon, when the live beacon is sampling a now empty ring\nbuffer at [12], and thus sends a live inactivity beacon to the relay\ndaemon. Then, when the index is sent by consumer_stream_send_index\n[16], its timestamp is in the past compared to the inactivity beacon\nsent by send_live_beacon [11].\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nHistory\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nThis issue is present since lttng-tools 2.7.0 (backported to 2.6.1),\nwhere lttng_ustconsumer_read_subbuffer() invokes\nconsumer_flush_ust_index() prior to call consumer_stream_write_index().\nIt was introduced by commit 288bdb302a1 (\"Fix: sessiond vs consumerd\npush/get metadata deadlock\").\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003ddf0c07a6049dca668e35b59b7c23a74f7f36dced"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003ddf0c07a6049dca668e35b59b7c23a74f7f36dced"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"241861e695f7e9d58d813d7c130080845f1e4e72","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":10,"change_status":"MERGED"}],"branch":"refs/heads/master"},"04035d029efcbbe923273fd69a99b5ec13aec691":{"kind":"TRIVIAL_REBASE","_number":13,"created":"2021-12-09 15:22:54.000000000","uploader":{"_account_id":1000009,"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","username":"compudj","avatars":[{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/7326b3e1d32b971a5e69d9730228c428.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/13","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/13","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/13 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/13 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/13 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/13 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/13","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/13 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"d1e954e33e325bfaa6f837d0d34530617e0545b5","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003dd1e954e33e325bfaa6f837d0d34530617e0545b5"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-12-09 15:22:36.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket sent at [3] can be anywhere on the network, not even received by\nthe relay daemon, when the live beacon is sampling a now empty ring\nbuffer at [12], and thus sends a live inactivity beacon to the relay\ndaemon. Then, when the index is sent by consumer_stream_send_index\n[16], its timestamp is in the past compared to the inactivity beacon\nsent by send_live_beacon [11].\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nHistory\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nThis issue is present since lttng-tools 2.7.0 (backported to 2.6.1),\nwhere lttng_ustconsumer_read_subbuffer() invokes\nconsumer_flush_ust_index() prior to call consumer_stream_write_index().\nIt was introduced by commit 288bdb302a1 (\"Fix: sessiond vs consumerd\npush/get metadata deadlock\").\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d04035d029efcbbe923273fd69a99b5ec13aec691"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d04035d029efcbbe923273fd69a99b5ec13aec691"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"d1e954e33e325bfaa6f837d0d34530617e0545b5","is_merged_in_target_branch":false,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":11,"change_status":"MERGED"}],"branch":"refs/heads/master"},"6653fca60fdb6bf4ea4781bfee8304832d6a7e83":{"kind":"TRIVIAL_REBASE_WITH_MESSAGE_UPDATE","_number":14,"created":"2022-01-11 00:09:17.000000000","uploader":{"_account_id":1000006,"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","username":"jgalar","avatars":[{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d32","height":32},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d56","height":56},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d100","height":100},{"url":"https://www.gravatar.com/avatar/8689763327e5bbda7dec9f18846b60ae.jpg?d\u003dretro\u0026r\u003dr\u0026s\u003d120","height":120}]},"ref":"refs/changes/21/6821/14","fetch":{"anonymous http":{"url":"https://review.lttng.org/lttng-tools","ref":"refs/changes/21/6821/14","commands":{"Branch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/14 \u0026\u0026 git checkout -b change-6821 FETCH_HEAD","Checkout":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/14 \u0026\u0026 git checkout FETCH_HEAD","Cherry Pick":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/14 \u0026\u0026 git cherry-pick FETCH_HEAD","Format Patch":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/14 \u0026\u0026 git format-patch -1 --stdout FETCH_HEAD","Pull":"git pull https://review.lttng.org/lttng-tools refs/changes/21/6821/14","Reset To":"git fetch https://review.lttng.org/lttng-tools refs/changes/21/6821/14 \u0026\u0026 git reset --hard FETCH_HEAD"}}},"commit":{"parents":[{"commit":"b9973dea9fc1eaf88ef997561878e45cdfd369b4","subject":"Fix: relayd: ressource leaks on viewer_stream_create error","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003db9973dea9fc1eaf88ef997561878e45cdfd369b4"}]}],"author":{"name":"Mathieu Desnoyers","email":"mathieu.desnoyers@efficios.com","date":"2021-11-22 19:38:35.000000000","tz":-300},"committer":{"name":"Jérémie Galarneau","email":"jeremie.galarneau@efficios.com","date":"2022-01-10 23:21:30.000000000","tz":-300},"subject":"Fix: consumer-stream: live viewers observe timestamps going backwards","message":"Fix: consumer-stream: live viewers observe timestamps going backwards\n\nObserved issue\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nWhen stress-testing over an entire weekend, we caught the following\noccurrences of timestamps going backwards with babeltrace in live viewer\nmode:\n\nOne occurrence:\n\n  # Test ust streaming live clear with viewer with new metadata after clear\n  # Parameters: tracing_active\u003d0, clear_twice\u003d0, buffer_type\u003dpid\n\n6-7 occurrences:\n\n  # Test ust basic streaming live with viewer\n  # Parameters: tracing_active\u003d1, clear_twice\u003d0, buffer_type\u003duid\n\nRelevant log of the relayd focused on the context of the stream\ntriggering the issue (stream 994):\n\n506:DBG1 - 17:26:54.966288486 [3866648/3866655]: Relay viewer stream 994 not found (in viewer_stream_get_by_id() at viewer-stream.cpp:268)\n603:DBG1 - 17:26:54.970265349 [3866648/3866655]: Sending stream 994 to viewer (in send_viewer_streams() at live.cpp:241)\n843:DBG1 - 17:26:55.959835404 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n844:DBG3 - 17:26:55.959866303 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n845:DBG2 - 17:26:55.959896611 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 0 (in relay_index_create() at index.cpp:34)\n846:DBG2 - 17:26:55.959928097 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 0 (in relay_index_add_unique() at index.cpp:70)\n847:DBG2 - 17:26:55.959968163 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n884:DBG3 - 17:26:55.961676906 [3866648/3866652]: Receiving data for stream id 994 seqnum 0, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n885:DBG1 - 17:26:55.961783540 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n886:DBG1 - 17:26:55.961862441 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n887:DBG1 - 17:26:55.961895593 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 0 data offset 0 (in stream_update_index() at stream.cpp:1140)\n888:DBG3 - 17:26:55.961945901 [3866648/3866652]: Finding index for stream id 994 and seq_num 0 (in relay_index_get_by_id_or_create() at index.cpp:112)\n889:DBG2 - 17:26:55.961983056 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 0 (in relay_index_get_by_id_or_create() at index.cpp:144)\n894:DBG2 - 17:26:55.962334285 [3866648/3866652]: Writing index for stream ID 994 and seq num 0 (in relay_index_try_flush() at index.cpp:275)\n895:DBG2 - 17:26:55.962390756 [3866648/3866652]: index put for stream id 994 and seqnum 0 refcount 1 (in relay_index_put() at index.cpp:237)\n1743:DBG1 - 17:26:56.083287172 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 0 for stream 994 (in check_index_status() at live.cpp:1454)\n1746:DBG1 - 17:26:56.083446379 [3866648/3866655]: Sending viewer index for stream 994 offset 0 (in viewer_get_next_index() at live.cpp:1801)\n1748:DBG1 - 17:26:56.083544877 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n1751:DBG1 - 17:26:56.083778149 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n2858:DBG1 - 17:26:56.907744916 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n2859:DBG1 - 17:26:56.907762798 [3866648/3866652]: Received live beacon for stream 994 (in stream_add_index() at stream.cpp:1237)\n2862:DBG3 - 17:26:56.907855597 [3866648/3866652]: Receiving data for stream id 994 seqnum 1, 0 bytes received, 84 bytes left to receive (in relay_process_data_receive_payload() at main.cpp:3583)\n2863:DBG1 - 17:26:56.907950195 [3866648/3866652]: Wrote to stream 994: data_length \u003d 84, padding_length \u003d 0 (in stream_write() at stream.cpp:1113)\n2864:DBG1 - 17:26:56.908002911 [3866648/3866652]: Wrote to stream 994: data_length \u003d 0, padding_length \u003d 4012 (in stream_write() at stream.cpp:1113)\n2865:DBG1 - 17:26:56.908024312 [3866648/3866652]: handle_index_data: stream 994 net_seq_num 1 data offset 4096 (in stream_update_index() at stream.cpp:1140)\n2866:DBG3 - 17:26:56.908043082 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n2867:DBG2 - 17:26:56.908061879 [3866648/3866652]: Creating relay index for stream id 994 and seqnum 1 (in relay_index_create() at index.cpp:34)\n2868:DBG2 - 17:26:56.908082115 [3866648/3866652]: Adding relay index with stream id 994 and seqnum 1 (in relay_index_add_unique() at index.cpp:70)\n2869:DBG2 - 17:26:56.908101275 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3011:DBG1 - 17:26:56.913436908 [3866648/3866655]: Check index status: index_received_seqcount 1 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n3012:DBG1 - 17:26:56.913457688 [3866648/3866655]: Check index status: inactive with beacon, for stream 994 (in check_index_status() at live.cpp:1492)\n3014:DBG1 - 17:26:56.913507164 [3866648/3866655]: Index 1 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n3043:DBG1 - 17:26:56.914167206 [3866648/3866652]: stream_add_index for stream 994 (in stream_add_index() at stream.cpp:1233)\n3044:DBG3 - 17:26:56.914186324 [3866648/3866652]: Finding index for stream id 994 and seq_num 1 (in relay_index_get_by_id_or_create() at index.cpp:112)\n3045:DBG2 - 17:26:56.914205597 [3866648/3866652]: Index found or created in HT for stream ID 994 and seqnum 1 (in relay_index_get_by_id_or_create() at index.cpp:144)\n3046:DBG2 - 17:26:56.914227502 [3866648/3866652]: Writing index for stream ID 994 and seq num 1 (in relay_index_try_flush() at index.cpp:275)\n3047:DBG2 - 17:26:56.914299536 [3866648/3866652]: index put for stream id 994 and seqnum 1 refcount 1 (in relay_index_put() at index.cpp:237)\n3587:DBG1 - 17:26:57.067977800 [3866648/3866652]: Set begin data pending flag to stream 994 (in relay_begin_data_pending() at main.cpp:2240)\n3644:DBG1 - 17:26:57.070915787 [3866648/3866652]: Data pending for stream id 994: prev_data_seq 1, prev_index_seq 1, and last_seq 1 (in relay_data_pending() at main.cpp:2091)\n3913:DBG1 - 17:26:57.093492259 [3866648/3866652]: try_rotate_stream_index: Stream 994 (rotate_at_packet_seq_num \u003d 2, received_packet_seq_num \u003d (value \u003d 1, is_set \u003d 1)) (in try_rotate_stream_index() at stream.cpp:482)\n3914:DBG1 - 17:26:57.093525950 [3866648/3866652]: Rotating stream 994 index file (in try_rotate_stream_index() at stream.cpp:511)\n3915:DBG1 - 17:26:57.093561064 [3866648/3866652]: try_rotate_stream_data: Stream 994 (rotate_at_index_packet_seq_num \u003d 2, rotate_at_prev_data_net_seq \u003d 1, prev_data_seq \u003d 1) (in try_rotate_stream_data() at stream.cpp:357)\n3916:DBG1 - 17:26:57.093591085 [3866648/3866652]: Rotating stream 994 data file with size 8192 (in stream_rotate_data_file() at stream.cpp:138)\n3917:DBG1 - 17:26:57.093626697 [3866648/3866652]: stream_rotate_data_file: reset tracefile_size_current for stream 994 was 8192 (in stream_rotate_data_file() at stream.cpp:169)\n3918:DBG1 - 17:26:57.093656578 [3866648/3866652]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n4238:DBG1 - 17:26:57.635064782 [3866648/3866652]: Trying to close stream 994 (in try_stream_close() at stream.cpp:883)\n4239:DBG1 - 17:26:57.635098224 [3866648/3866652]: Succeeded in closing stream 994 (in try_stream_close() at stream.cpp:983)\n4744:DBG1 - 17:26:57.741972785 [3866648/3866655]: Check index status: index_received_seqcount 2 index_sent_seqcount 1 for stream 994 (in check_index_status() at live.cpp:1454)\n4745:DBG1 - 17:26:57.742030216 [3866648/3866655]: Sending viewer index for stream 994 offset 4096 (in viewer_get_next_index() at live.cpp:1801)\n4747:DBG1 - 17:26:57.742088421 [3866648/3866655]: Index 2 for stream 994 sent (in viewer_get_next_index() at live.cpp:1842)\n4750:DBG1 - 17:26:57.742197990 [3866648/3866655]: Sent 4108 bytes for stream 994 (in viewer_get_packet() at live.cpp:1968)\n4755:DBG1 - 17:26:57.932525633 [3866648/3866655]: Releasing stream id 994 (in stream_release() at stream.cpp:778)\n4756:DBG1 - 17:26:57.932555313 [3866648/3866655]: Rotation completed for stream 994 (in stream_complete_rotation() at stream.cpp:66)\n\nCause\n\u003d\u003d\u003d\u003d\u003d\n\nThis log shows the following sequence:\n\n   consumer-data                       consumer-live-timer                     relayd-main                     relayd-live               babeltrace\n\n1) lttng_consumer_read_subbuffer\n  2) get next subbuf\n  3) write data packet to data socket\n     (starting here the data packet is\n     somewhere on the network)\n  4) put next subbuf\n  5) post_consume()\n    6) consumer_stream_sync_metadata_index()\n      7) wait for metadata\n         8) consumer_stream_sync_metadata()\n                                       9) check_stream()\n                                       10) set missed_metadata_flush\n      11) call send_live_beacon(()\n        12) sample empty ring buffer\n        13) read current timestamp\n        14) send inactivity beacon (empty packet)\n                                                                               15) receives a live beacon                                                     (@ 17:26:56.907762798)\n      16) call consumer_stream_send_index()\n         17) send packet index to relayd\n                                                                               18) receives a data packet                                                     (@ 17:26:56.907855597)\n                                                                                  (at this point the data\n                                                                                   packet is received from the\n                                                                                   network)\n                                                                                                                                         19) ask for next index\n                                                                                                               20) informs the live viewer of the live beacon (@ 17:26:56.913457688)\n                                                                               21) receives an index packet                                                   (@ 17:26:56.914227502)\n                                                                                                                                         22) ask for next index\n                                                                                                               23) sends the packet to the viewer             (@ 17:26:57.742197990)\n                                                                                                                                         24) observes time going\n                                                                                                                                             backwards between the\n                                                                                                                                             previous live beacon\n                                                                                                                                             and the data packet.\n\nThe issue is caused by consumer_stream_sync_metadata_index which is\ncalled after sending a data packet (therefore after having consumed a\ndata packet from the ring buffer). It invokes the send_live_beacon\ncallback before sending the index associated with the data packet that\nwas just sent.\n\nHowever, this introduces a discrepancy between the live beacon\ninactivity guarantees and the yet-to-be-sent packet index: the data\npacket sent at [3] can be anywhere on the network, not even received by\nthe relay daemon, when the live beacon is sampling a now empty ring\nbuffer at [12], and thus sends a live inactivity beacon to the relay\ndaemon. Then, when the index is sent by consumer_stream_send_index\n[16], its timestamp is in the past compared to the inactivity beacon\nsent by send_live_beacon [11].\n\nThe purpose of the field \"stream-\u003eindexes_in_flight\" is to prevent\nsetting the inactivity timestamp in the relay stream when data is\nmissing for indexes that were received. This works because the indexes\nare sent over the control socket, which is where the inactivity beacons\nare also sent. It does not however prevent issues the other way around:\ndata sent prior to the inactivity beacon may or may not have reached\nthe relay daemon. It is therefore important to make sure that consuming\nring buffer data and sending that data\u0027s index vs sampling for an empty\nring buffer and sending an inactivity beacon are correctly ordered.\n\nSolution\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nSend inactivity beacon after packet index.\n\nAlso document the purpose of sending an inactivity beacon in this\nscenario.\n\nNote\n\u003d\u003d\u003d\u003d\n\nThis issue is present since lttng-tools 2.7.0 (backported to 2.6.1),\nwhere lttng_ustconsumer_read_subbuffer() invokes\nconsumer_flush_ust_index() prior to call consumer_stream_write_index().\nIt was introduced by commit 288bdb302a1 (\"Fix: sessiond vs consumerd\npush/get metadata deadlock\").\n\nKnown drawbacks\n\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\u003d\n\nNone.\n\nSigned-off-by: Mathieu Desnoyers \u003cmathieu.desnoyers@efficios.com\u003e\nSigned-off-by: Jérémie Galarneau \u003cjeremie.galarneau@efficios.com\u003e\nChange-Id: Ied43cc8229413b25e69b40ac218adad420f18797\n","web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d6653fca60fdb6bf4ea4781bfee8304832d6a7e83"}],"resolve_conflicts_web_links":[{"name":"gitweb","tooltip":"Open in GitWeb","url":"/gitweb?p\u003dlttng-tools.git;a\u003dcommit;h\u003d6653fca60fdb6bf4ea4781bfee8304832d6a7e83"}]},"parents_data":[{"branch_name":"refs/heads/master","commit_id":"b9973dea9fc1eaf88ef997561878e45cdfd369b4","is_merged_in_target_branch":true,"change_id":"I864b58eda94ebda5e6faea45c922d4e814a15daa","change_number":6789,"patch_set_number":12,"change_status":"MERGED"}],"branch":"refs/heads/master"}},"requirements":[],"submit_records":[],"submit_requirements":[]}