From 39cd40be02fb4e3393d6cf64295cddde2242b642 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?S=C3=B6nke=20Liebau?= Date: Thu, 14 Apr 2022 00:32:32 +0200 Subject: [PATCH 1/3] Add draft ADR on configmap/secret content access for operators. --- .../ARDx-using_values_from_configmaps.adoc | 89 +++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 modules/contributor/pages/adr/drafts/ARDx-using_values_from_configmaps.adoc diff --git a/modules/contributor/pages/adr/drafts/ARDx-using_values_from_configmaps.adoc b/modules/contributor/pages/adr/drafts/ARDx-using_values_from_configmaps.adoc new file mode 100644 index 000000000..74196c2d2 --- /dev/null +++ b/modules/contributor/pages/adr/drafts/ARDx-using_values_from_configmaps.adoc @@ -0,0 +1,89 @@ += How Should Operators Use Values from ConfigMaps & Secrets +Doc Writer +v0.1, 13.04.2022 +:status: draft + +* Status: {status} +* Deciders: +** Felix Hennig +** Razvan Mihai +** Sebastian Bernauer +** Sönke Liebau +** Teo Klestrup Röijezon +* Date: 13.04.2022 + +Technical Story: https://github.com/stackabletech/issues/issues/208[#208] + +== Context and Problem Statement + +One of the main focus areas of Stackable is making components of the stack work together well. +This requires a high degree of composability where one component refers to another compenent in its definition. +While this is a simple concept in principle it requires that operators obtain information about the object that is referred to and inject this information into the configuration of the object that refers to it. +An example for this is a NiFi cluster definition referring to a ZooKeeper cluster definition via a znode name: + +[,yaml] +---- +apiVersion: nifi.stackable.tech/v1alpha1 +kind: NifiCluster +metadata: + name: simple-nifi +spec: + zookeeperConfigMapName: simple-nifi-znode +---- + +For this example the NiFi operator needs to somehow insert the content of this config map into specific NiFi config files. + +This ADR is about the preferred way of implementing dependencies like described above. + +== Considered Options + +* <> Inserting ConfigMap/Secret content into target config in the operator +* <> Mounting ConfigMap/Secret into the pods + +== Decision Outcome + +Chosen option: "<>", because it makes better use of Kubernetes features to model dependencies between definitions which in turn will make restart implementations and general debugging easier and more predictable. + +This decision will not hold true for all possible scenarios, but should be understood more as a implementation guideline: + +[sidebar] +If a pod needs to access the content of a ConfigMap or Secret it should do this by mounting the ConfigMap/Secret unless there is a good reason to access the ConfigMap directly in the operator instead. + +=== Positive Consequences + +* Dependencies will be clearly modelled in Kubernetes by mounts, which makes behavior of the entire architecture more predictable +* Operators need to implement a much simpler logic around watches, as triggering reconciliations is handled via k8s dependencies (mounts) + +=== Negative Consequences + +* Complexity is moved from the operator code into init containers that need to perform insertion of the ConfigMap/Secret content into actual config files +* Debugging a running configuration becomes harder in cases where users have no access to a shell in the pods, as this would be the only way to actually see the running configuration (https://github.com/stackabletech/issues/issues/210)[#210] has been created to investigate mitigating this) + +== Pros and Cons of the Options + +[[option1]] +=== Option 1 - Inserting ConfigMap/Secret content into target config in the operator + +The operator can retrieve the content from ConfigMaps/Secrets during reconciliation and insert the values directly into the target config. +The target config would then be written either to a ConfigMap or a Secret, depending on whether sensitive input data is contained in it. + + +* Good, because it isolates most of the complexity in the operator +* Good, because it would allow referencing ConfigMaps and Secrets in different namespaces which would potentially allow for less duplication +* Good, because it gives an almost complete manifestation of a running config in k8s objects that can be used for debugging +* Bad, because it would allow referencing ConfigMaps and Secrets in different namespaces, which would dilute namespace separation +* Bad, because it makes debugging startup issues of products much harder, as pods will simply not appear if required ConfigMaps are missing, effectively bypassing well established k8s patterns (like a pod missing a mount not starting and writing events about why it is not starting) +* Bad, because the dependencies between pods and configmaps / secrets become very complex and hard to debug as they are not modelled in any way as k8s dependencies (mounts) +* Bad, because setting up the correct watches in the operator becomes very complex, with a likelyhood of inadvertently forgetting to add a new watch that becomes necessary as CRDs evolve +* Bad, because it potentially propagates sensitive values to multiple places by copying secret content to additional places + +[[option2]] +=== Option 2 - Mounting ConfigMap/Secret into the pods + +Instead of accessing the content of ConfigMaps/Secrets in the operator the operator would simply mount these object into the generated pods. +Accessing the content of these mounted objects would then need to be done in an init container, currently this is done with shell commands, but this would be replaced with a more suitable tool in a follow-up ADR. + +* Good, because [argument a] +* Good, because [argument b] +* Bad, because [argument c] +* … From e83290a35360df6a5bf152d971d0372f7a0f9323 Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Thu, 28 Apr 2022 10:08:26 +0200 Subject: [PATCH 2/3] Added some text --- .../adr/drafts/ARDx-using_values_from_configmaps.adoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/modules/contributor/pages/adr/drafts/ARDx-using_values_from_configmaps.adoc b/modules/contributor/pages/adr/drafts/ARDx-using_values_from_configmaps.adoc index 74196c2d2..d01ec108d 100644 --- a/modules/contributor/pages/adr/drafts/ARDx-using_values_from_configmaps.adoc +++ b/modules/contributor/pages/adr/drafts/ARDx-using_values_from_configmaps.adoc @@ -76,14 +76,14 @@ The target config would then be written either to a ConfigMap or a Secret, depen * Bad, because the dependencies between pods and configmaps / secrets become very complex and hard to debug as they are not modelled in any way as k8s dependencies (mounts) * Bad, because setting up the correct watches in the operator becomes very complex, with a likelyhood of inadvertently forgetting to add a new watch that becomes necessary as CRDs evolve * Bad, because it potentially propagates sensitive values to multiple places by copying secret content to additional places +* Bad, because the Operator is seeing the Secrets. This is not a problem _now_, but if we could avoid it that would be better [[option2]] === Option 2 - Mounting ConfigMap/Secret into the pods -Instead of accessing the content of ConfigMaps/Secrets in the operator the operator would simply mount these object into the generated pods. +Instead of accessing the content of ConfigMaps/Secrets in the operator, the operator would simply mount these object into the generated pods. Accessing the content of these mounted objects would then need to be done in an init container, currently this is done with shell commands, but this would be replaced with a more suitable tool in a follow-up ADR. -* Good, because [argument a] -* Good, because [argument b] -* Bad, because [argument c] -* … +* Good, because mounting the ConfigMap/Secret directly into the pod allows the restart controller to automatically restart a Pod if a mounted object changes. Otherwise, changes would not be propagated or the Operator would have to watch the ConfigMap/Secret for changes itself. The watching/restarting is done once by Kubernetes + the restart controller and makes our Operators simpler +* Bad, because the mounted properties cannot be validated by the Operator. Although Kubernetes at least verifies that all the properties that should be mounted exist +* Bad, because it is more difficult to see which config is actually used to run the product, as the actual config is only finally assembled inside of the container \ No newline at end of file From 1f4be850129e1c3e2e1b13dca5204882c7003081 Mon Sep 17 00:00:00 2001 From: Felix Hennig Date: Thu, 28 Apr 2022 10:15:05 +0200 Subject: [PATCH 3/3] gave the ADR the number 14, added it to the nav --- .../pages/adr/ADR001-choose_project_language.adoc | 2 +- .../pages/adr/ADR002-choose_repository_structure.adoc | 2 +- .../pages/adr/ADR003-choose_review_mechanism.adoc | 2 +- .../adr/ADR004-choose_agent_programming_language.adoc | 2 +- .../pages/adr/ADR005-systemd_unit_file_location.adoc | 2 +- .../pages/adr/ADR007-defined_reuse_of_k8s.adoc | 2 +- .../pages/adr/ADR008-decide_reuse_of_operators.adoc | 2 +- ...s.adoc => ARD014-using_values_from_configmaps.adoc} | 10 +++++----- modules/contributor/partials/adr-nav.adoc | 1 + 9 files changed, 13 insertions(+), 12 deletions(-) rename modules/contributor/pages/adr/{drafts/ARDx-using_values_from_configmaps.adoc => ARD014-using_values_from_configmaps.adoc} (96%) diff --git a/modules/contributor/pages/adr/ADR001-choose_project_language.adoc b/modules/contributor/pages/adr/ADR001-choose_project_language.adoc index 2f4bd980b..ec5b3b166 100644 --- a/modules/contributor/pages/adr/ADR001-choose_project_language.adoc +++ b/modules/contributor/pages/adr/ADR001-choose_project_language.adoc @@ -1,4 +1,4 @@ -= Use English as Documentation Language += ADR001: Use English as Documentation Language Sönke Liebau v1.0, 19.08.2020 :status: accepted diff --git a/modules/contributor/pages/adr/ADR002-choose_repository_structure.adoc b/modules/contributor/pages/adr/ADR002-choose_repository_structure.adoc index a5a833b57..d7ceab56b 100644 --- a/modules/contributor/pages/adr/ADR002-choose_repository_structure.adoc +++ b/modules/contributor/pages/adr/ADR002-choose_repository_structure.adoc @@ -1,4 +1,4 @@ -= Use Multiple Repositories instead of one Large Repository += ADR002: Use Multiple Repositories instead of one Large Repository Sönke Liebau v1.0, 19.08.2020 :status: accepted diff --git a/modules/contributor/pages/adr/ADR003-choose_review_mechanism.adoc b/modules/contributor/pages/adr/ADR003-choose_review_mechanism.adoc index 458c2f004..d7fd3656e 100644 --- a/modules/contributor/pages/adr/ADR003-choose_review_mechanism.adoc +++ b/modules/contributor/pages/adr/ADR003-choose_review_mechanism.adoc @@ -1,4 +1,4 @@ -= Use RTC as Review Mechanism for Changes += ADR003: Use RTC as Review Mechanism for Changes Sönke Liebau v1.0, 19.08.2020 :status: accepted diff --git a/modules/contributor/pages/adr/ADR004-choose_agent_programming_language.adoc b/modules/contributor/pages/adr/ADR004-choose_agent_programming_language.adoc index 7d643e578..3810dd7fe 100644 --- a/modules/contributor/pages/adr/ADR004-choose_agent_programming_language.adoc +++ b/modules/contributor/pages/adr/ADR004-choose_agent_programming_language.adoc @@ -1,4 +1,4 @@ -= Use Rust as programming language for the agent += ADR004: Use Rust as programming language for the agent Sönke Liebau v1.0, 02.10.2020 :status: accepted diff --git a/modules/contributor/pages/adr/ADR005-systemd_unit_file_location.adoc b/modules/contributor/pages/adr/ADR005-systemd_unit_file_location.adoc index 5ad92429a..58dd21f79 100644 --- a/modules/contributor/pages/adr/ADR005-systemd_unit_file_location.adoc +++ b/modules/contributor/pages/adr/ADR005-systemd_unit_file_location.adoc @@ -1,4 +1,4 @@ -= Decide on handling and location of systemd unit files += ADR005: Decide on handling and location of systemd unit files Sönke Liebau v1.0, 07.01.2021 :status: accepted diff --git a/modules/contributor/pages/adr/ADR007-defined_reuse_of_k8s.adoc b/modules/contributor/pages/adr/ADR007-defined_reuse_of_k8s.adoc index 0e4a92f03..01ecf8ec3 100644 --- a/modules/contributor/pages/adr/ADR007-defined_reuse_of_k8s.adoc +++ b/modules/contributor/pages/adr/ADR007-defined_reuse_of_k8s.adoc @@ -1,4 +1,4 @@ -= Decide if Kubernetes Components Are to be Reused for Stackable += ADR007: Decide if Kubernetes Components Are to be Reused for Stackable Sönke Liebau v0.1, 27.08.2020 :status: draft diff --git a/modules/contributor/pages/adr/ADR008-decide_reuse_of_operators.adoc b/modules/contributor/pages/adr/ADR008-decide_reuse_of_operators.adoc index 2bb9106d7..9e6a6c1e4 100644 --- a/modules/contributor/pages/adr/ADR008-decide_reuse_of_operators.adoc +++ b/modules/contributor/pages/adr/ADR008-decide_reuse_of_operators.adoc @@ -1,4 +1,4 @@ -= Allow Reuse of Existing Kubernetes Operators += ADR008: Allow Reuse of Existing Kubernetes Operators Sönke Liebau v0.1, 15.01.2021 :status: draft diff --git a/modules/contributor/pages/adr/drafts/ARDx-using_values_from_configmaps.adoc b/modules/contributor/pages/adr/ARD014-using_values_from_configmaps.adoc similarity index 96% rename from modules/contributor/pages/adr/drafts/ARDx-using_values_from_configmaps.adoc rename to modules/contributor/pages/adr/ARD014-using_values_from_configmaps.adoc index d01ec108d..06301fec8 100644 --- a/modules/contributor/pages/adr/drafts/ARDx-using_values_from_configmaps.adoc +++ b/modules/contributor/pages/adr/ARD014-using_values_from_configmaps.adoc @@ -1,7 +1,7 @@ -= How Should Operators Use Values from ConfigMaps & Secrets -Doc Writer -v0.1, 13.04.2022 -:status: draft += ADR014: How Should Operators Use Values from ConfigMaps & Secrets +Sönke Liebau and Felix Hennig +v0.1, 2022-04-28 +:status: accepted * Status: {status} * Deciders: @@ -10,7 +10,7 @@ v0.1, 13.04.2022 ** Sebastian Bernauer ** Sönke Liebau ** Teo Klestrup Röijezon -* Date: 13.04.2022 +* Date: 2022-04-28 Technical Story: https://github.com/stackabletech/issues/issues/208[#208] diff --git a/modules/contributor/partials/adr-nav.adoc b/modules/contributor/partials/adr-nav.adoc index 196f25bf6..0b010ac0a 100644 --- a/modules/contributor/partials/adr-nav.adoc +++ b/modules/contributor/partials/adr-nav.adoc @@ -12,6 +12,7 @@ **** xref:adr/ADR011-directory_structure.adoc[] **** xref:adr/ADR012-authn_token_management.adoc[] **** xref:adr/ADR013-user_authentication_for_products.adoc[] +**** xref:adr/ARD014-using_values_from_configmaps.adoc[] * *** Deprecated **** xref:adr/deprecated/ADR006-choose_orchestrator_storage_backend.adoc[]