Updated deployment scripts

f9f8a557 · Alexander Lercher · 1b4c365e · f9f8a557 · f9f8a557 · f9f8a557
Commit f9f8a557 authored Dec 11, 2019 by Alexander Lercher
39 changed files
--- a/README.md
+++ b/README.md
@@ -2,18 +2,18 @@
 We propose a framework underlying decentralized social media called SMART, capable of finding relevant interest communities without violating users’ privacy and anonymity. Its objective is to improve trust and eliminate malicious actors in participatory exchanges and collaborative decision making.

 ## Project structure
+The *tools* folder contains supportive scripts used during development and must not be used by any microservice.
+
+### Source code in *src/*
 The five folders *data-hub*, *message-broker*, *rest-gateway*, *transaction-hub-in* and *transaction-hub-out* contain the microservices for the five architecture layers respectively.

 The *modules* folder contains various Python modules used by multiple microservices to improve code-reuse. When building the Docker images this folder is copied into the app path.

-The *tools* folder contains supportive scripts used during development and must not be used by any microservice.
-
-The scripts *build.py* and *deploy.py* are used to create Docker images and deploy these images in Kubernetes respectively.
-
-The *images* folder holds the images used in this document.
+### Deployment scripts in *bin/*
+The scripts *build.py* and *deploy.py* are used to create Docker images and deploy these images with Kubernetes respectively.

 ## Overall microservice architecture
-![SMART architecture image](images/smart-architecture.png)
+![SMART architecture image](documentation/images/smart-architecture.png)

 ## Trace input handling
 1. New traces are POSTed to the REST gateway
@@ -21,4 +21,4 @@ The *images* folder holds the images used in this document.
 1. The trace retrieval microservice receives the message, stores the trace in its document-based database and sends a notification to the message broker
 1. The semantic linking microservice receives the notification and GETs all traces (including the new one) from the trace retrieval microservice
 1. All traces can now be processed
-![Input handling image](images/input-handling.png)
+![Input handling image](documentation/images/input-handling.png)
--- a/bin/build.py
+++ b/bin/build.py
 import os
 import shutil

-docker_reg = 'localhost:5000'
 DOCKER_COMPOSE_NAME = "Dockerfile"
+PATH = '../src/'

 paths = []
-for r, _, f in os.walk('./'):
-    for file in f:
-        if DOCKER_COMPOSE_NAME == file:
-            paths.append(os.path.join(r, ''))
+for r, _, f in os.walk(PATH):
+    for filename in f:
+        if DOCKER_COMPOSE_NAME == filename:
+            paths.append(os.path.normpath(r))

 command_args = [{'path': path,
-    'name': str(path).split('/')[-2]}
+    'name': str(path).split(os.path.normpath('/'))[-1]}
    for path
    in paths]

+res_str = []
 for command_arg in command_args:
    path = command_arg['path']
-    image_name = f"{docker_reg}/{command_arg['name']}"
+    image_name = f"{command_arg['name']}"

    # copy Dockerfile into root dir to have access to modules folder
    shutil.copy2(os.path.join(path, DOCKER_COMPOSE_NAME), './')
-    os.system(f"docker image build -t {image_name} .")
+    
+    # build then remove Dockerfile 
+    exit_val = os.system(f"docker image build -t {image_name} .")
    os.remove(DOCKER_COMPOSE_NAME)

-    os.system(f"docker push {image_name}")
+    res_str .append (f"### {image_name} built with exit code {exit_val}")
+
+#     os.system(f"docker push {image_name}")

 print(f"Found {len(command_args)} images")
+for s in res_str:
+    print(s)
\ No newline at end of file
--- a/src/data-hub/agent-discovery-microservice/Dockerfile
+++ b/src/data-hub/agent-discovery-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY data-hub/agent-discovery-microservice/app/ /app/
+COPY src/data-hub/agent-discovery-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/data-hub/agent-discovery-microservice/deployment/deployment.yml
+++ b/src/data-hub/agent-discovery-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: agent-discovery
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: agent-discovery
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: agent-discovery
-        image: 143.205.173.97:5000/agent-discovery-microservice
+        image: agent-discovery-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/data-hub/community-detection-microservice/Dockerfile
+++ b/src/data-hub/community-detection-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY data-hub/community-detection-microservice/app/ /app/
+COPY src/data-hub/community-detection-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/data-hub/community-detection-microservice/deployment/deployment.yml
+++ b/src/data-hub/community-detection-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: community-detection
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: community-detection
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: community-detection
-        image: 143.205.173.97:5000/community-detection-microservice
+        image: community-detection-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/data-hub/democratic-reasoning-microservice/Dockerfile
+++ b/src/data-hub/democratic-reasoning-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY data-hub/democratic-reasoning-microservice/app/ /app/
+COPY src/data-hub/democratic-reasoning-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/data-hub/democratic-reasoning-microservice/deployment/deployment.yml
+++ b/src/data-hub/democratic-reasoning-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: democratic-reasoning
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: democratic-reasoning
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: democratic-reasoning
-        image: 143.205.173.97:5000/democratic-reasoning-microservice
+        image: democratic-reasoning-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/data-hub/geo-profiling-microservice/Dockerfile
+++ b/src/data-hub/geo-profiling-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY data-hub/geo-profiling-microservice/app/ /app/
+COPY src/data-hub/geo-profiling-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/data-hub/geo-profiling-microservice/deployment/deployment.yml
+++ b/src/data-hub/geo-profiling-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: geo-profiling
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: geo-profiling
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: geo-profiling
-        image: 143.205.173.97:5000/geo-profiling-microservice
+        image: geo-profiling-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/data-hub/pareto-trust-microservice/Dockerfile
+++ b/src/data-hub/pareto-trust-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY data-hub/pareto-trust-microservice/app/ /app/
+COPY src/data-hub/pareto-trust-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/data-hub/pareto-trust-microservice/deployment/deployment.yml
+++ b/src/data-hub/pareto-trust-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: pareto-trust
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: pareto-trust
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: pareto-trust
-        image: 143.205.173.97:5000/pareto-trust-microservice
+        image: pareto-trust-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/data-hub/reputation-calculation-microservice/Dockerfile
+++ b/src/data-hub/reputation-calculation-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY data-hub/reputation-calculation-microservice/app/ /app/
+COPY src/data-hub/reputation-calculation-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/data-hub/reputation-calculation-microservice/deployment/deployment.yml
+++ b/src/data-hub/reputation-calculation-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: reputation-calculation
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: reputation-calculation
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: reputation-calculation
-        image: 143.205.173.97:5000/reputation-calculation-microservice
+        image: reputation-calculation-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/data-hub/role-discovery-microservice/Dockerfile
+++ b/src/data-hub/role-discovery-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY data-hub/role-discovery-microservice/app/ /app/
+COPY src/data-hub/role-discovery-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/data-hub/role-discovery-microservice/deployment/deployment.yml
+++ b/src/data-hub/role-discovery-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: role-discovery
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: role-discovery
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: role-discovery
-        image: 143.205.173.97:5000/role-discovery-microservice
+        image: role-discovery-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/data-hub/semantic-linking-microservice/Dockerfile
+++ b/src/data-hub/semantic-linking-microservice/Dockerfile
@@ -12,7 +12,7 @@ RUN pip install pika
 EXPOSE 5000

 WORKDIR /app
-COPY data-hub/semantic-linking-microservice/app/ /app/
+COPY src/data-hub/semantic-linking-microservice/app/ /app/
 COPY modules/ /app/
 RUN chmod a+x main.py


--- a/src/data-hub/semantic-linking-microservice/deployment/deployment.yml
+++ b/src/data-hub/semantic-linking-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: semantic-linking
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: semantic-linking
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: semantic-linking
-        image: 143.205.173.97:5000/semantic-linking-microservice
+        image: semantic-linking-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/data-hub/stage-discovery-microservice/Dockerfile
+++ b/src/data-hub/stage-discovery-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY data-hub/stage-discovery-microservice/app/ /app/
+COPY src/data-hub/stage-discovery-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/data-hub/stage-discovery-microservice/deployment/deployment.yml
+++ b/src/data-hub/stage-discovery-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: stage-discovery
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: stage-discovery
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: stage-discovery
-        image: 143.205.173.97:5000/stage-discovery-microservice
+        image: stage-discovery-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/data-hub/stage-discovery-microservice/deployment/service.yml
+++ b/src/data-hub/stage-discovery-microservice/deployment/service.yml
--- a/src/message-broker/deployment/deployment.yml
+++ b/src/message-broker/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: rabbit-mq
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: rabbit-mq
  ports:

--- a/src/rest-gateway/Dockerfile
+++ b/src/rest-gateway/Dockerfile
@@ -13,7 +13,7 @@ RUN pip install deprecated
 EXPOSE 5000

 WORKDIR /app
-COPY rest-gateway/app/ /app/
+COPY src/rest-gateway/app/ /app/
 COPY modules/ /app/
 RUN chmod a+x main.py


--- a/src/rest-gateway/deployment/deployment.yml
+++ b/src/rest-gateway/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: rest-gateway
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: rest-gateway
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: rest-gateway
-        image: 143.205.173.97:5000/rest-gateway
+        image: rest-gateway
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/transaction-hub-in/network-metrics-retrieval-microservice/Dockerfile
+++ b/src/transaction-hub-in/network-metrics-retrieval-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY transaction-hub-in/network-metrics-retrieval-microservice/app/ /app/
+COPY src/transaction-hub-in/network-metrics-retrieval-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/transaction-hub-in/network-metrics-retrieval-microservice/deployment/deployment.yml
+++ b/src/transaction-hub-in/network-metrics-retrieval-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: network-metrics-retrieval
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: network-metrics-retrieval
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: network-metrics-retrieval
-        image: 143.205.173.97:5000/network-metrics-retrieval-microservice
+        image: network-metrics-retrieval-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/transaction-hub-in/network-metrics-retrieval-microservice/deployment/service.yml
+++ b/src/transaction-hub-in/network-metrics-retrieval-microservice/deployment/service.yml
--- a/src/transaction-hub-in/trace-retrieval-microservice/Dockerfile
+++ b/src/transaction-hub-in/trace-retrieval-microservice/Dockerfile
@@ -14,7 +14,7 @@ RUN pip install deprecated
 EXPOSE 5000

 WORKDIR /app
-COPY transaction-hub-in/trace-retrieval-microservice/app/ /app/
+COPY src/transaction-hub-in/trace-retrieval-microservice/app/ /app/
 COPY modules/ /app/
 RUN chmod a+x main.py


--- a/src/transaction-hub-in/trace-retrieval-microservice/deployment/deployment.yml
+++ b/src/transaction-hub-in/trace-retrieval-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: trace-retrieval
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: trace-retrieval
  ports:
@@ -31,7 +29,7 @@ spec:
    spec:
      containers:
      - name: trace-retrieval
-        image: 143.205.173.97:5000/trace-retrieval-microservice
+        image: trace-retrieval-microservice
        ports:
        - containerPort: 5000
        imagePullPolicy: Always
@@ -42,8 +40,6 @@ metadata:
  name: trace-retrieval-db
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: trace-retrieval-db
  ports:

--- a/src/transaction-hub-out/data-access-microservice/Dockerfile
+++ b/src/transaction-hub-out/data-access-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY transaction-hub-out/data-access-microservice/app/ /app/
+COPY src/transaction-hub-out/data-access-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/transaction-hub-out/data-access-microservice/deployment/deployment.yml
+++ b/src/transaction-hub-out/data-access-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: data-access
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: data-access
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: data-access
-        image: 143.205.173.97:5000/data-access-microservice
+        image: data-access-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/transaction-hub-out/event-detection-microservice/Dockerfile
+++ b/src/transaction-hub-out/event-detection-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY transaction-hub-out/event-detection-microservice/app/ /app/
+COPY src/transaction-hub-out/event-detection-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/transaction-hub-out/event-detection-microservice/deployment/deployment.yml
+++ b/src/transaction-hub-out/event-detection-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: event-detection
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: event-detection
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: event-detection
-        image: 143.205.173.97:5000/event-detection-microservice
+        image: event-detection-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/transaction-hub-out/smart-matching-microservice/Dockerfile
+++ b/src/transaction-hub-out/smart-matching-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY transaction-hub-out/smart-matching-microservice/app/ /app/
+COPY src/transaction-hub-out/smart-matching-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/transaction-hub-out/smart-matching-microservice/deployment/deployment.yml
+++ b/src/transaction-hub-out/smart-matching-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: smart-matching
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: smart-matching
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: smart-matching
-        image: 143.205.173.97:5000/smart-matching-microservice
+        image: smart-matching-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/transaction-hub-out/tokenized-decision-making-microservice/Dockerfile
+++ b/src/transaction-hub-out/tokenized-decision-making-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY transaction-hub-out/tokenized-decision-making-microservice/app/ /app/
+COPY src/transaction-hub-out/tokenized-decision-making-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/transaction-hub-out/tokenized-decision-making-microservice/deployment/deployment.yml
+++ b/src/transaction-hub-out/tokenized-decision-making-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: tokenized-decision-making
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: tokenized-decision-making
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: tokenized-decision-making
-        image: 143.205.173.97:5000/tokenized-decision-making-microservice
+        image: tokenized-decision-making-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file
--- a/src/transaction-hub-out/trust-sla-microservice/Dockerfile
+++ b/src/transaction-hub-out/trust-sla-microservice/Dockerfile
@@ -11,7 +11,7 @@ RUN pip install connexion[swagger-ui]
 EXPOSE 5000

 WORKDIR /app
-COPY transaction-hub-out/trust-sla-microservice/app/ /app/
+COPY src/transaction-hub-out/trust-sla-microservice/app/ /app/
 RUN chmod a+x main.py

 CMD ["python", "./main.py"]
\ No newline at end of file
--- a/src/transaction-hub-out/trust-sla-microservice/deployment/deployment.yml
+++ b/src/transaction-hub-out/trust-sla-microservice/deployment/deployment.yml
@@ -4,8 +4,6 @@ metadata:
  name: trust-sla
 spec:
  type: LoadBalancer
-  externalIPs:
-    - 143.205.173.102
  selector:
      app: trust-sla
  ports:
@@ -31,6 +29,6 @@ spec:
    spec:
      containers:
      - name: trust-sla
-        image: 143.205.173.97:5000/trust-sla-microservice
+        image: trust-sla-microservice
        ports:
        - containerPort: 5000
\ No newline at end of file