Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
S
SMART
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
3
Issues
3
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Registry
Registry
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
UNI-KLU
SMART
Commits
9fa5ed39
Commit
9fa5ed39
authored
Mar 25, 2020
by
Alex
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Added Layer REST interface and database persistence
Moved REST routes
parent
ad6f1889
Changes
13
Hide whitespace changes
Inline
Side-by-side
Showing
13 changed files
with
170 additions
and
24 deletions
+170
-24
swagger.yml
.../community-detection-microservice/app/configs/swagger.yml
+74
-11
agi_repository.py
...unity-detection-microservice/app/db/agi/agi_repository.py
+6
-6
__init__.py
...munity-detection-microservice/app/db/entities/__init__.py
+2
-1
layer.py
...community-detection-microservice/app/db/entities/layer.py
+51
-0
repository.py
...hub/community-detection-microservice/app/db/repository.py
+10
-2
clusterer.py
...ction-microservice/app/processing/clustering/clusterer.py
+4
-4
requirements.txt
...hub/community-detection-microservice/app/requirements.txt
+2
-0
cluster.py
...ub/community-detection-microservice/app/routes/cluster.py
+0
-0
debug.py
...-hub/community-detection-microservice/app/routes/debug.py
+0
-0
functions.py
.../community-detection-microservice/app/routes/functions.py
+0
-0
layers.py
...hub/community-detection-microservice/app/routes/layers.py
+21
-0
location.py
...b/community-detection-microservice/app/routes/location.py
+0
-0
user_cluster.py
...mmunity-detection-microservice/app/routes/user_cluster.py
+0
-0
No files found.
src/data-hub/community-detection-microservice/app/configs/swagger.yml
View file @
9fa5ed39
...
...
@@ -14,7 +14,7 @@ basePath: "/api"
paths
:
/debug
:
post
:
operationId
:
"
r
est
.debug.echo"
operationId
:
"
r
outes
.debug.echo"
tags
:
-
"
Echo"
summary
:
"
Echo
function
for
debugging
purposes"
...
...
@@ -31,7 +31,7 @@ paths:
/locations
:
post
:
operationId
:
"
r
est
.location.post"
operationId
:
"
r
outes
.location.post"
tags
:
-
"
Locations"
summary
:
"
Add
new
location
data"
...
...
@@ -48,7 +48,7 @@ paths:
400
:
description
:
"
Invalid
input"
get
:
operationId
:
"
r
est
.location.get"
operationId
:
"
r
outes
.location.get"
tags
:
-
"
Locations"
summary
:
"
Get
location
data"
...
...
@@ -61,7 +61,7 @@ paths:
/location-collections
:
post
:
operationId
:
"
r
est
.location.post_many"
operationId
:
"
r
outes
.location.post_many"
tags
:
-
"
Locations"
summary
:
"
Add
new
location
data
collection"
...
...
@@ -78,9 +78,39 @@ paths:
400
:
description
:
"
Invalid
input"
/layers
:
post
:
operationId
:
"
routes.layers.post"
tags
:
-
"
Layers"
summary
:
"
Add
a
new
layer
or
overwrite
an
existing
one"
parameters
:
-
in
:
body
name
:
"
Layer"
description
:
"
The
layer
data
to
be
added"
required
:
true
schema
:
$ref
:
"
#/definitions/Layer"
responses
:
201
:
description
:
"
Successful
operation"
400
:
description
:
"
Invalid
input"
get
:
operationId
:
"
routes.layers.get"
tags
:
-
"
Layers"
summary
:
"
Get
all
layer
data"
parameters
:
[]
responses
:
200
:
description
:
"
Successful
operation"
schema
:
$ref
:
"
#/definitions/Layer-pythonicCollection"
/location-clusters
:
get
:
operationId
:
"
r
est
.cluster.get_locations"
operationId
:
"
r
outes
.cluster.get_locations"
tags
:
-
"
Clusters"
summary
:
"
Get
user
communities
clustered
by
location"
...
...
@@ -93,7 +123,7 @@ paths:
# /clusters/cluster.png:
# get:
# operationId: "r
est
.cluster.get_image"
# operationId: "r
outes
.cluster.get_image"
# tags:
# - "Clusters"
# summary: "Get user communities per date per hour as image"
...
...
@@ -106,7 +136,7 @@ paths:
/time-clusters
:
get
:
operationId
:
"
r
est
.cluster.get_times"
operationId
:
"
r
outes
.cluster.get_times"
tags
:
-
"
Clusters"
summary
:
"
Get
user
communities
clustered
by
time
per
hour"
...
...
@@ -119,7 +149,7 @@ paths:
# /agi/clusters/cluster.png:
# get:
# operationId: "r
est
.agi_cluster.get_image"
# operationId: "r
outes
.agi_cluster.get_image"
# tags:
# - "Clusters"
# summary: "Get user communities per date per hour from agi data as image"
...
...
@@ -132,7 +162,7 @@ paths:
/user-cluster-graphs
:
get
:
operationId
:
"
r
est
.user_cluster.get"
operationId
:
"
r
outes
.user_cluster.get"
tags
:
-
"
User
Graphs"
summary
:
"
Get
user
graphs
per
layer
per
cluster"
...
...
@@ -145,7 +175,7 @@ paths:
/rfc/run
:
post
:
operationId
:
"
r
est
.functions.run_agi_clustering_and_graph_creation"
operationId
:
"
r
outes
.functions.run_agi_clustering_and_graph_creation"
tags
:
-
"
Remote
function
calls"
summary
:
"
Insert
locations
from
AGI,
create
clusters
for
starting
time
and
location
layers,
create
graphs
for
the
location
clusters"
...
...
@@ -235,4 +265,37 @@ definitions:
UserClusterGraphCollection
:
type
:
array
items
:
$ref
:
"
#/definitions/UserClusterGraph"
\ No newline at end of file
$ref
:
"
#/definitions/UserClusterGraph"
Layer
:
type
:
object
properties
:
LayerName
:
type
:
string
Nodes
:
type
:
array
items
:
type
:
object
Properties
:
type
:
array
items
:
type
:
string
Layer-pythonic
:
type
:
object
properties
:
layer_name
:
type
:
string
nodes
:
type
:
array
items
:
type
:
object
properties
:
type
:
array
items
:
type
:
string
Layer-pythonicCollection
:
type
:
array
items
:
$ref
:
"
#/definitions/Layer-pythonic"
\ No newline at end of file
src/data-hub/community-detection-microservice/app/db/agi/agi_repository.py
View file @
9fa5ed39
...
...
@@ -32,18 +32,18 @@ class AgiRepository:
def
getLocationsBasedOnNewDataSchema
(
self
):
'''Creates the new data generic schema to be used beginning on 24.03.2020'''
data
=
{
'
LayerN
ame'
:
'Destination'
,
'
N
odes'
:
self
.
getLocations
(),
'
P
roperties'
:
[
'latitude'
,
'longitude'
]
'
layer_n
ame'
:
'Destination'
,
'
n
odes'
:
self
.
getLocations
(),
'
p
roperties'
:
[
'latitude'
,
'longitude'
]
}
return
data
def
getTimesBasedOnNewDataSchema
(
self
):
'''Creates the new data generic schema to be used beginning on 24.03.2020'''
data
=
{
'
LayerN
ame'
:
'Starting_Time'
,
'
N
odes'
:
self
.
getLocations
(),
'
P
roperties'
:
[
'timestamp'
]
'
layer_n
ame'
:
'Starting_Time'
,
'
n
odes'
:
self
.
getLocations
(),
'
p
roperties'
:
[
'timestamp'
]
}
return
data
...
...
src/data-hub/community-detection-microservice/app/db/entities/__init__.py
View file @
9fa5ed39
from
db.entities.location
import
Location
from
db.entities.popular_location
import
PopularLocation
from
db.entities.cluster
import
Cluster
,
LocationCluster
,
TimeCluster
from
db.entities.user_cluster_graph
import
UserClusterGraph
\ No newline at end of file
from
db.entities.user_cluster_graph
import
UserClusterGraph
from
db.entities.layer
import
Layer
\ No newline at end of file
src/data-hub/community-detection-microservice/app/db/entities/layer.py
0 → 100644
View file @
9fa5ed39
import
json
from
datetime
import
datetime
from
typing
import
Dict
class
Layer
:
'''
This class represents a single layer of the Multilayer Graph.
:param layer_info: Information as dictionary to restore the layer object.
'''
def
__init__
(
self
,
layer_info
:
Dict
=
None
,
from_db
=
False
):
if
layer_info
is
not
None
:
self
.
from_serializable_dict
(
layer_info
,
from_db
)
def
to_serializable_dict
(
self
,
for_db
=
False
)
->
Dict
:
return
{
"layer_name"
:
self
.
layer_name
,
"properties"
:
self
.
properties
,
"nodes"
:
json
.
dumps
(
self
.
nodes
)
if
for_db
else
self
.
nodes
}
def
from_serializable_dict
(
self
,
layer_info
:
Dict
,
from_db
=
False
):
self
.
layer_name
=
layer_info
[
'layer_name'
]
self
.
properties
=
layer_info
[
'properties'
]
self
.
nodes
=
json
.
loads
(
layer_info
[
"nodes"
])
\
if
from_db
else
layer_info
[
"nodes"
]
def
__repr__
(
self
):
return
json
.
dumps
(
self
.
to_serializable_dict
())
def
__str__
(
self
):
return
f
"Layer({self.__repr__()})"
layer_d
=
{
"layer_name"
:
"Destination"
,
"nodes"
:
[
{
"TravelID"
:
1
,
"UserID"
:
"Micah"
,
"Latitude_Destination"
:
-
5.95081
,
"Longitude_Destination"
:
37.415281
,
"Finished_time"
:
1579143634812589
,
"TravelPrice"
:
19
}],
"properties"
:
[
'Latitude_StartingPoint'
,
'Longitude_StartingPoint'
]
}
layer
=
Layer
(
layer_d
)
print
(
layer
.
to_serializable_dict
(
for_db
=
True
))
\ No newline at end of file
src/data-hub/community-detection-microservice/app/db/repository.py
View file @
9fa5ed39
...
...
@@ -5,12 +5,12 @@ import json
from
db.agi.agi_repository
import
AgiRepository
from
db.entities
import
Location
,
TimeCluster
,
PopularLocation
,
LocationCluster
,
UserClusterGraph
from
db.entities
import
*
from
typing
import
List
class
Repository
(
MongoRepositoryBase
):
'''This
repository stores and loads locations and clusters with
MongoDb.'''
'''This
is a repository for
MongoDb.'''
def
__init__
(
self
):
super
()
.
__init__
(
netconst
.
COMMUNITY_DETECTION_DB_HOSTNAME
,
...
...
@@ -21,6 +21,7 @@ class Repository(MongoRepositoryBase):
self
.
_location_cluster_collection
=
'location_cluster'
self
.
_time_cluster_collection
=
'time_cluster'
self
.
_user_cluster_graph_collection
=
'user_cluster_graph'
self
.
_layer_collection
=
'layer'
self
.
agi_repo
=
AgiRepository
()
...
...
@@ -58,3 +59,10 @@ class Repository(MongoRepositoryBase):
def
get_user_cluster_graphs
(
self
)
->
List
[
UserClusterGraph
]:
user_graphs
=
super
()
.
get_entries
(
self
.
_user_cluster_graph_collection
)
return
[
UserClusterGraph
(
dict_
=
u
,
from_db
=
True
)
for
u
in
user_graphs
]
def
add_layer
(
self
,
layer
:
Layer
):
super
()
.
insert_entry
(
self
.
_layer_collection
,
layer
.
to_serializable_dict
())
def
get_layers
(
self
)
->
List
[
Layer
]:
entries
=
super
()
.
get_entries
(
self
.
_layer_collection
)
return
[
Layer
(
e
)
for
e
in
entries
]
\ No newline at end of file
src/data-hub/community-detection-microservice/app/processing/clustering/clusterer.py
View file @
9fa5ed39
...
...
@@ -153,22 +153,22 @@ if __name__ == '__main__':
agi_repo
=
AgiRepository
()
if
True
:
res_old
=
clusterer
.
cluster_locations
(
agi_repo
.
getLocationsBasedOnNewDataSchema
()[
'
N
odes'
])
res_old
=
clusterer
.
cluster_locations
(
agi_repo
.
getLocationsBasedOnNewDataSchema
()[
'
n
odes'
])
# print(res_old[11])
# [{'id': 'adad64cb-bd71-4b2b-9a70-e08eb8b19901-1570900602', 'latitude': -20.2695062, 'longitude': 57.6297389, 'timestamp': 1570900602, 'user': 'b57ad1fb396cfc18b8867fb2e08be723c2cdc2a6', 'cluster_label': 11}, {'id': '127af17b-e823-4d30-8227-00f5421bd48b-1549291309', 'latitude': -20.5362627, 'longitude': 47.2459749, 'timestamp': 1549291309, 'user': 'ca34bd51c4dc65cbc021cb27bcaa014ca082b8c4', 'cluster_label': 11}]
data
=
agi_repo
.
getLocationsBasedOnNewDataSchema
()
res
=
clusterer
.
cluster_dataset
(
data
[
'
Nodes'
],
data
[
'P
roperties'
])
res
=
clusterer
.
cluster_dataset
(
data
[
'
nodes'
],
data
[
'p
roperties'
])
# if res is not None:
# print(res[11])
assert
(
res_old
==
res
)
# time
res_old
=
clusterer
.
cluster_times
(
agi_repo
.
getTimesBasedOnNewDataSchema
()[
'
N
odes'
])
res_old
=
clusterer
.
cluster_times
(
agi_repo
.
getTimesBasedOnNewDataSchema
()[
'
n
odes'
])
data
=
agi_repo
.
getTimesBasedOnNewDataSchema
()
res
=
clusterer
.
cluster_dataset
(
data
[
'
Nodes'
],
data
[
'P
roperties'
])
res
=
clusterer
.
cluster_dataset
(
data
[
'
nodes'
],
data
[
'p
roperties'
])
print
(
res_old
[
20
])
print
(
res
[
20
])
...
...
src/data-hub/community-detection-microservice/app/requirements.txt
View file @
9fa5ed39
...
...
@@ -5,9 +5,11 @@ certifi==2019.11.28
chardet==3.0.4
Click==7.0
clickclick==1.2.2
colorama==0.4.3
connexion==2.6.0
cycler==0.10.0
decorator==4.4.1
Deprecated==1.2.7
Flask==1.1.1
idna==2.8
importlib-metadata==1.5.0
...
...
src/data-hub/community-detection-microservice/app/r
est
/cluster.py
→
src/data-hub/community-detection-microservice/app/r
outes
/cluster.py
View file @
9fa5ed39
File moved
src/data-hub/community-detection-microservice/app/r
est
/debug.py
→
src/data-hub/community-detection-microservice/app/r
outes
/debug.py
View file @
9fa5ed39
File moved
src/data-hub/community-detection-microservice/app/r
est
/functions.py
→
src/data-hub/community-detection-microservice/app/r
outes
/functions.py
View file @
9fa5ed39
File moved
src/data-hub/community-detection-microservice/app/routes/layers.py
0 → 100644
View file @
9fa5ed39
from
flask
import
request
,
Response
from
db.repository
import
Repository
from
db.entities
import
Layer
repo
=
Repository
()
def
post
():
body
=
request
.
json
_insert_layer
(
body
)
return
Response
(
status
=
201
)
def
get
():
return
[
l
.
to_serializable_dict
()
for
l
in
repo
.
get_layers
()]
def
_insert_layer
(
layer_data
:
dict
):
# convert object keys from ext source
layer_data
[
'layer_name'
]
=
layer_data
.
pop
(
'LayerName'
)
layer_data
[
'nodes'
]
=
layer_data
.
pop
(
'Nodes'
)
layer_data
[
'properties'
]
=
layer_data
.
pop
(
'Properties'
)
repo
.
add_layer
(
Layer
(
layer_data
))
src/data-hub/community-detection-microservice/app/r
est
/location.py
→
src/data-hub/community-detection-microservice/app/r
outes
/location.py
View file @
9fa5ed39
File moved
src/data-hub/community-detection-microservice/app/r
est
/user_cluster.py
→
src/data-hub/community-detection-microservice/app/r
outes
/user_cluster.py
View file @
9fa5ed39
File moved
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment