Case Study



1. Abstract

We have seen a huge shift in computing architecture over the past decade, with engineers opting to modularize and distribute their services to remain agile. This approach enables quick integration and iteration over the traditional monolithic approach to system design. But by distributing services over the network, we trade this development agility for processing time, as we accrue latency by increasing the number of requests made over the network to the various services.

The latency problem is made worse by the inflexibility of traditional API technologies. Although new approaches to API design have emerged to address some of these issues, adoption of these alternatives typically requires a complete redesign of each service.

In addition to adopting a microservice architecture by distributing services Acture serves a complex deep learning model that aims to serve upscaling of images for any type of image, in any scenario. In the following case study, I will discuss the creation of the machine learning model as well as the design decisions made in creating Acture.

2. GraphQL

The inflexibility of traditional API technologies led to the development of GraphQL. GraphQL is a strongly-typed query language for APIs developed by Facebook in 2012 to improve the performance of their mobile applications by defining a specification that reduced the need to prepare data on the server and parse it on the client’s end. It allows the client to request and receive exactly the data that is needed, no less, no more. GraphQL provides three ways to provide data to an application: mutations, queries, and subscriptions. Queries are the means by which the client can request data it needs from the server. Mutations are the means by which clients can create, update, and delete data on the server. Subscriptions allow for clients to listen to real-time messages from the server, in doing so, we maintain a real-time connection with the to the server.

2.1 What is GraphQL?

With GraphQL, the underlying available data is organized by a schema. The schema creates a hierarchy of type definitions, which typically represent objects. Type definitions have one or more fields, which indicates the return type of the data.

Fig. 1: Example GraphQL API Schema

This structure and strict type system means that we can query the schema to see the data available for querying, as well as how return objects may be structured. It also allows us to declare exactly which fields from a returned type are needed, meaning the client won’t overfetch from the application.

Fig. 2: Example GraphQL Query

And lastly, the GraphQL specification allows us to perform multiple queries in one request, even if the return types are independent, solving the under-fetching issues and preventing the need for multiple round-trips.

Fig. 3: Example GraphQL Response

Here is an example of a feature of GraphQL that allows nested queries and therefore requires only one client network request to obtain data from various resources-this prevents both under and over-fetching. Books are related to an author, we are querying for the authors and then sub-querying their books. Since the books are not a scalar type but a custom type, we need to specify which of their properties must be retrieved.

Fig. 4: Example GraphQL Nested Query and Response

In summary, the main benefits of GraphQL include:

  1. The client can customize queries to fetch the exact data that is needed when they need it
  2. It reduces the over and under-fetching of data
  3. It reduces the number of calls made over the network by the client to the API
  4. GraphQL provides a unified and optimized public API of services reachable through a single endpoint.

3. Containerization

Containerization is a type of virtualization where applications run on isolated spaces, known as clusters, while using the same operationg system. Each cluster is a fully-packaged computing environment containing binaries libraries, dependencies, and anything that the app requires to run. The cluster is abstracted away from the OS, and allows very little access to underlying resources of the host machine. Thus, containerized applications can be run on any infrastructure. In addition to this, there's less overhead when starting, and no need to setup guest processes for each cluster. Because of this, we are able to adopt a more agile software development process, reduce the cost of virtual machines, isolate problems with containers, and easily manage container orchestration through platforms such as Kubernetes.

3.1 Kubernetes

Kubernetes is a portable, extensible, open-source platform for managing containerized services released by Google in 2014. Kubernetes provides a framework to run distributed systems automatically, taking care of container failture, scaling by providing proper load balancing by measuring traffic to certain containers, and automated rollback and rollout for containers. Overall, Kubernetes is used to create applications that makes it easy for teams to manage by increasing development speed, allowing applications to be deployed anywhere, and by running efficient services. Kubernetes can automatically adjust the size of the cluster required to run services. It allows for automatic scaling based on demand of the application. To be clear, here are the benefits of Kubernetes:

  1. Increased development speed: it helps build cloud-native microservice applications. Supports containerization of prebuilt apps
  2. Flexible Deployment: allows applications to be run across on-site deployments and public clouds
  3. Efficient Deployment: automatically adjust cluster sizes required to run a service as to not be wasteful of resources

4. Implementation Details

As seen above, Acture attempts to implement a microservice architecture in order to allow for proper scaling of workload. We can investigate further how this is possible.

4.1 Flask and Kubernetes

In order to deploy the machine learning model efficiently, we attempt to access the model from a Flask API hosted on the Google Kubernetes Engine. In this case, our Flask API only contains one route, in which we pass in an image, open it using the openCV package, perform processing on it, including padding, separating alpha layers, performing image enhancement on each, combining the results and returning the image represented as a numpy array. In order to perform this, we first containerized the basic Flask API using Docker, and wrote a simple deployment file and the Kubernetes command-line tool, kubectl, allows us to deploy and inspect our containerized application on a Kubernetes Engine

4.2 Django and GraphQL

In order to process requests, as further image processing as the Flask API only returns a numpy array, we used Django in the backend of the application in order to allow for asynchronous processing. In this, we receive a GraphQL mutation that serves the image needed for enhancing, we can start a worker in order to process this request asynchronously and we simply redirect the user back to the original homepage. When the process is finished we can further process the received numpy array and repopulate a model field containing the newly enhanced image. A configured GraphQL subscription using ariadne returns the finished task, and notifies the user of the completed task.

4.3 React

In order to display finished tasks as quickly as possible to the user. The front end of the application was built using React, an open source tool developed by Facebook that allows for quickly building one-page applications. It is a simple front end, thus the implementation is not discussed, but rather why React was chosen. React is a modern JavaScript library that allows for the application to display the latest data on the UI without needing a page refresh. Thus users do not need to refresh the page in order to receive newly enhanced images.

5. Future Work

So, where does Acture go from here? As an open-source project, how can our team, and others in the community, extend or build upon Acture?

Though Acture is ready to use, we have three main features we would like to see included:

  • Specialized Models for Specialized Image Enhancing Currently, Acture only serves one model that allows users to upscale images, and the model was designed to be a universal model that increases the resolution on images regardless of whether it was an artifact, a portrait, a landscape, et cetera. Although the model works well on all types of inputs, limiting the scope of the model does not benefit the clients, as they are not able to upscale images to the utmost quality. In the future, I would like to have the same model trained on different training data, allowing for more specificity when training and allows for more options to clients.
  • Video/Video Game Enhancement Currently, we are only allowing users to upscale their images, but users also have old videos they might need upscaled. As we are able to split a video into frames and essentially pictures, we are able to add new processing to process video files. Additionally, although light research has been done into it, as we are able to enhance images and videos, another avenue of improvement will be upscaling of frames in video games for clearer gameplay and allowing users to upscale old video games.
  • Integration with Cloud Image Stores And lastly, although not beneficial to users, it will be much easier to scale the application for more microservices as described above, if the application was to implement a cloud image store such as AWS S3, or Cloudinary as we can make our implementation of the software much more efficient. Instead of returning a numpy array of the represented upscaled image, we can simply store the image in a cloud image store and return a link to the aws/cloudinary image. Then, we are able to serve the image much more cleanly and we can possibly abstract away Django, which will make the app easier to deploy.

6. Training Data Development

Although the building of the model was standard, the generation of the data was a challenge for building a holistic model to enhance all images, regardless of the type of image degredation. In order to do this holistically, we can take an N-step approach to generating training pairs for our GAN to consume. In this case, our N is random. We can classify image degredation to come in four categories: Blur, Resizing, Noise, and JPEG compression. The high-order degredation process, programatically applies these degredations to images, allowing for any type of degredation affecting the image, to be cleared by the model. This allows us to a set of regular images, apply random N-step degredation to these images and allow for the model to fix different types of noise.