Recently, we released vantage6 version 4.3, which introduces two new major components of the infrastructure: a brand new user interface (UI) that is aimed at easily running analyses, and an algorithm store, used to find and manage your project's algorithms. Here, we introduce these new components and explain how they interact with each other and with the existing components.
Researcher UI
The previously existing vantage6 user interface was mostly aimed at administration: creating collaborations and nodes, registering users and handling their permissions. The functionality to execute analyses in that user interface was limited: a form had to be manually filled in, which was very error prone.
The new user interface is a full rewrite of the user interface code. In doing so, we have not just added functionality for the researcher, but we have also improved the code with lessons learned from the old UI. The new UI is more future-proof and faster than the previous implementation.
Apart from offering all the functionality of the previous UI, the new UI contains three main pages that help researchers in their work:
- Creating tasks
- Viewing tasks in progress
- Viewing finished tasks
Creating tasks
Creating tasks no longer has to be done by the user filling in free text fields. In the new UI, a researcher can select from a list of available algorithms. Then, the functions that are available for that algorithm will be visible for the user to select from, and so on. All the steps in the process are created such that whenever possible, a user can select from available options. To achieve this, certain metadata is automatically requested from data stations, such as which databases are available and which column names are in the data.
Viewing tasks in progress
When the task has been created, the UI shows the progress of the task. In a common scenario where the user executes an aggregator task, it will be shown first that this task is pending, and will update when the aggregator node starts the task. Then, when child tasks are created, these are also shown and their progress is updated live.
Below is an example of how the status page might look like for a task in progress. The main task is in progress and the subtask has been completed. However, one of the other nodes is not yet online - indicating this helps you to find out why a certain task is not finishing as quickly as you may expect.
Viewing finished tasks
On the final page, a user can download the results of the task. If the task failed, the different subtasks will be shown so that the user can easily find out where the analysis failed and retrieve the logs easily. In the near future, we aim to add also visualizations of the task's results to this page - this is detailed in the final section below.
Below is an example of how the results of a contingency table may be displayed in the UI. In this case, it shows how many people are overweight by gender - and if values at a certain node are below a privacy threshold, ranges are shown.
Algorithm store
In vantage6, algorithms are Docker images that may be hosted in several places: in the Docker hub and in private repositories such as harbor2.vantage6.ai. If you as a researcher don't know which algorithms are available and how to run them, it can be difficult to run tasks.
Algorithm findability
The algorithm store aims to mitigate this by being a central place where the algorithms are managed. Researchers can view which algorithms are available, and the store also knows how the algorithms can be run: it contains data about which functions are available within the algorithms and which arguments these functions require. This data is used in the new UI to allow the user to pick from the available options only.
Algorithm stores can be coupled to one or more vantage6 servers, and vantage6 servers can also be coupled to multiple algorithm stores. If you have a project for which you develop your own private vantage6 algorithms, it is useful to create your own algorithm store with private algorithms, that only couples to your project's vantage6 server. If you want to use algorithms already provided by the community, you can couple your vantage6 server to the community algorithm store at https://store.cotopaxi.vantage6.ai. Note that it is also possible to couple a store to a particular collaboration within your server. You can use that if one collaboration uses a different set of algorithms than other collaborations in the same server.
Algorithm trust
A second important improvement that we aim for with the algorithm store is to increase the trust in algorithms. When an algorithm is simply a Docker image, you cannot easily find out who created it and you don't know if it went through appropriate code review. For algorithms from the algorithm store, in future versions you will know which developer created the algorithm and who reviewed the code. At present, however, an authorized developer can still put their algorithm in the algorithm store without review.
Future plans
We aim to implement several important new functionalities in the algorithm store and researcher UI over the coming months.
First of all, we intend to implement a review process so that, depending on policies set by the algorithm store owner, each algorithm has to go through a review process before becoming available to the users of the algorithm store.
Second, the visualization of algorithm results and algorithms themselves will be improved in the UI. We will build functionality into the algorithm store so that developers can specify how algorithms should be visualized, and then the UI will use that information to show the results in a particular figure or table. There will also be pages created in the UI to view more details on the algorithms themselves, such as who created them and where you may find their documentation or code.
At a later stage, we also aim to implement an algorithm build pipeline. With this service, an algorithm developer will simply provide their code to the algorithm store and it will use that to build an algorithm Docker image. By doing this in the algorithm store, we reduce the workload for the developer (especially if they are not familiar with Docker) and also improve the trust that algorithms are built correctly.
Conclusion
Altogether, we hope that the new user interface and algorithm store will make it easier to do your research with the vantage6 infrastructure and that we make a new step in increasing the trust in the vantage6 network!