Original software publication| Volume 14, 100416, November 01, 2022

# RatingsLib: A python library for rating methods with applications

Open AccessPublished:August 31, 2022

## Highlights

• Rating methods are required in many applications.
• RatingsLib is a python library for rating methods with applications in many fields.
• It contains a comprehensive application that deals with sports outcome predictions.
• It can be extended to several fields.
• In RatingsLib, new methods and applications can be easily implemented.

## Abstract

RatingsLib is an open-source library in Python dedicated to the implementation of rating/ranking systems with applications in sports and other fields. Since forecasting of sports event results has attracted considerable attention in recent years, our work confronts challenges related to the utilization of rating systems in sports results predictions. RatingsLib combines the rating methods with rank-based, statistical, and machine learning approaches to forecast sports event outcomes. Additionally, applications in other fields such as movie rankings are comprised. To demonstrate the applicability and extendibility of the library, several examples and use cases on real datasets are provided.

## Keywords

Tabled 1
 Current code version v1.0.0 Permanent link to code/repository used for this code version https://github.com/SoftwareImpacts/SIMPAC-2022-129 Permanent link to reproducible capsule https://codeocean.com/capsule/4120328/tree/v3 Legal code license MIT License Code versioning system used git Software code languages, tools and services used python Compilation requirements, operating environments and dependencies python $≥$ 3.8 numpy $≥$ 1.18.2, pandas $≥$ 1.4.2, scikit_learn $≥$ 1.0.2, scipy $≥$ 1.4.1, joblib $≥$ 1.1.0 If available, link to developer documentation/manual https://ktalattinis.github.io/ratingslib/ Support email for questions [email protected]

## 1. Introduction

A noteworthy category that has attracted much of scholars’ interest is the pairwise comparison ranking systems. Those systems are based on the pairwise comparison method [
• Fechner G.T.
Elemente der psychophysik, Vol. 2.
,
• Fechner G.T.
Elements of Psychophysics: Transl. by Helmut E. Adler, Vol. I.
] which offers an easy way to rate and rank objects by comparing them in pairs. Usually, ratings are generated by employing linear algebra and computational methods. In this work, we study several rating systems that belong to this category, namely Win-Loss, Colley [
• Colley W.
Colley’s bias free college football ranking method: The Colley Matrix Explained.
], Massey [
• Massey K.
Statistical models applied to the rating of sports teams.
], Keener [
• Keener J.P.
The Perron–Frobenius theorem and the ranking of football teams.
], Elo system [
• Elo A.E.
The Rating of Chessplayers, Past and Present.
], Offense–Defense [
• Govan A.Y.
• Langville A.N.
• Meyer C.D.
Offense-defense approach to ranking team sports.
,
• Govan A.Y.
Ranking theory with application to popular sports.
], GeM [
• Govan A.Y.
Ranking theory with application to popular sports.
,
• Brin S.
• Page L.
Reprint of: The anatomy of a large-scale hypertextual web search engine.
,

L.C. Ingram, Ranking NCAA Sports Teams with Linear Algebra, Charleston, 2007.

] and AccuRATE [
• Kyriakides G.
• Talattinis K.
• Stephanides G.
A hybrid approach to predicting sports results and an AccuRATE rating system.
]. Particularly, AccuRATE has been proposed in a previous publication [
• Kyriakides G.
• Talattinis K.
• Stephanides G.
A hybrid approach to predicting sports results and an AccuRATE rating system.
]. Finally, aggregation methods for ratings and rankings are also examined.
Although the above methods have their origin in sports teams’ ratings, they can be applied successfully in other fields. Nevertheless, utilizing rating systems in real-world problems and applications involves several stages that each one has different requirements. Data preparation and parameters tuning are some examples. In addition, the integration with other methods such as statistical or machine learning approaches makes the rating procedure a small part of the overall process. In general, the development of an application requires a big effort from researchers and usually programming skills.
The range of open-source software developed for the mentioned methods is very limited. There is a Matlab implementation included in Govan’s PhD thesis for some of those ranking systems [
• Govan A.Y.
Ranking theory with application to popular sports.
]. Furthermore, two open-source packages are implemented in R, the first one is the “comperank” [
• Chasnovski E.
comperank: ranking methods for competition results.
] while the second is called “raincin” [
• Gou J.
• Zhang F.
• Wu S.
A judging system for project showcase: rating and ranking with incomplete information.
,
• Gou J.
• Zhang F.
• Wu S.
raincin: ranking with incomplete information.
]. Last, “RankIt” [
• Wang R.
A simple ranking solution for matches.
] is the only available library in Python that offers pairwise comparison methods.
While existing packages are focused mainly on the implementation of the rating methods and not on the application part, in this paper, we introduce RatingsLib, a library in Python that implements a plethora of rating methods with applications in various fields. Although this work originally started as an application of rating methods that are utilized for sports event prediction, a sophisticated and generalized library with applications and examples in various fields emerged along the way.

## 2. Implementation

RatingsLib is a free Python library offered as open-source software under the MIT license. It requires Python 3.8 or newer, and the following python libraries: numpy, scipy, pandas, scikit-learn and joblib. As a python package, it can be easily installed with the command “pip install”. Additionally, RatingsLib provides high-quality and comprehensive documentation for use and API package. Also, it contains detailed step-by-step examples and several use cases.
The architecture of the software consists of three basic components: datasets, ratings, and applications. An overview is presented in Fig. 1. The “datasets” component is responsible for data functions i.e. parsing, data preparation, pre-processing, etc. For this component, the pandas library [
• Pandas Development Team T.
Pandas-Dev/Pandas: Pandas.
] was used. The “applications” component embeds the necessary modules for each application and examples from various fields.
Rating is the key component of the library and interacts with the other two components. Basically, it comprises the implementation of the rating systems, aggregation methods, and comparison measures. As mentioned before, the rating values in most cases are generated by using linear algebra approaches and computational methods. For this reason, most of the rating systems and methods in this work have been implemented by exploiting several functions of NumPy [
• Harris C.R.
• Millman K.J.
• van der Walt S.J.
• Gommers R.
• Virtanen P.
• Cournapeau D.
• Wieser E.
• Taylor J.
• Berg S.
• Smith N.J.
• Kern R.
• Picus M.
• Hoyer S.
• van Kerkwijk M.H.
• Brett M.
• Haldane A.
• del Río J.F.
• Wiebe M.
• Peterson P.
• Gérard-Marchant P.
• Sheppard K.
• Reddy T.
• Weckesser W.
• Abbasi H.
• Gohlke C.
• Oliphant T.E.
Array programming with NumPy.
] and SciPy [
• Virtanen P.
• Gommers R.
• Oliphant T.E.
• Haberland M.
• Reddy T.
• Cournapeau D.
• Burovski E.
• Peterson P.
• Weckesser W.
• Bright J.
• van der Walt S.J.
• Brett M.
• Wilson J.
• Millman K.J.
• Mayorov N.
• Nelson A.R.J.
• Jones E.
• Kern R.
• Larson E.
• Carey C.J.
• Polat İ.
• Feng Y.
• Moore E.W.
• VanderPlas J.
• Laxalde D.
• Perktold J.
• Cimrman R.
• Henriksen I.
• Quintero E.A.
• Harris C.R.
• Archibald A.M.
• Ribeiro A.H.
• Pedregosa F.
• van Mulbregt P.
• Contributors S.1.
SciPy 1.0: Fundamental algorithms for scientific computing in python.
] libraries in python that are intended for algebraic and scientific computations.
The class diagram of the ratings package is depicted in Fig. 2. The rating procedure is split into three phases: preparation, computation, and rate. As we can observe, the RatingSystem is an abstract class and each rating system implements the abstract methods: preparation_phase(), computation_phase(), and rate. We can perceive that the extension and implementation of new systems and methods can be easily made.

## 3. Functionalities & applications

The most important features of the library will be presented here. More details on the functionalities and the use of software can be found in the documentation. Also, examples and small tutorials are available as Jupyter notebooks.
• 1.
Data collection and preparation-related functions
• Raw data collection: this function is available for soccer data where the user can collect and store online data from football-data.co.uk repository [

Football-data, football results, statistic & soccer betting odds data.

].
• Pre-process data: basic pre-process procedures are implemented. For example, to remove unrated or NaN values.
• Calculation of statistics: calculates statistics of the dataset. For example, soccer teams’ statistics (number of wins, total goals).
• Data preparation: prepares the data in a suitable form required by the method or application. Since the library deals with pairwise comparisons, the data for rating and ranking must have the form of pairs.
• 2.
Rating/Ranking systems, aggregation methods, and comparisons measures In this work, we have implemented the following systems and methods:
• Rating systems: WinLoss, Colley [
• Colley W.
Colley’s bias free college football ranking method: The Colley Matrix Explained.
], Massey [
• Massey K.
Statistical models applied to the rating of sports teams.
], Keener [
• Keener J.P.
The Perron–Frobenius theorem and the ranking of football teams.
], Elo(wins and points version) [
• Elo A.E.
The Rating of Chessplayers, Past and Present.
], Offense–Defense [
• Govan A.Y.
• Langville A.N.
• Meyer C.D.
Offense-defense approach to ranking team sports.
,
• Govan A.Y.
Ranking theory with application to popular sports.
], GeM [
• Govan A.Y.
Ranking theory with application to popular sports.
,
• Brin S.
• Page L.
Reprint of: The anatomy of a large-scale hypertextual web search engine.
,

L.C. Ingram, Ranking NCAA Sports Teams with Linear Algebra, Charleston, 2007.

], AccuRATE [
• Kyriakides G.
• Talattinis K.
• Stephanides G.
A hybrid approach to predicting sports results and an AccuRATE rating system.
].
• Rating aggregations: Markov [
• Govan A.Y.
Ranking theory with application to popular sports.
,

L.C. Ingram, Ranking NCAA Sports Teams with Linear Algebra, Charleston, 2007.

,
• Langville A.N.
• Meyer C.D.
Who’s# 1?: The Science of Rating and Ranking.
], Perron [
• Keener J.P.
The Perron–Frobenius theorem and the ranking of football teams.
,
• Langville A.N.
• Meyer C.D.
Who’s# 1?: The Science of Rating and Ranking.
], Offense–Defense [
• Govan A.Y.
• Langville A.N.
• Meyer C.D.
Offense-defense approach to ranking team sports.
,
• Govan A.Y.
Ranking theory with application to popular sports.
,
• Langville A.N.
• Meyer C.D.
Who’s# 1?: The Science of Rating and Ranking.
].
• Ranking aggregation methods: Borda count [
• Borda J.d.
Mémoire sur les élections au scrutin.
], Average Rank [
• Langville A.N.
• Meyer C.D.
Who’s# 1?: The Science of Rating and Ranking.
].
• Comparison Measures: Kendall Tau [
• Kendall M.G.
A new measure of rank correlation.
].
• 3.
Sports Application
• Soccer outcome prediction The prediction techniques can be performed in three different ways:
• 1.
Rank-based: The logic of prediction is that a lower ranking (higher rating) is preferred over a higher ranking (lower rating).
• 2.
MLE: The ratings can be turned into predictions by selecting the highest probability from outcomes. The probabilities can be computed by applying the modified logistic function which is utilized by J. Lasek et al. [
• Lasek J.
• Szlávik Z.
• Bhulai S.
The predictive power of ranking systems in association football.
] and takes into account the ratings of teams. Also, J. Lasek et al. adopt the idea of Glickman [
• Glickman M.E.
Parameter estimation in large dynamic paired comparison experiments.
] for the computation of the draw probability. We have given the acronym “MLE” for this method since the required parameters for the probabilities are set to their Maximum Likelihood Estimation. We note that for their computation, we have used the function minimize of numpy [
• Harris C.R.
• Millman K.J.
• van der Walt S.J.
• Gommers R.
• Virtanen P.
• Cournapeau D.
• Wieser E.
• Taylor J.
• Berg S.
• Smith N.J.
• Kern R.
• Picus M.
• Hoyer S.
• van Kerkwijk M.H.
• Brett M.
• Haldane A.
• del Río J.F.
• Wiebe M.
• Peterson P.
• Gérard-Marchant P.
• Sheppard K.
• Reddy T.
• Weckesser W.
• Abbasi H.
• Gohlke C.
• Oliphant T.E.
Array programming with NumPy.
] library.
• 3.
Machine Learning Classification: the prediction can be made by utilizing the rating of teams as features for the machine learning classifiers. Also, it is possible to combine other features (i.e. games statistics). The available classifiers from scikit-learn library [
• Pedregosa F.
• Varoquaux G.
• Gramfort A.
• Michel V.
• Thirion B.
• Grisel O.
• Blondel M.
• Prettenhofer P.
• Weiss R.
• Dubourg V.
• Vanderplas J.
• Passos A.
• Cournapeau D.
• Brucher M.
• Perrot M.
• Duchesnay E.
Scikit-learn: machine learning in python.
] can be passed in RatingsLib’s functions.
There are two types of predictions that are offered by RatingsLib: hindsight and foresight. Hindsight prediction refers to predicting past games using the ratings of entire season games while foresight is to predicting upcoming games using the ratings of previous weeks. For the foresight predictions, we have included the following options to split the train and test sets:
• 1.
percentage split (e.g. 70% train, 30% test)
• 2.
match-week number split (e.g. 1–25 week train, 26–38 week test)
• 3.
anchored walk-forward [
• Pagano M.
Financial markets and growth: An overview.
] with a window size (explained in the first illustrative example)
Additionally, in order to test multiple rating methods and classifiers simultaneously, the multiprocessing library joblib [
• Joblib Development Team M.
Joblib: running Python functions as pipeline jobs.
] has been employed to improve the overall computational time. Finally, the prediction procedure provides a classification and accuracy report.
• Parameters Tuning The rating systems parameters can be tuned to improve a specific metric e.g. accuracy, f1-score, etc. The tuning is performed by grid-search after defining the search space of each parameter. An example of the tuning of Elo system parameters is provided in the manual.
• 4.
Movies Application Rating and ranking of movies from users’ ratings. This application is tested on MovieLens dataset [
• Harper F.M.
• Konstan J.A.
The MovieLens datasets: history and context.
].
• 5.
Applications & Examples in other fields
• Domain names ranking An illustrative example of domain names ranking based on [

K. Talattinis, C. Zervopoulou, G. Stephanides, Ranking Domain Names Using Various Rating Methods, in: Proceedings of the Ninth International Multi-Conference on Computing in the Global Information Technology, Seville, 2014.

], is provided.
• Finance Examples are provided from investment selection and portfolios ranking by utilizing rating and ranking aggregation methods.
• 6.
Testing — Validation
• RatingsLib provides unit tests for most of the mentioned functionalities.

## 4. Illustrative example

In this example, we assess the predictive performance of each rating system. We test three different methods and the target class is the final outcome of the English Premier League (2009–2018 seasons) matches. The methods are the Rank-based, the MLE and for the machine learning classifier we have chosen the Naïve Bayes. The overall application is divided into three stages.
The first stage is the data collection where we collect the data of the English Premier League from football-data.co.uk for the seasons 2009/10 to 2017/18. The second stage is devoted to the ratings procedure where we test all available rating systems. For each season the ratings are computed on a weekly basis based on the data from the previous match-weeks. The first match-week of each season is removed since there are no available ratings. After the generation of ratings, in the third stage, we perform predictions. We aim to test all rating systems, so each prediction method utilizes the rating values of one rating system every time. Totally 9 models are produced by each prediction method. The predictions are made by employing the anchored walk-forward and starting from the 4th week of each season. This means for every new week predictions are made by utilizing as a training set the ratings of previous weeks. The walk-forward process restarts at the beginning of every season. Fig. 3. illustrates the procedure. The accuracy results from each method are shown in Table 1.
Table 1Accuracy results.
Rating SystemRank-basedMLE-basedNaïve Bayes
AccuracyCorrect GamesWrong GamesAccuracyCorrect GamesWrong GamesAccuracyCorrect GamesWrong Games
Win-Loss0.4879153516110.5067159415520.488915381608
Colley0.4793150816380.5105160615400.490515431603
Massey0.4886153716090.5179159314830.494515211555
Elo-Win0.4803151116350.513161415320.489215391607
Elo-Point0.4879153516110.513161415320.492715501596
Keener0.4854152716190.5153162115250.498115671579
Offense–Defense0.486152916170.5048158815580.475214951651
GeM0.4943155515910.5079159815480.496515621584
AccuRATE0.4898154116050.5143161815280.492415491597

## 5. Impact overview

RatingsLib is an open-source library in Python that is not only limited to the implementation of rating systems but comprises applications as well. Moreover, several examples are provided with the aim to make clearer the use of implemented methods in other areas and fields. The fact that the code is written in Python makes the tool more accessible since Python is very popular.
The project has recently been made publicly available as open-source software and has not been established in the community. However, our software has been tested for the experiments conducted (partly or fully) in our publications [
• Kyriakides G.
• Talattinis K.
• Stephanides G.
A hybrid approach to predicting sports results and an AccuRATE rating system.
,

K. Talattinis, C. Zervopoulou, G. Stephanides, Ranking Domain Names Using Various Rating Methods, in: Proceedings of the Ninth International Multi-Conference on Computing in the Global Information Technology, Seville, 2014.

,

G. Kyriakides, K. Talattinis, S. George, Rating Systems Vs Machine Learning on the Context of Sports, in: Proceedings of the 18th Panhellenic Conference on Informatics, New York, NY, USA, 2014.

,

G. Kyriakides, K. Talattinis, G. Stephanides, Raw Rating Systems and Strategy Approaches to Sports Betting, in: Proceedings of MathSport International 2015, Loughborough, 2015.

]. Particularly, our software constitutes an upgraded version related to the initial version of software utilized in those studies since it uses newer technology, includes popular and established packages, and is faster. Furthermore, it has been successfully applied in [
• Harper F.M.
• Konstan J.A.
The MovieLens datasets: history and context.
] where it ranks movies in a real benchmark dataset.
The objective of RatingsLib is twofold: (1) to rate, rank, and compare a set of objects and (2) to apply rating methods in real-world applications. In this way, the software can be helpful for researchers, data scientists, academics, students, and professionals. For example, this library may significantly help sports analysts, bettors, and coaches to rate, rank, and make predictions for the outcome of future sports games. Furthermore, decision-makers, investors, and portfolio managers can adapt code to solve ranking problems in a simple way. Also, it can help scientists to combine a plethora of well-known rating methods with other approaches and techniques such as machine learning. This software can also be very useful for educational purposes. Since most of the ranking systems provided in this library use linear algebra and computational methods, this could help students to understand the applications and extensions of linear algebra, optimization models, and computational methods.
The library can be used by different user groups. Below we show the overall impact and impact per application for several groups:
• Overall: researchers, data scientists, programmers, students, decision-makers
• Sports prediction: sports analysts, bettors, coaches
• Movie ranking: online rental services, video rental shops
• Domain market ranking: investors, alternative fund managers
• Finance: investors, portfolio managers, traders, decision-makers
Another important aspect of RatingsLib is that each application and example provided can be utilized as a template or a baseline. In particular, a new application can be developed by small modifications to existing application or example. For instance, the development of wine rankings can have movie rankings as a baseline. Similarly, other applications such as real estate selection or car selection can be done by taking into consideration the example of domain market ranking. Table 2 shows some examples of possible new applications and we can conclude that RatingsLib can be extended to several fields.
Finally, our work has the following contributions:
Table 2Overview of possible new applications.
Baseline applicationPossible new applications
Sports predictionsports betting strategies, players’ ratings
Movie rankingsproducts & services recommendations
Domain market rankingreal estate selection, university rankings
Financefinancial instrument rankings, trading strategies selection
• Implementation of rating and ranking systems, aggregation methods, and comparison measures.
• A comprehensive application is developed and deals with sports outcome prediction. This application covers all the stages needed to create and test predictive models.
• Other applications and examples [

K. Talattinis, C. Zervopoulou, G. Stephanides, Ranking Domain Names Using Various Rating Methods, in: Proceedings of the Ninth International Multi-Conference on Computing in the Global Information Technology, Seville, 2014.

] based on real datasets [

Football-data, football results, statistic & soccer betting odds data.

,
• Harper F.M.
• Konstan J.A.
The MovieLens datasets: history and context.
] are comprised in the library.
• Free and open-source code is available on GitHub with extensive documentation.
• Easy-to-use software without the requirement of advanced programming skills.

## 6. Conclusions

Obviously, rating and ranking comprise a scientific field that attracts not only the interest of academia but also of the business community. For this reason, unlike other available tools, our work is more oriented to applications of rating methods. RatingsLib is the first open-source library that integrates rating methods with machine learning in the field of sports outcome prediction. Also, applications in other fields such as movie rankings are tested with a benchmark dataset. In addition, the software embeds several methods such as rating aggregation methods [
• Langville A.N.
• Meyer C.D.
Who’s# 1?: The Science of Rating and Ranking.
] and calculation of soccer outcome probabilities [
• Lasek J.
• Szlávik Z.
• Bhulai S.
The predictive power of ranking systems in association football.
]. Furthermore, comparison measures and data functions are provided to make the library more comprehensive.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## References

• Fechner G.T.
Elemente der psychophysik, Vol. 2.
Breitkopf u. Härtel, 1860
• Fechner G.T.
Elements of Psychophysics: Transl. by Helmut E. Adler, Vol. I.
Holt, Rinehart and Winston, 1966
• Colley W.
Colley’s bias free college football ranking method: The Colley Matrix Explained.
2002
• Massey K.
Statistical models applied to the rating of sports teams.
1997
• Keener J.P.
The Perron–Frobenius theorem and the ranking of football teams.
SIAM Rev. 1993; 35: 80-93
• Elo A.E.
The Rating of Chessplayers, Past and Present.
Arco Pub, 1978
• Govan A.Y.
• Langville A.N.
• Meyer C.D.
Offense-defense approach to ranking team sports.
J. Quant. Anal. Sports. 2009; 5
• Govan A.Y.
Ranking theory with application to popular sports.
2008
• Brin S.
• Page L.
Reprint of: The anatomy of a large-scale hypertextual web search engine.
Comput. Netw. 2012; 56: 3825-3833
1. L.C. Ingram, Ranking NCAA Sports Teams with Linear Algebra, Charleston, 2007.

• Kyriakides G.
• Talattinis K.
• Stephanides G.
A hybrid approach to predicting sports results and an AccuRATE rating system.
Int. J. Appl. Comput. Math. 2017; 3: 239-254
• Chasnovski E.
comperank: ranking methods for competition results.
2020
• Gou J.
• Zhang F.
• Wu S.
A judging system for project showcase: rating and ranking with incomplete information.
2020
• Gou J.
• Zhang F.
• Wu S.
raincin: ranking with incomplete information.
2020
• Wang R.
A simple ranking solution for matches.
2019
• Pandas Development Team T.
Pandas-Dev/Pandas: Pandas.
Zenodo, 2020
• Harris C.R.
• Millman K.J.
• van der Walt S.J.
• Gommers R.
• Virtanen P.
• Cournapeau D.
• Wieser E.
• Taylor J.
• Berg S.
• Smith N.J.
• Kern R.
• Picus M.
• Hoyer S.
• van Kerkwijk M.H.
• Brett M.
• Haldane A.
• del Río J.F.
• Wiebe M.
• Peterson P.
• Gérard-Marchant P.
• Sheppard K.
• Reddy T.
• Weckesser W.
• Abbasi H.
• Gohlke C.
• Oliphant T.E.
Array programming with NumPy.
Nature. 2020; 585: 357-362
• Virtanen P.
• Gommers R.
• Oliphant T.E.
• Haberland M.
• Reddy T.
• Cournapeau D.
• Burovski E.
• Peterson P.
• Weckesser W.
• Bright J.
• van der Walt S.J.
• Brett M.
• Wilson J.
• Millman K.J.
• Mayorov N.
• Nelson A.R.J.
• Jones E.
• Kern R.
• Larson E.
• Carey C.J.
• Polat İ.
• Feng Y.
• Moore E.W.
• VanderPlas J.
• Laxalde D.
• Perktold J.
• Cimrman R.
• Henriksen I.
• Quintero E.A.
• Harris C.R.
• Archibald A.M.
• Ribeiro A.H.
• Pedregosa F.
• van Mulbregt P.
• Contributors S.1.
SciPy 1.0: Fundamental algorithms for scientific computing in python.
Nature Methods. 2020; 17: 261-272
2. Football-data, football results, statistic & soccer betting odds data.

• Langville A.N.
• Meyer C.D.
Who’s# 1?: The Science of Rating and Ranking.
Princeton University Press, 2012
• Borda J.d.
Mémoire sur les élections au scrutin.
in: Histoire de l’Academie Royale des Sciences pour 1781, Paris, 17841784
• Kendall M.G.
A new measure of rank correlation.
Biometrika. 1938; 30
• Lasek J.
• Szlávik Z.
• Bhulai S.
The predictive power of ranking systems in association football.
Int. J. Appl. Pattern Recognit. 2013; 1: 27-46
• Glickman M.E.
Parameter estimation in large dynamic paired comparison experiments.
J. R. Stat. Soc. Ser. C. Appl. Stat. 1999; 48: 377-394
• Pedregosa F.
• Varoquaux G.
• Gramfort A.
• Michel V.
• Thirion B.
• Grisel O.
• Blondel M.
• Prettenhofer P.
• Weiss R.
• Dubourg V.
• Vanderplas J.
• Passos A.
• Cournapeau D.
• Brucher M.
• Perrot M.
• Duchesnay E.
Scikit-learn: machine learning in python.
J. Mach. Learn. Res. 2011; 12: 2825-2830
• Pagano M.
Financial markets and growth: An overview.
Eur. Econ. Rev. 1993; 37: 613-622
• Joblib Development Team M.
Joblib: running Python functions as pipeline jobs.
2020
• Harper F.M.
• Konstan J.A.
The MovieLens datasets: history and context.
ACM Trans. Interact. Intell. Syst. 2015; 5
3. K. Talattinis, C. Zervopoulou, G. Stephanides, Ranking Domain Names Using Various Rating Methods, in: Proceedings of the Ninth International Multi-Conference on Computing in the Global Information Technology, Seville, 2014.

4. G. Kyriakides, K. Talattinis, S. George, Rating Systems Vs Machine Learning on the Context of Sports, in: Proceedings of the 18th Panhellenic Conference on Informatics, New York, NY, USA, 2014.

5. G. Kyriakides, K. Talattinis, G. Stephanides, Raw Rating Systems and Strategy Approaches to Sports Betting, in: Proceedings of MathSport International 2015, Loughborough, 2015.