Table of Contents
Pytomo is a Python based tomographic tool to perform analysis of Youtube video download rates. We first select an initial list of videos that we would like to start the analysis with. For the videos in this list the Pytomo tool first finds the IP address of the cache servers on which these videos are located. The cache server is pinged to obtain the RTT times. Then we try to download the video for a limited amount of time to calculate the different statistics of the download.
The setup file used to create the python package.
This file in RestructuredText.
This is the global launcher for the Pytomo tool.
This folder contains the pytomo package. The contents of it are listed below.
The top-most module that is used to run the Pytomo tool.
Terminal:
python start_pytomo.pyInteractive python:
import start_pytomo.main import config_pytomo start_pytomo.main()
compute_stats(url) Returns a list of the statistics related to the url. The contents of the list are : (url, cache_url, current_stats) where current_stats is a list containing: [
Ping_times, download_statistics, DNS resolver used ]
format_stats(stats) Functions used to format the stats obtained from compute_stats function so that they can be inserted into the sqlite3 database. The stats are converted into a tuple. The arguments to this function is the list returned by compute_stats().
File containing the various parameters and constants that are used for the analysis. The following parameters determine the nature of the crawl.
Module to download youtube video for a limited amount of time and calculate the different statistics needed for the analysis. It has the FileDownloader class and the YoutubeInfoExtractor class. The following functions defined in this module are used to get the statistics of the download.
Usage
Terminal:
python lib_youtube_download.py url
Functions
Return the cache url of the youtube video.
Returns a tuple of stats for download from an url based on ip address. The tuple contains stats containing:
[
downloadtime, data duration, datalength, video encodingrate, size of video in bytes, Nb.of interruptions, accumulated buffer size, accumulated playback , current remaining buffer, Maximum Instant Throughput achieved]
Module to retrieve the IP address of a URL out of a set of nameservers (default nameservers and the ones provided in the config_pytomo.py file as EXTRA_NAME_SERVERS).
Usage
Not meant to be called directly. But provides the two functions discussed below.
Functions
Returns a list of IP addresses of default name servers.
Resolves the IP address of the url and returns a list of tuples with the IP address and the DNS resolver used. Multiple nameservers are tried. The default nameservers used are Google DNS, Open DNS, the default nameserver for the machine.
Module to generate the RTT times of a ping.This module has been configured to work on Linux, Windows and Mac systems.
Usage
Not meant to be called directly. But provides the two functions discussed below.
Functions
Function that generate the Regular Expression to match the RTT patterns on different operating systems.
[
pingpackets=config_pytomo.PING_PACKETS]
):Returns a list of the min, avg and max ping values. Currently designed to work with Windows, Mac and Linux systems.
Module to retrieve the related videos from a file with a list of Youtube links and to store it for next round of the crawl.
Usage
Terminal:
lib_cache_url.py
[
-w out_file]
[
-p 10]
[
-u 2]
[
-V]
url_file_inInteractive python:
import lib_cache_url urlfilein = 'urlfile.txt' lib_cache_url.get_next_round_urls(url_file_in, [max_per_page=20, max_per_url=5])
Arguments
max_per_url: Number of links to select per page (max_per_url links will be randomly selected out of the max_per_page considered).
Functions
get_all_links(url):
Returns a list of all links from the url.
Returns a set of links that direct to Youtube.
Returns the interesting part of a Youtube url(the url with only the video ID).
[
max_per_page=20, max_per_url=5]
):Return a set of max_per_url links from max_per_page randomly chosen related urls (other links are ignored).
Return a tuple of the set of input urls and a set of related url of videos.
Function to get the most popular Youtube videos according to the time frame.
Usage
Not meant to be called directly. But provides one function as discussed below.
Arguments
Function
[
max_results=25]
):Returns the most popular Youtube links(world-wide). The number of videos needed is given as max_results. (The results returned are in no particular order).
Module that creates and manages the database for th Pytomo results. It has PytomoDatabase class.The columns of the database are listed in the docstring of the module.
Usage
Not meant to be called directly but the PytomoDatabase class is to be used.
Functions The PytomoDatabase class has the following functions:
Function to create a table.
Function to insert a record into the database.
Function to print all the records of the table.
Closes the connection to the database.
This is a stripped down version of the Kaa Metadata Python Package (Version :'0.7.7').The package has been modified to be used with the lib_youtube_download.py module so that we can obtain the metadata of the video.The main modification was to make it independent of kaabase.
This is the DNS Python Package(Version : 1.9.2) that is used to obtain the nameservers for the machine and also to send DNS queries to the nameservers to obtain the IP addresses of the Youtube cache servers.
This folder contains the log files generated by the logger. These files contain the log details generated during the crawl run.
This folder contains the database files used to store the results. The columns of the database are as follows:
This folder contains the result files. The results in these file are listed in text format. It is a list containing [Video url, Cache url, IP address of cache server, Ping RTT times to the Cache server, Download stats, Name - IP address of the DNS resolver]
where Download stats = [
downloadtime, data duration, data length, video encoding rate, size of video in bytes, Nb.of interruptions, accumulated buffer-size, accumulated playback , current remaining buffer ]