UberStudent Forums

Help and Community for Users of Uberstudent
Return to UberStudent Home Page
It is currently Tue Nov 21, 2017 6:31 am

All times are UTC




UberStudent Logo Shirt
Hey! You can wear this UberCool UberStudent Logo Shirt

Post new topic Reply to topic  [ 13 posts ]  Go to page 1, 2  Next
Author Message
PostPosted: Sat May 21, 2016 8:37 am 
Offline
UberStudent Forum Helper

Joined: Fri Oct 18, 2013 12:48 pm
Posts: 54
Location: University of South Florida, Tampa, FL
Hi UberStudent Community,

I'm proposing to change the default base source from being http://us.archive.ubuntu.com/ubuntu to
Code:
curl -s http://mirrors.ubuntu.com/mirrors.txt | xargs -n1 -I {} sh -c 'echo `curl -r 0-102400 -s -w %{speed_download} -o /dev/null {}/ls-lR.gz` {}' |sort -g -r |head -1| awk '{ print $2  }'

as proposed in http://askubuntu.com/a/719551

What do other UberStudents think?

_________________
Berker Peköz
PhD Student in Electrical Engineering
University of South Florida at Tampa


Top
 Profile  
 
Click Here to Get Inexpensive UberStudent Discs

Click here to get UberStudent discs delivered to your mailbox
Delivered Directly to Your Mailbox!
PostPosted: Sat May 21, 2016 5:40 pm 
Offline
UberStudent Founder and Lead Developer
User avatar

Joined: Sun Nov 11, 2012 8:54 am
Posts: 951
Location: UberStudent Headquarters
I think this may be a fabulous idea but it needs testing. Why isn't this default upstream? I don't know.

Running the line you suggested would have to be run as a script very early in the boot process that would add several seconds to boot time. The way more native to apt and simple way to do it seems to be this. I tested it and it works.

The initial best mirror for me was http://mirrors.namecheap.com/. A few hours later it was http://reflector.westga.edu. It's working well for me so far, and is apparently faster.

_________________
UberStudent Must Become Self-Sustaining. If UberStudent and my dedicated support of it has benefited you, it's important to make a donation. Thanks!

The UberStudent Headquarters:
    * Mobo: MSI 870-G45
    * Processor: AMD Phenom II X4 3.4GHz
    * RAM: G.SKILL 8GB DDR3 1600
    * Graphics: MSI NVIDIA GeForce GTX 750 Ti
    * Drive: OCZ Vertex SSD
    * Case: Rosewill ATX Mid Tower
    * Monitors: 2 x Dell UltraSharp
    * With UberStudent 4.3 Development Build
You should build your own computers, too!


Top
 Profile  
 
PostPosted: Fri May 27, 2016 5:55 am 
Offline
UberStudent Founder and Lead Developer
User avatar

Joined: Sun Nov 11, 2012 8:54 am
Posts: 951
Location: UberStudent Headquarters
I'll not go into too many details right now but after thinking through this over the past days and running multiple different tests, I don't think there is any good way to do anything like this by default--which, probably, is why no distro has this by default.

The option with the least drawbacks and highest possible return for users is to incorporate an apt mirror speed test in the ubiquity installer; but, the cons even to that seem to outweigh the pros, e.g., if I install in California and move to Florida, I've probably achieved a persistent negative in terms of speed.

In the case of doing a speed test at some other point, what the fastest mirror is still changes due to many factors, even day to day and sometimes hour to hour. In the case of doing a test for only the geographically nearest mirror, there isn't necessarily any value in that, since the nearest mirror can actually be much slower than one further away.

Bottom line: I can't come up with a good way to do this that works well for everyone's scenarios. Accordingly, I think it's a good idea to leave this entirely in user control but create a small program that users can run at will, or by default in the background at startup, to detect the fastest mirror and place that mirror into their apt configs. It would only take several hours to write and I could do that and then make the program available in the repository, and include it by default in future releases.

Thoughts?

_________________
UberStudent Must Become Self-Sustaining. If UberStudent and my dedicated support of it has benefited you, it's important to make a donation. Thanks!

The UberStudent Headquarters:
    * Mobo: MSI 870-G45
    * Processor: AMD Phenom II X4 3.4GHz
    * RAM: G.SKILL 8GB DDR3 1600
    * Graphics: MSI NVIDIA GeForce GTX 750 Ti
    * Drive: OCZ Vertex SSD
    * Case: Rosewill ATX Mid Tower
    * Monitors: 2 x Dell UltraSharp
    * With UberStudent 4.3 Development Build
You should build your own computers, too!


Top
 Profile  
 
PostPosted: Fri Jun 10, 2016 1:15 pm 
Offline
UberStudent Forum Helper

Joined: Fri Oct 18, 2013 12:48 pm
Posts: 54
Location: University of South Florida, Tampa, FL
How about we get the result of the command in the first post as part of UberStudentUpdate whenever, let's say, an update download is required larger than a specific threshold. Until that threshold is exceeded again, the updater can use that mirror.

Since uberstudentsources reconfigures the base source to be us.archive.ubuntu.com from time to time, now that I returned to Turkey from Florida for the summer, us.archive.ubuntu.com is killing me and this makes me wanna remove uberstudentsources completely. But instead of removing it, I thought, why not try to make it better :)

_________________
Berker Peköz
PhD Student in Electrical Engineering
University of South Florida at Tampa


Top
 Profile  
 
PostPosted: Sat Jun 11, 2016 6:30 am 
Offline
UberStudent Founder and Lead Developer
User avatar

Joined: Sun Nov 11, 2012 8:54 am
Posts: 951
Location: UberStudent Headquarters
Well, running the curl string you posted in the uberstudentsource postinst file would seem a good solution. In this way, one can simply reinstall that package to get the fastest mirror if/when they move.

Let me run some tests on this. I'll post again in a day or so.

_________________
UberStudent Must Become Self-Sustaining. If UberStudent and my dedicated support of it has benefited you, it's important to make a donation. Thanks!

The UberStudent Headquarters:
    * Mobo: MSI 870-G45
    * Processor: AMD Phenom II X4 3.4GHz
    * RAM: G.SKILL 8GB DDR3 1600
    * Graphics: MSI NVIDIA GeForce GTX 750 Ti
    * Drive: OCZ Vertex SSD
    * Case: Rosewill ATX Mid Tower
    * Monitors: 2 x Dell UltraSharp
    * With UberStudent 4.3 Development Build
You should build your own computers, too!


Top
 Profile  
 
PostPosted: Mon Jun 13, 2016 4:36 am 
Offline
UberStudent Founder and Lead Developer
User avatar

Joined: Sun Nov 11, 2012 8:54 am
Posts: 951
Location: UberStudent Headquarters
This seems to work flawlessly as part of the postinst script for the package uberstudentsources:
Code:
#!/bin/bash
##############################################################
# MIRROR SPEED TEST AND FASTEST MIRROR PLACEMENT
# Ref: http://uberstudent.com/phpBB/viewtopic.php?f=15&t=1195
##############################################################

# Create this directory, it's required for the below actions.
mkdir /tmp/aptmirror.test

# Run the mirror speed test string and write the result to a temp file
# we read from below, and add a terminal progress bar.
curl -s http://mirrors.ubuntu.com/mirrors.txt | xargs -n1 -I {} sh -c 'echo `curl -r 0-102400 -s -w %{speed_download} -o /dev/null {}/ls-lR.gz` {}' |sort -g -r |head -1| awk '{ print $2  }' > /tmp/aptmirror.test/test & PID=$!

# Inform the user
echo
echo "Determining the fastest software repository mirror for your location."
echo
echo "Please be patient. This may take a few minutes. Doing this now will"
echo "save you time in the future."
echo

# The printf bracket here is just for ascetic reasons.
printf "["

# While the above curl process is running we here give a progress bar in
# the terminal to inform the user about what is going on.
while kill -0 $PID 2> /dev/null; do
    printf  "▓"
    sleep 1
done

# The printf bracket here is just for ascetic reasons.
printf "]"

# File editing magic here that is perfect for when precision matters.
# Ref: http://unix.stackexchange.com/a/164284
FILE1=$(</tmp/aptmirror.test/test)
FILE2=$(</etc/apt/sources.list.d/official-package-repositories.list)
echo "${FILE2//marker/$FILE1}" > /tmp/aptmirror.test/package

sleep 1

# Most users don't have source files added to their apt configs, so we only
# run this if they do. They are not enabled by default in UberStudent.
if [ -f /etc/apt/sources.list.d/official-source-repositories.list ]; then
     FILE3=$(</etc/apt/sources.list.d/official-source-repositories.list)
     echo "${FILE3//marker/$FILE1}" > /tmp/aptmirror.test/source
     sleep 1
     mv /tmp/aptmirror.test/source /etc/apt/sources.list.d/official-source-repositories.list
     sleep 1
fi

# Required for every user case. We here place the list file we made
# containing the fastest mirror and clean up the temp file(s) we made.
mv /tmp/aptmirror.test/package /etc/apt/sources.list.d/official-package-repositories.list
sleep 1
rm -rf /tmp/aptmirror.test
sleep 1

# Inform the user
echo
echo "Found the fastest mirror! It's been placed into your apt configurations."
echo
echo "If in the future you geographically move a long distance away from your"
echo "current location, re-install this package and it will again find and set"
echo "the fastest mirror for you!"
echo
echo "Using the program Software Sources, you can restore the default U.S.-based"
echo "repository."
echo
sleep 1

##############################################################
I'll test it some more over the next couple of days and if all is well, I'll release a new package. You can test it too, if you wish. The proposed new package is at http://uberstudent.net/incoming-socrate ... .6_all.deb

_________________
UberStudent Must Become Self-Sustaining. If UberStudent and my dedicated support of it has benefited you, it's important to make a donation. Thanks!

The UberStudent Headquarters:
    * Mobo: MSI 870-G45
    * Processor: AMD Phenom II X4 3.4GHz
    * RAM: G.SKILL 8GB DDR3 1600
    * Graphics: MSI NVIDIA GeForce GTX 750 Ti
    * Drive: OCZ Vertex SSD
    * Case: Rosewill ATX Mid Tower
    * Monitors: 2 x Dell UltraSharp
    * With UberStudent 4.3 Development Build
You should build your own computers, too!


Top
 Profile  
 
PostPosted: Mon Jun 13, 2016 6:46 am 
Offline
UberStudent Forum Helper

Joined: Fri Oct 18, 2013 12:48 pm
Posts: 54
Location: University of South Florida, Tampa, FL
Worked perfect on a clean Heraclitus install I just few weeks ago on the university computer!

Lemme also try it on my laptop which was upgraded from Socrates and my desktop back home which was upgraded from Aristotle :D

_________________
Berker Peköz
PhD Student in Electrical Engineering
University of South Florida at Tampa


Top
 Profile  
 
PostPosted: Mon Jun 13, 2016 8:10 pm 
Offline
UberStudent Founder and Lead Developer
User avatar

Joined: Sun Nov 11, 2012 8:54 am
Posts: 951
Location: UberStudent Headquarters
I've just now released a new version of uberstudentsources that incorporates the feature request asked for here. I tested the proposed package repeatedly on both upgraded installations and fresh ones and am confident.

The new version of uberstudentsources will also install a small new program/script, set-fastest-mirror, as a dependency. People can run it any time to get and set the fastest mirror into their apt configurations if they move.

Berker, you'll need to reinstall uberstudentsources to get some minor code improvements and the new dependency, set-fastest-mirror.
Code:
sudo apt-get update
sudo apt-get install --reinstall uberstudentsources

_________________
UberStudent Must Become Self-Sustaining. If UberStudent and my dedicated support of it has benefited you, it's important to make a donation. Thanks!

The UberStudent Headquarters:
    * Mobo: MSI 870-G45
    * Processor: AMD Phenom II X4 3.4GHz
    * RAM: G.SKILL 8GB DDR3 1600
    * Graphics: MSI NVIDIA GeForce GTX 750 Ti
    * Drive: OCZ Vertex SSD
    * Case: Rosewill ATX Mid Tower
    * Monitors: 2 x Dell UltraSharp
    * With UberStudent 4.3 Development Build
You should build your own computers, too!


Top
 Profile  
 
PostPosted: Sat Jul 23, 2016 8:45 am 
Offline
UberStudent Founder and Lead Developer
User avatar

Joined: Sun Nov 11, 2012 8:54 am
Posts: 951
Location: UberStudent Headquarters
I have a test version of uberstudentupdate that automatically tests for and places the fastest mirror every 30 days, or whichever day after 30 that you are connected to the internet.

The startup script adds a subprocess:
Code:
#!/usr/bin/python

import os, commands, subprocess

cmdline = commands.getoutput("cat /proc/cmdline")

if ((not "boot=casper" in cmdline) and (not "boot=live" in cmdline)):

   subprocess.Popen(["sudo", "/usr/lib/uberstudent/uberstudentUpdate/apt-mirror-check"]).wait()

   os.system("/usr/lib/uberstudent/uberstudentUpdate/uberstudentUpdate.py")

The subprocess script the python startup script calls, apt-mirror-check, is:
Code:
#!/bin/bash

##############################################################
# MIRROR SPEED TEST AND FASTEST MIRROR PLACEMENT
# Ref: http://uberstudent.com/phpBB/viewtopic.php?f=15&t=1195
##############################################################

# For testing file age, make it old: touch -a -m -t 201512180130.09 /tmp/aptmirrortest/age

#if test `find "/tmp/aptmirrortest/age" -mtime +30` ; then

# If the find command *does not equal* (!=) empty, i.e., if it's not over 30 days old.
# This should use less resources then the *if test* line, above.
if [ "$(find /tmp/aptmirrortest/age -mtime +30)" != "" ]; then
   # Needed sleep to ensure network is up and all other start programs have started
   sleep 15
   # Test if decently internet-connected, and if so, proceed, or else pass to line 135 "else"
   wget -q --tries=1 --timeout=15 http://uberstudent.net -O /tmp/test.idx &> /dev/null
      if [ -s /tmp/test.idx ]; then
         echo "Mirror is over 30 days old. Taking actions...."

         # Notify the user.
         notify-send -u critical -t 0 --icon /usr/share/pixmaps/uberstudentupdate.png "IMPORTANT!" \
"<b>UberStudent Update's software mirrors are more than 30 days old.</b>

The mirrors are currently being updated in the background. It should take about 5 minutes.

<i>Do not try to install or remove any software until this process has finished</i>. You'll be notified when it's done.

<b>You must click this message to dismiss it.</b>"

          # Prevent a catastrophe by locking /var/lib/dpkg/lock so apt can't run while this does
          exec {lock_fd}>/var/lib/dpkg/lock || exit 1
          flock -n "$lock_fd" || { echo "ERROR: flock() failed." >&2; exit 1; }
         
         if [ ! -d /tmp/aptmirrortest ]; then
            mkdir /tmp/aptmirrortest
         fi

         # Create this directory, it's required for the below actions.
         if [ -d /tmp/aptmirror.test ]; then
            rm -rf /tmp/aptmirror.test
            mkdir /tmp/aptmirror.test
         elif [ ! -d /tmp/aptmirror.test ]; then
            mkdir /tmp/aptmirror.test
         fi

         sleep 1

         # Run the mirror speed test string and write the result to a temp file we read from below.
         # This is the magic line that tests the mirror speeds and selects the fastest one,
         # which is from http://askubuntu.com/a/719551/274446
                       
         curl -s http://mirrors.ubuntu.com/mirrors.txt | xargs -n1 -I {} sh -c 'echo `curl -r 0-102400 -s -w %{speed_download} -o /dev/null {}/ls-lR.gz` {}' |sort -g -r |head -1| awk '{ print $2  }' > /tmp/aptmirror.test/test

         sleep 1

         # Below, take actions contingent on the user's installation and apt state:

         #
         # Pre-4.3
         #

         # If the old xorg version and user DOES NOT have sources list enabled
         if [ ! -f /usr/share/doc/xserver-xorg-lts-vivid/copyright ] && [ ! -f /etc/apt/sources.list.d/official-source-repositories.list ]; then
            # File editing magic here that is perfect for when precision matters.
            # Ref: http://unix.stackexchange.com/a/164284
            FILE1=$(</tmp/aptmirror.test/test)
            FILE2=$(</usr/share/uberstudent/uberstudentsources/pre-heraclitus/official-package-repositories.list)
            echo "${FILE2//marker/$FILE1}" > /tmp/aptmirror.test/package
            mv /tmp/aptmirror.test/package /etc/apt/sources.list.d/official-package-repositories.list

         # If the old xorg version and user DOES have sources list enabled
         elif [ ! -f /usr/share/doc/xserver-xorg-lts-vivid/copyright ] && [ -f /etc/apt/sources.list.d/official-source-repositories.list ]; then
            # File editing magic here that is perfect for when precision matters.
            # Ref: http://unix.stackexchange.com/a/164284
            FILE1=$(</tmp/aptmirror.test/test)
            FILE2=$(</usr/share/uberstudent/uberstudentsources/pre-heraclitus/official-package-repositories.list)
            FILE3=$(</usr/share/uberstudent/uberstudentsources/pre-heraclitus/official-source-repositories.list)
            echo "${FILE2//marker/$FILE1}" > /tmp/aptmirror.test/package
            echo "${FILE3//marker/$FILE1}" > /tmp/aptmirror.test/source
            mv /tmp/aptmirror.test/package /etc/apt/sources.list.d/official-package-repositories.list
            mv /tmp/aptmirror.test/source /etc/apt/sources.list.d/official-source-repositories.list

         #
         # 4.3
         #

         # If the new xorg version and user DOES NOT have sources.list enabled, we add heraclitus-backports to the default apt configs
         elif [ -f /usr/share/doc/xserver-xorg-lts-vivid/copyright ] && [ ! -f /etc/apt/sources.list.d/official-source-repositories.list ]; then
            # File editing magic here that is perfect for when precision matters.
            # Ref: http://unix.stackexchange.com/a/164284
            FILE1=$(</tmp/aptmirror.test/test)
            FILE2=$(</usr/share/uberstudent/uberstudentsources/heraclitus/official-package-repositories.list)
            echo "${FILE2//marker/$FILE1}" > /tmp/aptmirror.test/package
            mv /tmp/aptmirror.test/package /etc/apt/sources.list.d/official-package-repositories.list

         # If the new xorg version and user DOES have sources.list enabled
         elif [ -f /usr/share/doc/xserver-xorg-lts-vivid/copyright ] && [ -f /etc/apt/sources.list.d/official-source-repositories.list ]; then
            # File editing magic here that is perfect for when precision matters.
            # Ref: http://unix.stackexchange.com/a/164284
            FILE1=$(</tmp/aptmirror.test/test)
            FILE2=$(</usr/share/uberstudent/uberstudentsources/heraclitus/official-package-repositories.list)
            FILE3=$(</usr/share/uberstudent/uberstudentsources/heraclitus/official-source-repositories.list)
            echo "${FILE2//marker/$FILE1}" > /tmp/aptmirror.test/package
            echo "${FILE3//marker/$FILE1}" > /tmp/aptmirror.test/source
            mv /tmp/aptmirror.test/package /etc/apt/sources.list.d/official-package-repositories.list
            mv /tmp/aptmirror.test/source /etc/apt/sources.list.d/official-source-repositories.list
         fi

         sleep 1

         # Silently update apt with the new mirror.
         apt-get update > /dev/null
         sleep 1

         # Notify the user.
         FILE=$(</tmp/aptmirror.test/test)

         notify-send -u critical -t 0 --icon /usr/share/pixmaps/uberstudentupdate.png "SOFTWARE MIRRORS HAVE BEEN UPDATED!" \
"<b>The fastest possible mirror was determined to be:</b>

    • $FILE

<b>The new mirror has been set into your apt software configurations.</b>

<b>You must click this message to dismiss it.</b>"

         # Clean up the temp file(s) we made.
         if [ -d /tmp/aptmirror.test ]; then
            rm -rf /tmp/aptmirror.test
         fi

         if [ -s /tmp/test.idx ]; then
            rm /tmp/test.idx
         fi

         if [ -f /tmp/aptmirrortest/age ]; then
            rm /tmp/aptmirrortest/age
            echo "This file is needed by apt-mirror-test. Please leave it here." > /tmp/aptmirrortest/age
         fi
      fi
   else
      # If either /tmp/aptmirrortest/age is younger than 30 days old or there is no decent internet connection,
      # we pass the whole script to do nothing, the ":". Comment in echo and comment out : for CLI testing.
      :
      # echo "Mirror is younger than 30 days old, so doing nothing."
   fi

The script apt-mirror-check is set in /etc/sudoers.d/ to run without root requirements, something neccesary in this case.

What do you think about these additions? It works without flaw on my system, but I worry that this might be too intrusive.

_________________
UberStudent Must Become Self-Sustaining. If UberStudent and my dedicated support of it has benefited you, it's important to make a donation. Thanks!

The UberStudent Headquarters:
    * Mobo: MSI 870-G45
    * Processor: AMD Phenom II X4 3.4GHz
    * RAM: G.SKILL 8GB DDR3 1600
    * Graphics: MSI NVIDIA GeForce GTX 750 Ti
    * Drive: OCZ Vertex SSD
    * Case: Rosewill ATX Mid Tower
    * Monitors: 2 x Dell UltraSharp
    * With UberStudent 4.3 Development Build
You should build your own computers, too!


Top
 Profile  
 
PostPosted: Mon Jul 25, 2016 4:42 pm 
Offline
UberStudent Forum Helper

Joined: Fri Oct 18, 2013 12:48 pm
Posts: 54
Location: University of South Florida, Tampa, FL
I find it perfectly useful for those who want it, but doing this periodically might be unwanted for some. I vote we add this as another package that is recommended by the set-fastest-mirror, like set-fastest-mirror-periodically, but not a dependency to the original package :-)
I'm trying it out on my Socrates upgraded laptop when I go home :)

_________________
Berker Peköz
PhD Student in Electrical Engineering
University of South Florida at Tampa


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 13 posts ]  Go to page 1, 2  Next

Click Here to Get Inexpensive UberStudent Discs


Click here to get UberStudent discs delivered to your mailbox
Delivered Directly to Your Mailbox!



All times are UTC


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group