How to Make an Offline Repository of Your Own Ubuntu System

Introduction

Sometimes when you need to replicate an Ubuntu system to another computer, the usual step will be:

1. Install Ubuntu using Installer CD/ Live CD
2. Reboot
3. Connect to the Internet
4. Update
5. Installing additional packages that you need

However, if it occurs that you need to install to another computer at another place with uncertainty of the availability and/or reliability of the Internet connection over there, one would wonder if you could prepare those updates and additional packages yourself in an offline repository.

Assuming the additional packages is the same with your Ubuntu System, this howto will explain how to make such offline repository.

Creating Offline Repository

There are 4 steps to setting up a simple repository for yourself

1. Import the packages in a directory
2. Scan the packages and create a file apt-get update can read
3. Replicate the repository to portable media
4. Add a line to your sources.list pointing at your repository

Import Packages

In your Ubuntu System, the .deb files you have installed are stored in /var/cache/apt/archives/

To replicate these files, first create the directory you want to paste into. For this example, we'll use ~/mydebs.

mkdir ~/mydebs

Then, replicate the files using rsync

rsync -upt --progress /var/cache/apt/archives/*.deb ~/mydebs

Sometimes some files you have installed is not cached. If so, you need to retrieve them from the internet. First we need to generate the list of files to be downloaded.

sudo apt-get install $(dpkg -l|grep ^ii|awk '{print $2}') --reinstall --print-uris \

-y |awk '{print $1}'|sed "s/'//g"|grep tp:// > downlist

Then download it using wget

wget -ci downlist -P ~/mydebs

Scan Packages

To be included in apt-get process, we net to generate packages list of those .deb files.

First go into mydebs directory.

cd ~/mydebs

then generate Packages.gz using dpkg-scanpackages

dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz

Replicate mydebs Directory to Portable Media

The repository can now be replicated to portable media, such as DVD-ROM or External Hard Drive. Assuming you are using external harddrive with volume name REPO:

rsync -a ~/mydebs /media/REPO/mydebs

Then, after finishing installing the new system, restore the directory to your new home folder

rsync -a /media/REPO/mydebs ~/mydebs

Adding mydebs Directory as Repository Source

To use these repository, these mydebs directory then must be included in the Repository Source file on your new system.

sudo su -pc 'echo deb file:$HOME/mydebs ./ >> /etc/apt/sources.list'

sudo apt-get update

Your repository is now ready to use

Counting Down on Intrepid Ibex

I just can't wait :-D

It works!!!

One thing I just realized after upgrading to Edgy, all hotkey in my desknote, which by the way is an ECS A907, works out of the box. No hacking, no tweaking, no user intervention. The volume manager key works, the "internet" key fires up Firefox and the "mail" key opens Evolution.

Now...if only I could make use of that f***in' flyin' windows key ...

Edgy upgrade fail on acpid

I just upgraded my dapper into edgy. everything went fine but acpid and
everything that depends on it.
After several hours of googling... I found the answer from launchpad.
The workaround are:
1. Shutdown the acpid (I removed it instead)
2. sudo killall hald
3. sudo dpkg --configure -a
4. sudo apt-get install acpid

Perhaps 4 months is a bit too short for a release cycle of a distro as
massive as Ubuntu...

Mounting Remote Directories using SSHFS

Due to limited harddisk capacity, I stored my edgy repository in an external USB hard drive.
To access it, I mount it using sshfs. Here is how I did it in Ubuntu 6.06.

$ sudo apt-get install sshfs

$ sudo modprobe fuse

$ mkdir /somewhere/localfolder/

$ sshfs -o allow_other user@remotehost:/somewhere/somefolder /somewhere/localfolder

To unmount it just use the normal umount command

$ sudo umount /somewher/localfolder

Now I have edgy repo, shall I start upgrading now... ?
I heard some bad rumors about upgrading dapper to edgy.
I think I need to do some backup first.

screen bash

When executing application using screen, I found that if the application went bad, we couldn't find out the error unless we specificy to log it. My debmirror script sometimes fail and I don't know if it is fail unless I repeat the script again.

Hence, bash must go first :-)

It's also quite handy to hide whatever I am doing on the screen as I only need to press C-a d to detach it.

pmount vs perl

After modifying debmirror, i copied it into a folder in my mounted portable hard drive.
The remaining space is quite large, so I put dapper & edgy as the target and start executing the script.
*bam*
myscript: /usr/bin/perl: bad interpreter: Permission denied

what the h###?!

I am absolutely sure I have no problem with permissions as the same script also copied into the parent directory of
a running ubuntu local ftp repository, and it works.

After checking every possible permission problem I encountered, I finally came into mount permission.
I mounted the portable hd using pmount (it's a usb-connected external hd) and it turnout that the default parameter
used by pmount include a noexec option, therefore, when I mounted it using the classic mount, the script worked.

Now, one question remain...
How come a perl script is considered a binary executable, while bash script is not?
Hmm...

Blogging via mutt

In blogspot, one can post his/her blog via mail. I like this feature because of two reasons :
1. The built in blogspot editor is terribly *cough*bloated*cough* slow
2. Due to aging hardware, opening firefox in my laptop takes zillion years
The problem is, the reason #2 is also the reason why I hate opening evolution in my laptop.

Few days ago, I installed mutt. The setup is not perfect yet, but I found myself quite enjoying it.
It loads extremely fast... that I only keep evolution just for reading html-formatted-mail sent to me.
I can also deal with junk mails much better :-)

Patching debmirror...

I just realized that debmirror has a weakness in its design. Debmirror is designed to download the packages lists, selecting correct packages to be downloaded, then and only then,  download the packages. However, the packages selection, is somewhat long and tedious. Even worse, there is no progress indicator to tell us how much time left for package selection process to start downloading those damn packages.

I wonder if I can modify it to make it faster.

* searching google with keywords: perl tutorial*

apt-get moo

$ apt-get moo

        (__)
        (oo)
  /------\/
 / |    ||
*  /\---/   ~~   ~~
...."Have you mooed today?"...

Partitioning USB Drive from Linux

USB drive is indeed can be partitioned.
What you need to do is:
1. Unmount USB drive,
2. Disable auto-mounting
3. gparted /dev/sda

As I haven't been able to tame the automounting feature of Kubuntu, I used
Gparted-Live and partitioned from there. When I partitioned my USB drive in
Kubuntu, it persistently automounted the first partition (sda1) after it had
been created, even the whole partitioning process hadn't been finished yet.

Thus, it's the value of having a pocket-CD of Gparted-Live everywhere you go.

Now I have a 128MiB USB pen-drive partitioned into 80s MiB FAT16 and 30s MiB EXT2

And...

250 GiB USB Harddrive partitioned into 40s GiB FAT32, and two 100s GiB EXT3

Now ... I am backing-up everything :-D

Managing Wireless Equipment (802.11x) using ap-utils

Being maintaining snmp-based nms for months, I've just found this tool few
hours ago. With ap-utils, I can manage my SmartBridge devices natively from
Linux. Ironically, I've been struggling with wine for years just to configure
my WiFi equipment, and even installing windows 98 via Qemu just to have the
wireless configuration software installed.

To install it in Ubuntu :

sudo apt-get install ap-utils

then

ap-config

The menu inside ap-utils is self-explanatory.
If only this tools can manage other proprietary wi-fi devices, (alvarion,
canopy, trango, etc) it will be much better.

QEMU in Ubuntu Dapper

Should you need another OS emulated within your system, QEMU is the Free Software that do the job well.
In Ubuntu 6.06, you can have the version 0.8.0 packaged in the universe repo.

To use it, just install it using apt-get (or synaptic if you prefer):

$ sudo apt-get install qemu

then create the virtual hard drive using qemu-img

$ qemu-img create mydisk 3G

to install an OS into your emulated hard drive (using CD-ROM installer)

$ qemu -hda mydisk -cdrom /dev/cdrom -boot d

Qemu can be emulating at higher speed when it uses its accelerator module named Kqemu. However Kqemu is a proprietary software (although it'll cost 0 in price). To install the accelerator, you can use the method posted here

"Crippled" Repo

After hours of trying and failing net-install Ubuntu 6.06 (Dapper Drake) from my local repository, I finally found out what's missing from it.
I did not include main/debian-installer in my debmirror script!
Thus, the repository is only good for installing packages into up & running Ubuntu system.
With less than 3 GB disk space remaining, I don't think I can afford to add main/debian-installer into my local repo.
Thus, having CD-installer ready is a must.

Proxy Traceroute

When I need to ensure that an IP address is properly routed and reachable, I often use the traceroute service of www.nwtools.com. However when I have to query frequently, the slowness of firefox on my laptop is such a hinderance.
To keep myself from hammering down my laptop, I create shell script to do the task using elinks and sed. Here is my code:

#!/bin/sh
elinks -dump --no-references --no-numbering \ "http://www.nwtools.com/default.asp?prog=trace&host=$1"|sed '1,21d'|sed -n -e :a -e '1,7!{P;N;D;};N;ba'

I saved the script as proxytrace.sh and using it with the traceroute target as the only parameter.

example:
$ sh proxytrace.sh www.blahblahblah.com
or
$ sh proxytrace.sh 123.234.123.234

This way is much faster, really.

A Geek's Bedroom

Kubuntu, with ATI RADEON 9250 Chipset, and TV-OUT, means Multimedia Playback in the Bedroom!
Now...if I can build MythTV on it...

My sysadmin toolbox

Inspired by linux.com :-)

For my day to day network administration, I use these apps frequently:

ssh,telnet,bash,coreutils,rsync,scp,elinks,wget,nano....if you don't know what
are these apps for, then probably you could skip this
*cough*article*cough* :-)
For serial console access, I use minicom. I love configuring my huawei box
with minicom using transparent konsole, it's so damn pretty.
For quick traffic monitoring via snmp poll, I use ifstat. If only I could
figure out how to parse the capture result into rrd files, I would make it
even more useful.
For more comperehensive network provisioning, JFFNMS is what I use. I just
wonder how to reduce its resource usage. I'll probably give nagios a shot
someday.

The others will be most of what GNOME & KDE offers....

Aliases

Going into bash, aliases make my life easier, really!
I kept adding these aliases in wherever .bashrc I have

someone@somewhere:~$ alias
alias acs='apt-cache search'
alias acso='apt-cache show'
alias dusk='du -s -k -c * | sort -rn'
alias greph='history | grep '
alias ls='ls --color=auto'
alias nsl='netstat -alnp --protocol=inet | grep -v CLOSE_WAIT | cut -c-6,21-94
| tail +2'
alias sadug='sudo apt-get dist-upgrade'
alias sapi='sudo apt-get install'
alias sapu='sudo apt-get update'
alias sapup='sudo apt-get upgrade'
someone@somewhere:~$

As you can see... it's very ubuntu, right? :-)
I think I need to add some more...
Any idea ?

Messing up with debmirror

From debmirror manpages:

DEBMIRROR(1) User Contributed Perl Documentation DEBMIRROR(1)

NAME
debmirror - Debian partial mirror script, with ftp, http, hftp or rsync
and package pool support

SYNOPSIS
debmirror [options] mirrordir

DESCRIPTION
This program downloads and maintains a partial local Debian mirror. It
can mirror any combination of architectures, distributions, and sec‐
tions. Files are transferred by ftp, and package pools are fully sup‐
ported. It also does locking and updates trace files.

To support package pools, this program mirrors in three steps.

1. download Packages and Sources files
First it downloads all Packages and Sources files for the subset of
Debian it was instructed to get.

2. clean up unknown files
Any files and directories on the local mirror that are not in the
list are removed.

3. download everything else
The Packages and Sources files are scanned, to build up a list of
all the files they refer to. A few other miscellaneous files are
added to the list. Then the program makes sure that each file in
the list is present on the local mirror and is up-to-date, using
file size (and optionally md5sum) checks. Any necessary files are
downloaded.


I had never paid attention step two before I tried to add breezy-updates and breezy-security, into my existing Ubuntu Breezy repository. I executed debmirror without putting breezy in the dist parameter (instead, only breezy-updates and breezy-security), hence, I lost 11 Gigs of data.

Mental note: RTFM RTFM RTFM

Scripting Addiction

Since I can do some bash scripting, I fell like I can't stop.
I even try to automate anything I usually do by scripting it.
Hmm... should I dive into python and made my own NMS?
I'm quite dissappointed with JFFNMS performance, btw.
Probably, I should take a look at ifstat source code and learn from it, and then make my own lightweight network monitoring system....
Hmm... scripting is really really really addictive....