How to Make an Offline Repository of Your Own Ubuntu System

Introduction

Sometimes when you need to replicate an Ubuntu system to another computer, the usual step will be:

1. Install Ubuntu using Installer CD/ Live CD
2. Reboot
3. Connect to the Internet
4. Update
5. Installing additional packages that you need

However, if it occurs that you need to install to another computer at another place with uncertainty of the availability and/or reliability of the Internet connection over there, one would wonder if you could prepare those updates and additional packages yourself in an offline repository.

Assuming the additional packages is the same with your Ubuntu System, this howto will explain how to make such offline repository.

Creating Offline Repository

There are 4 steps to setting up a simple repository for yourself

1. Import the packages in a directory
2. Scan the packages and create a file apt-get update can read
3. Replicate the repository to portable media
4. Add a line to your sources.list pointing at your repository

Import Packages

In your Ubuntu System, the .deb files you have installed are stored in /var/cache/apt/archives/

To replicate these files, first create the directory you want to paste into. For this example, we'll use ~/mydebs.

mkdir ~/mydebs

Then, replicate the files using rsync

rsync -upt --progress /var/cache/apt/archives/*.deb ~/mydebs

Sometimes some files you have installed is not cached. If so, you need to retrieve them from the internet. First we need to generate the list of files to be downloaded.

sudo apt-get install $(dpkg -l|grep ^ii|awk '{print $2}') --reinstall --print-uris \

-y |awk '{print $1}'|sed "s/'//g"|grep tp:// > downlist

Then download it using wget

wget -ci downlist -P ~/mydebs

Scan Packages

To be included in apt-get process, we net to generate packages list of those .deb files.

First go into mydebs directory.

cd ~/mydebs

then generate Packages.gz using dpkg-scanpackages

dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz

Replicate mydebs Directory to Portable Media

The repository can now be replicated to portable media, such as DVD-ROM or External Hard Drive. Assuming you are using external harddrive with volume name REPO:

rsync -a ~/mydebs /media/REPO/mydebs

Then, after finishing installing the new system, restore the directory to your new home folder

rsync -a /media/REPO/mydebs ~/mydebs

Adding mydebs Directory as Repository Source

To use these repository, these mydebs directory then must be included in the Repository Source file on your new system.

sudo su -pc 'echo deb file:$HOME/mydebs ./ >> /etc/apt/sources.list'

sudo apt-get update

Your repository is now ready to use

Counting Down on Intrepid Ibex

I just can't wait :-D

Extracting Similarity Between Two (Text) Files

When you need to compare difference between two files, you have diff.
When you need to find similarity between such files, as far as I know, you need to devise your own script.
This is my oneliner script for such need (assuming files to be compared named file-01 & file-02):

for ((i=1;i<=$(wc -l file-01|awk '{print $1}');i+=1)); do grep $(awk -v a=$i 'NR==a' file-01) file-02 ; done

Ugly, but working...

Reverse Proxy with Apache

Sometimes, when we need to configure network devices within LAN (via http), we need to do it on site due to private addressing. While making the devices directly accessible from the Internet is a big no-no, there is another way to connect remotely and securely: Reverse HTTP Proxy (with Apache).

Apache has proxy-related modules to enable this operation. To get it working, you need to enable this modules:
- proxy
- proxy_http

Using Ubuntu 7.10, this modules can be loaded simply by typing:

sudo a2enmod proxy
sudo a2enmod proxy_http
sudo /etc/init.d/apache2 restart

To forward any http request to your internal address, you need to add the destination url into your virtual host configuration, i.e:

sudo nano /etc/apache2/sites-available/yourvirtualhostconfig

then add this line:

ProxyPass /your-public-url/ http://your-private-address

enable your site by typing:

sudo a2ensite yourvirtualhostconfig
sudo /etc/init.d/apache2/restart

By default, in Ubuntu 7.10 all proxy access are denied. Therefore, you
need to white list your address into the proxy.conf file

sudo nano /etc/apache2/mods-available/proxy.conf

change the config to allow your source address i.e:

<Proxy *>
AddDefaultCharset off
Order deny,allow
Deny from all
Allow from 203.153.240.197
</Proxy>

Then reload the configurations

sudo a2enmod proxy
sudo /etc/init.d/apache2 restart

That's it folks. Enjoy working on your devices remotely, securely.
(It is not so secure as the traffic is not encrypted. What I mean by
secure, is that I can hide the private address from any port scan.
Therefore, what is deemed secure by me, is not necessarily secure by
your definition. This is MY blog, so no complain!!!)

Connecting to XL 3G via Sony Ericsson W850i

While it is not so newbie friendly, I never thought that configuring Internet connectivity using my cellular phone will be a straightforward process. All I did just edit the /wtc/wvdial.conf into this:

Init1 = ATZ
Init2 = ATQ0 V1 E1 S0=0 &C1 &D2 +FCLASS=0

Modem Type = USB Modem
Baud = 460800
Modem = /dev/ttyACM1
ISDN = 0
Phone = *99***1#
Phone1 = *99***2#
Password = proxl
Username = xlgprs

then, I connect by calling wvdial (with root privillege)

sudo wvdial

Just be very cautious with the cost as it is charged per KB!
Enjoy...

It works!!!

One thing I just realized after upgrading to Edgy, all hotkey in my desknote, which by the way is an ECS A907, works out of the box. No hacking, no tweaking, no user intervention. The volume manager key works, the "internet" key fires up Firefox and the "mail" key opens Evolution.

Now...if only I could make use of that f***in' flyin' windows key ...

Edgy upgrade fail on acpid

I just upgraded my dapper into edgy. everything went fine but acpid and
everything that depends on it.
After several hours of googling... I found the answer from launchpad.
The workaround are:
1. Shutdown the acpid (I removed it instead)
2. sudo killall hald
3. sudo dpkg --configure -a
4. sudo apt-get install acpid

Perhaps 4 months is a bit too short for a release cycle of a distro as
massive as Ubuntu...

Mounting Remote Directories using SSHFS

Due to limited harddisk capacity, I stored my edgy repository in an external USB hard drive.
To access it, I mount it using sshfs. Here is how I did it in Ubuntu 6.06.

$ sudo apt-get install sshfs

$ sudo modprobe fuse

$ mkdir /somewhere/localfolder/

$ sshfs -o allow_other user@remotehost:/somewhere/somefolder /somewhere/localfolder

To unmount it just use the normal umount command

$ sudo umount /somewher/localfolder

Now I have edgy repo, shall I start upgrading now... ?
I heard some bad rumors about upgrading dapper to edgy.
I think I need to do some backup first.

screen bash

When executing application using screen, I found that if the application went bad, we couldn't find out the error unless we specificy to log it. My debmirror script sometimes fail and I don't know if it is fail unless I repeat the script again.

Hence, bash must go first :-)

It's also quite handy to hide whatever I am doing on the screen as I only need to press C-a d to detach it.

pmount vs perl

After modifying debmirror, i copied it into a folder in my mounted portable hard drive.
The remaining space is quite large, so I put dapper & edgy as the target and start executing the script.
*bam*
myscript: /usr/bin/perl: bad interpreter: Permission denied

what the h###?!

I am absolutely sure I have no problem with permissions as the same script also copied into the parent directory of
a running ubuntu local ftp repository, and it works.

After checking every possible permission problem I encountered, I finally came into mount permission.
I mounted the portable hd using pmount (it's a usb-connected external hd) and it turnout that the default parameter
used by pmount include a noexec option, therefore, when I mounted it using the classic mount, the script worked.

Now, one question remain...
How come a perl script is considered a binary executable, while bash script is not?
Hmm...

Blogging via mutt

In blogspot, one can post his/her blog via mail. I like this feature because of two reasons :
1. The built in blogspot editor is terribly *cough*bloated*cough* slow
2. Due to aging hardware, opening firefox in my laptop takes zillion years
The problem is, the reason #2 is also the reason why I hate opening evolution in my laptop.

Few days ago, I installed mutt. The setup is not perfect yet, but I found myself quite enjoying it.
It loads extremely fast... that I only keep evolution just for reading html-formatted-mail sent to me.
I can also deal with junk mails much better :-)

Patching debmirror...

I just realized that debmirror has a weakness in its design. Debmirror is designed to download the packages lists, selecting correct packages to be downloaded, then and only then,  download the packages. However, the packages selection, is somewhat long and tedious. Even worse, there is no progress indicator to tell us how much time left for package selection process to start downloading those damn packages.

I wonder if I can modify it to make it faster.

* searching google with keywords: perl tutorial*

apt-get moo

$ apt-get moo

        (__)
        (oo)
  /------\/
 / |    ||
*  /\---/   ~~   ~~
...."Have you mooed today?"...

Partitioning USB Drive from Linux

USB drive is indeed can be partitioned.
What you need to do is:
1. Unmount USB drive,
2. Disable auto-mounting
3. gparted /dev/sda

As I haven't been able to tame the automounting feature of Kubuntu, I used
Gparted-Live and partitioned from there. When I partitioned my USB drive in
Kubuntu, it persistently automounted the first partition (sda1) after it had
been created, even the whole partitioning process hadn't been finished yet.

Thus, it's the value of having a pocket-CD of Gparted-Live everywhere you go.

Now I have a 128MiB USB pen-drive partitioned into 80s MiB FAT16 and 30s MiB EXT2

And...

250 GiB USB Harddrive partitioned into 40s GiB FAT32, and two 100s GiB EXT3

Now ... I am backing-up everything :-D

Managing Wireless Equipment (802.11x) using ap-utils

Being maintaining snmp-based nms for months, I've just found this tool few
hours ago. With ap-utils, I can manage my SmartBridge devices natively from
Linux. Ironically, I've been struggling with wine for years just to configure
my WiFi equipment, and even installing windows 98 via Qemu just to have the
wireless configuration software installed.

To install it in Ubuntu :

sudo apt-get install ap-utils

then

ap-config

The menu inside ap-utils is self-explanatory.
If only this tools can manage other proprietary wi-fi devices, (alvarion,
canopy, trango, etc) it will be much better.

QEMU in Ubuntu Dapper

Should you need another OS emulated within your system, QEMU is the Free Software that do the job well.
In Ubuntu 6.06, you can have the version 0.8.0 packaged in the universe repo.

To use it, just install it using apt-get (or synaptic if you prefer):

$ sudo apt-get install qemu

then create the virtual hard drive using qemu-img

$ qemu-img create mydisk 3G

to install an OS into your emulated hard drive (using CD-ROM installer)

$ qemu -hda mydisk -cdrom /dev/cdrom -boot d

Qemu can be emulating at higher speed when it uses its accelerator module named Kqemu. However Kqemu is a proprietary software (although it'll cost 0 in price). To install the accelerator, you can use the method posted here

Linux Only

When my desknote crashed several days ago, I've tried to put another flavor of UNIX into it.
Yup, I tried to boot Belenix (Solaris-based) and PC-BSD (FreeBSD-based).
Both kernel got panic and dumped.
It seems that Linux is the only "free" kernel that will boot up my transmeta box.
(My Sempron 2100+ box has no problem at all)

Linux rules :-D

"Crippled" Repo

After hours of trying and failing net-install Ubuntu 6.06 (Dapper Drake) from my local repository, I finally found out what's missing from it.
I did not include main/debian-installer in my debmirror script!
Thus, the repository is only good for installing packages into up & running Ubuntu system.
With less than 3 GB disk space remaining, I don't think I can afford to add main/debian-installer into my local repo.
Thus, having CD-installer ready is a must.

Proxy Traceroute

When I need to ensure that an IP address is properly routed and reachable, I often use the traceroute service of www.nwtools.com. However when I have to query frequently, the slowness of firefox on my laptop is such a hinderance.
To keep myself from hammering down my laptop, I create shell script to do the task using elinks and sed. Here is my code:

#!/bin/sh
elinks -dump --no-references --no-numbering \ "http://www.nwtools.com/default.asp?prog=trace&host=$1"|sed '1,21d'|sed -n -e :a -e '1,7!{P;N;D;};N;ba'

I saved the script as proxytrace.sh and using it with the traceroute target as the only parameter.

example:
$ sh proxytrace.sh www.blahblahblah.com
or
$ sh proxytrace.sh 123.234.123.234

This way is much faster, really.

A Geek's Bedroom

Kubuntu, with ATI RADEON 9250 Chipset, and TV-OUT, means Multimedia Playback in the Bedroom!
Now...if I can build MythTV on it...